[bug] Pygmy dnsmasq issues with Colima

[keyle]

Describe the bug
We use the pygmy stack with amazee projects on macOS and I made the transition to Colima from Docker App (enough of that turd).

Everything appears honky dory except for the routing. Dnsmasq is not happy with *.docker.amazee.io. We also found a temporary fix.

To Reproduce
Steps to reproduce the behavior:

1. Install Colima (after uninstalling Docker Desktop)

• brew install docker docker-compose colima
• docker context use colima
• edit ~/.docker/config.json and remove line "credsStore" if it is set to "desktop"
• add export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock" to zhrc (for apps unaware of docker context)

2. colima start --cpu 8 --memory 10 --network-address
3. pygmy up
4. docker-compose up on an amazee project

Expected behavior
Routing should work to project.docker.amazee.io

Output
Browser attempts to resolve the name and fails after about 30s.

ERR_NAME_NOT_RESOLVED

12:18 ~ docker exec amazeeio-dnsmasq ping project.docker.amazee.io
ping: bad address 'project.docker.amazee.io'
12:29 ~ docker exec amazeeio-dnsmasq ping amazee.io
PING amazee.io (151.101.2.191): 56 data bytes
64 bytes from 151.101.2.191: seq=0 ttl=63 time=2.217 ms

Additional context
We have the following fix: rm -f /etc/resolver/docker.amazee.io following every pygmy up. And it works without it. I am not exactly sure why it works without it, but it certainly doesn't work with it.

Contents of that file:

# Generated by amazeeio pygmy
nameserver 127.0.0.1
domain docker.amazee.io
port 6053

Exported configuration

Defaults: true
Domain: docker.amazee.io
Keys: null
Networks:
  amazeeio-network:
    Attachable: false
    ConfigFrom:
      Network: ""
    ConfigOnly: false
    Containers: null
    Created: "0001-01-01T00:00:00Z"
    Driver: ""
    EnableIPv6: false
    IPAM:
      Config:
      - Gateway: 10.99.99.1
        Subnet: 10.99.99.0/24
      Driver: ""
      Options: null
    Id: ""
    Ingress: false
    Internal: false
    Labels:
      pygmy.name: amazeeio-network
    Name: amazeeio-network
    Options: null
    Scope: ""
Resolvers:
- Data: |
    # Generated by amazeeio pygmy
    nameserver 127.0.0.1
    domain docker.amazee.io
    port 6053
  Enabled: true
  File: docker.amazee.io
  Folder: /etc/resolver
  Name: MacOS Resolver
Services:
  amazeeio-dnsmasq:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd:
      - --log-facility=-
      - -A
      - /docker.amazee.io/127.0.0.1
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/dnsmasq
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-dnsmasq
        pygmy.weight: "13"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd:
      - NET_ADMIN
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        53/tcp:
        - HostIp: ""
          HostPort: "6053"
        53/udp:
        - HostIp: ""
          HostPort: "6053"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-haproxy:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env:
      - AMAZEEIO_URL=docker.amazee.io
      Hostname: ""
      Image: pygmystack/haproxy
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-haproxy
        pygmy.network: amazeeio-network
        pygmy.url: http://docker.amazee.io/stats
        pygmy.weight: "14"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds:
      - /var/run/docker.sock:/tmp/docker.sock
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: ""
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        80/tcp:
        - HostIp: ""
          HostPort: "80"
        443/tcp:
        - HostIp: ""
          HostPort: "443"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-mailhog:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env:
      - MH_UI_BIND_ADDR=0.0.0.0:80
      - MH_API_BIND_ADDR=0.0.0.0:80
      - AMAZEEIO=AMAZEEIO
      - AMAZEEIO_URL=mailhog.docker.amazee.io
      ExposedPorts:
        80/tcp: {}
        1025/tcp: {}
        8025/tcp: {}
      Hostname: ""
      Image: pygmystack/mailhog
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-mailhog
        pygmy.network: amazeeio-network
        pygmy.url: http://mailhog.docker.amazee.io
        pygmy.weight: "15"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: "0"
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: ""
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        1025/tcp:
        - HostIp: ""
          HostPort: "1025"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-ssh-agent:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/ssh-agent
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-ssh-agent
        pygmy.network: amazeeio-network
        pygmy.output: "false"
        pygmy.purpose: sshagent
        pygmy.weight: "10"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings: null
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-ssh-agent-add-key:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/ssh-agent
      Labels:
        pygmy.defaults: "true"
        pygmy.discrete: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-ssh-agent-add-key
        pygmy.network: amazeeio-network
        pygmy.output: "false"
        pygmy.purpose: addkeys
        pygmy.weight: "31"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings: null
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: ""
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom:
      - amazeeio-ssh-agent
    NetworkConfig:
      EndpointsConfig: null
SortedServices:
- amazeeio-ssh-agent
- amazeeio-dnsmasq
- amazeeio-haproxy
- amazeeio-mailhog
- amazeeio-ssh-agent-add-key
Volumes: {}

[fubarhouse]

I'm afraid I'm facing issues getting Colima working on my systems too - facing similar issues.
If there's some guidance the community can recommend that would be ideal...

Frankly I have no way to replicate this exactly, but DNS is basically broken in Linux under Colima.

[rsau]

@keyle
I had exactly same issue and I can confirm that rm -f /etc/resolver/docker.amazee.io is the only thing that worked for me.
I get following if I run pygmy status but everything works without an issue.

@fubarhouse

I also had issues with Colima in the past on Mac M1 but after running following, all started working
https://github.com/abiosoft/colima/tree/main#upgrading

colima delete # delete existing instance
colima start

[fubarhouse]

That makes a bit of sense - though it doesn't help my situation.

If you could confirm this @keyle, I could perhaps write a command to write out a config value to disable the resolver.

[rsau]

I also forgot to mention that I always have to run pygmy up twice after switching to Colima from Docker Desktop

First time I always get

Second time it starts without any issue.

[keyle]

I can confirm, I delete the resolver after every pygmy up... Otherwise it doesn't work.

[fubarhouse]

@rsau you might be interested in the following solution - it is known but unrelated.

#442 (comment)

[rsau]

Thanks @fubarhouse
In Linux it so easy to fix that but on Mac it is another story.

[iijiang]

I can confirm that removing /etc/resolver/docker.amazee.io works for me. Previously, I was only editing the resolver config instead of actually deleting it. Removing the file solved the issue.

An alternative is to run a local dnsmasq (not in a container) on the MacOs to override the amazee-dnsmasq service. This approach requires a DNS configuration file under the /etc/resolver/ folder tho.

[iijiang]

Hi I did a fix: #625

Added proper support for --no-resolver in Pygmy by introducing a ResolverConfig struct with an Enabled flag in the main Config. Now, when --no-resolver is used, Pygmy skips creating /etc/resolver/docker.amazee.io.

[rsau]

@fubarhouse and @iijiang

It started working for me with latest release.
Now I don't have to delete /etc/resolver/docker.amazee.io anymore

[liberatr]

Instead of Colima, I had Rancher 1.20, and I was having a similar problem. Removing /etc/resolver/docker.amazee.io helped.

[fubarhouse]

@liberatr I should have updated it here, but @iijiang and I worked on a --no-resolver type thing a while back to help with colima support. You can use the following as well to make it permanent

~/.pygmy.yml

resolversDisabled: true