Skip to content

skydns

Jump to bottom
Jesús Daniel Colmenares Oviedo edited this page Mar 27, 2025 · 1 revision

Using SkyDNS with CoreDNS

SkyDNS is an older protocol that implements service discovery in combination with etcd. In the context of SkyDNS, etcd is used to manipulate DNS records dynamically, so Overlord takes advantage of this to allow you to implement services with a handy domain name.

Although the original protocol is no longer maintained, CoreDNS, a great DNS server, implements this protocol through a plugin called etcd.

Before deploying CoreDNS, you should have deployed an etcd cluster and configure Overlord to use it. We will configure CoreDNS to also point to the same etcd endpoints, so that it can resolve DNS records through our custom domain, which for this article is overlord.lan..

/usr/local/etc/overlord.yml:

skydns:
  zone: 'overlord.lan.'

The dot is very important to correctly format the records when writing them to etcd. By default it is a dot, but since we are going to configure CoreDNS to also be a forwarder, it is necessary to specify a custom domain to resolve correctly.

Also note that the above configuration must be synchronized with each API server, or at least with the one we want to use to deploy our projects. The same applies to the parameters related to etcd.

supervisorctl restart overlord:*

Now we need to write some metadata, one for the Corefile (the CoreDNS configuration file) and the other for the Makejail that installs and configures CoreDNS.

coredns-metadata.yml:

kind: metadata
datacenters:
  main:
    entrypoint: 'http://127.0.0.1:8888'
    access_token: '<access token>'
deployIn:
  labels:
    - service001
    - service002
metadata:
  Corefile: |
    .:53 {
      bind tailscale0
      log
      errors
      forward . 208.67.222.222 208.67.220.220
      etcd overlord.lan. {
        endpoint http://<host-1>:2379 http://<host-2>:2379 http://<host-3>:2379
      }
      cache 30
    }
  coredns.makejail: |
    ARG corefile

    OPTION start
    OPTION overwrite=force

    INCLUDE gh+DtxdF/efficient-makejail

    PKG coredns

    CMD mkdir -p /usr/local/etc/coredns
    COPY ${corefile} /usr/local/etc/coredns/Corefile

    SYSRC coredns_enable=YES
    SERVICE coredns start

The next step is to create the deployment file to deploy our project.

We use the environment variable OVERLORD_METADATA that is defined when creating the project to specify the path where our Makejail and Corefile are located.

coredns.yml:

kind: directorProject
datacenters:
  main:
    entrypoint: 'http://127.0.0.1:8888'
    access_token: '<access token>'
deployIn:
  labels:
    - service001
    - service002
projectName: dns-server
projectFile: |
  options:
    - alias:
    - ip4_inherit:
  services:
    coredns:
      makejail: !ENV '${OVERLORD_METADATA}/coredns.makejail'
      arguments:
        - corefile: !ENV '${OVERLORD_METADATA}/Corefile'

The deployment file for our project has nothing special except the labels. Overlord reads some labels and perform some tasks to further automate. In this case, it reads the labels related to SkyDNS. overlord.skydns:1 tells Overlord that this project needs SkyDNS, overlord.skydns.group:hello-http is the subdomain that groups other subdomains (see below for a better explanation) and overlord.skydns.interface:tailscale0 is the interface to get the IP address which must exist on each API server.

hello-http.yml:

kind: directorProject
datacenters:
  main:
    entrypoint: 'http://127.0.0.1:8888'
    access_token: '<access token>'
deployIn:
  labels:
    - services
projectName: hello-http
projectFile: |
  options:
    - virtualnet: ':<random> default'
    - nat:
  services:
    darkhttpd:
      makejail: 'gh+DtxdF/hello-http-makejail'
      options:
        - expose: '8123:80 ext_if:tailscale0 on_if:tailscale0'
        - label: 'overlord.skydns:1'
        - label: 'overlord.skydns.group:hello-http'
        - label: 'overlord.skydns.interface:tailscale0'
      arguments:
        - darkhttpd_tag: 14.2

After defining all these deployment files, it is time to deploy them.

# overlord apply -f coredns-metadata.yml
# overlord apply -f coredns.yml
# overlord get-info -f coredns.yml -t projects --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: delta
  labels:
    - all
    - service002
    - services
  projects:
    dns-server:
      state: UNFINISHED
      last_log: 2025-03-27_14h20m28s
      locked: True
      services:
        - {'name': 'coredns', 'status': 0, 'jail': '96f1f07f33'}
      up:
        operation: RUNNING
        last_update: 2 minutes and 49.78 seconds
        job_id: 18
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: echo
  labels:
    - all
    - service001
    - services
  projects:
    dns-server:
      state: UNFINISHED
      last_log: 2025-03-27_03h55m07s
      locked: True
      services:
        - {'name': 'coredns', 'status': 0, 'jail': '563dffcde5'}
      up:
        operation: RUNNING
        last_update: 2 minutes and 49.47 seconds
        job_id: 18
# # wait
# overlord get-info -f coredns.yml -t projects --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: delta
  labels:
    - all
    - service002
    - services
  projects:
    dns-server:
      state: DONE
      last_log: 2025-03-27_14h20m28s
      locked: False
      services:
        - {'name': 'coredns', 'status': 0, 'jail': '96f1f07f33'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 54.67 seconds
        job_id: 18
        labels:
         error: False
         message: None
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: echo
  labels:
    - all
    - service001
    - services
  projects:
    dns-server:
      state: DONE
      last_log: 2025-03-27_03h55m07s
      locked: False
      services:
        - {'name': 'coredns', 'status': 0, 'jail': '563dffcde5'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 3 minutes and 28.35 seconds
        job_id: 18
        labels:
         error: False
         message: None

We exclude for the moment the deployment of the project because we first need to get CoreDNS working, which is already done, so proceed to configure your resolv.conf(5) or your DNS forwarder to point to each CoreDNS instance.

After all this, we can deploy the main project.

# overlord apply -f hello-http.yml
# overlord get-info -f hello-http.yml -t projects --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: bravo
  labels:
    - all
    - centralita
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_18h26m09s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': '9ad5c8c287'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 6 minutes and 7.95 seconds
        job_id: 10
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: delta
  labels:
    - all
    - service002
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_14h34m40s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': 'e8dda46984'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 5 minutes and 53.59 seconds
        job_id: 19
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: echo
  labels:
    - all
    - service001
    - services
  projects:
    hello-http:
      state: DONE
      last_log: 2025-03-27_04h09m17s
      locked: False
      services:
        - {'name': 'darkhttpd', 'status': 0, 'jail': '5642f55ce7'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 6 minutes and 19.99 seconds
        job_id: 19
        labels:
         error: False
         message: None
         load-balancer:
           services:
             darkhttpd:
               error: False
               message: None
         skydns:
           services:
             darkhttpd:
               error: False
               message: (project:hello-http, service:darkhttpd, records:[address:True,ptr:None,srv:None] records has been updated.

Depending on the number of matching API servers, the number of deployments you will have. In this case there were three matching ones. As you can see each one is grouping in the subdomain we have specified in our label (overlord.skydns.group), so querying this will return three IPs.

# host hello-http.overlord.lan
hello-http.overlord.lan has address <host-1>
hello-http.overlord.lan has address <host-2>
hello-http.overlord.lan has address <host-3>

When the same domain returns more than one IP address, this has a useful side effect:

# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: 94fe7911-a2b7-4436-b82e-584224e63a13
# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: 94fe7911-a2b7-4436-b82e-584224e63a13
# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: b09fdc48-5416-4b01-b80e-3e974411706a
# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: 11da9435-c5c8-4932-bcc6-5baaa0e5bfa2
# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: 94fe7911-a2b7-4436-b82e-584224e63a13
# curl http://hello-http.overlord.lan:8123
Hello, world!
UUID: b09fdc48-5416-4b01-b80e-3e974411706a

Requests are balanced between each service, and when one of these instances fails for whatever reason, the request will be attempted on another until it succeeds. Note that this is entirely client-dependent, which should not be a problem on modern clients, but expect incompatibilities with older ones.

If you have access to an etcd client, you can get the records created by Overlord:

# etcdctl --endpoints=http://<etcd-host>:2379 get --prefix=true /skydns
/skydns/lan/overlord/hello-http/853b1cf5-28d5-4a47-83d6-b32a913685c1/
{"host": "<host-1>", "ttl": 60, "group": "hello-http"}
/skydns/lan/overlord/hello-http/cf60be22-3e3a-492a-ab73-9eb0b8880738/
{"host": "<host-2>", "ttl": 60, "group": "hello-http"}
/skydns/lan/overlord/hello-http/f4b9e170-67bb-403e-88da-112c55b45fce/
{"host": "<host-3>", "ttl": 60, "group": "hello-http"}

There is something interesting to note. As you can see after each group there is a UUID. This UUID represents the server identifier of each API server, which must be unique (and is created by Overlord itself only once). This means that we can also query each individual service if we want to.

# curl http://f4b9e170-67bb-403e-88da-112c55b45fce.hello-http.overlord.lan:8123
Hello, world!
UUID: b09fdc48-5416-4b01-b80e-3e974411706a

Clone this wiki locally