Without a clear reason, Cascade now replies NXDOMAIN for a zone

[bortzmeyer]

The zone worked fine for several days, and zone status shows no problem:

% cascade zone status --detailed internautique.fr
Status report for zone 'internautique.fr' using policy 'default'
✔ Waited for a new version of the internautique.fr zone
✔ Loaded <serial number not yet known>
  Loaded at 2025-11-06T15:21:47+00:00 (5days 17h 9m 17s ago)
  Loaded 333 B and 7 records from the filesystem in 0 seconds
✔ Waited for approval to sign <serial number not yet known>
• Approval received to sign <serial number not yet known>, signing requested
DNSSEC keys:
  KSK tagged 35690:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key
    Actively used for signing
  ZSK tagged 53132:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+53132.key
    Actively used for signing
  Details:
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key expires at 2026-01-01T16:28:15Z
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+53132.key expires at 2025-11-13T15:20:13Z

But suddenly, during the night, Cascade now claimed the zone does not exist:

% dig @127.0.0.1 -p 8053 internautique.fr SOA    

; <<>> DiG 9.20.15-1~deb13u1-Debian <<>> @127.0.0.1 -p 8053 internautique.fr SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 64255
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 0
;; QUESTION SECTION:
;internautique.fr.	IN SOA

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8053(127.0.0.1) (UDP)
;; WHEN: Wed Nov 12 09:32:05 CET 2025
;; MSG SIZE  rcvd: 45

Of course, nsd then stopped serving the zone (test internautique.fr).

The last entry in zone history is several hours before the problem:

2025-11-06T19:22:16+00:00 2025110606 Signing succeeded (triggered by keyset cron modified keyset state)
2025-11-06T19:22:16+00:00 2025110606 Signed zone review pending
2025-11-06T19:22:16+00:00 2025110606 Signed zone review approved

Here are the states:

%  cat /var/db/cascade/keys/internautique.fr.cfg 
{
  "state_file": "/var/db/cascade/keys/internautique.fr.state",
  "keys_dir": "/var/db/cascade/keys",
  "use_csk": false,
  "algorithm": "Ed25519",
  "ksk_validity": {
    "secs": 5270400,
    "nanos": 0
  },
  "zsk_validity": {
    "secs": 604800,
    "nanos": 0
  },
  "csk_validity": {
    "secs": 31536000,
    "nanos": 0
  },
  "auto_ksk": {
    "start": false,
    "report": true,
    "expire": true,
    "done": true
  },
  "auto_zsk": {
    "start": true,
    "report": true,
    "expire": true,
    "done": true
  },
  "auto_csk": {
    "start": true,
    "report": true,
    "expire": true,
    "done": true
  },
  "auto_algorithm": {
    "start": true,
    "report": true,
    "expire": true,
    "done": true
  },
  "dnskey_inception_offset": {
    "secs": 86400,
    "nanos": 0
  },
  "dnskey_signature_lifetime": {
    "secs": 1209600,
    "nanos": 0
  },
  "dnskey_remain_time": {
    "secs": 604800,
    "nanos": 0
  },
  "cds_inception_offset": {
    "secs": 86400,
    "nanos": 0
  },
  "cds_signature_lifetime": {
    "secs": 1209600,
    "nanos": 0
  },
  "cds_remain_time": {
    "secs": 604800,
    "nanos": 0
  },
  "ds_algorithm": "Sha256",
  "default_ttl": 3600,
  "autoremove": true,
  "update_ds_command": []
}

%  cat /var/db/cascade/keys/internautique.fr.state 
{
  "keyset": {
    "name": "internautique.fr",
    "keys": {
      "file:///var/db/cascade/keys/Kinternautique.fr.+015+53132.key": {
        "privref": "file:///var/db/cascade/keys/Kinternautique.fr.+015+53132.private",
        "decoupled": false,
        "keytype": {
          "Zsk": {
            "available": true,
            "old": false,
            "signer": true,
            "present": true,
            "at_parent": false
          }
        },
        "algorithm": 15,
        "key_tag": 53132,
        "timestamps": {
          "creation": {
            "secs": 1762442413,
            "nanos": 507966142
          },
          "published": {
            "secs": 1762442413,
            "nanos": 508003529
          },
          "visible": {
            "secs": 1762442516,
            "nanos": 364948500
          },
          "ds_visible": null,
          "rrsig_visible": {
            "secs": 1762446126,
            "nanos": 557568356
          },
          "withdrawn": null
        }
      },
      "file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key": {
        "privref": "file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.private",
        "decoupled": false,
        "keytype": {
          "Ksk": {
            "available": true,
            "old": false,
            "signer": true,
            "present": true,
            "at_parent": true
          }
        },
        "algorithm": 15,
        "key_tag": 35690,
        "timestamps": {
          "creation": {
            "secs": 1762014495,
            "nanos": 154763044
          },
          "published": {
            "secs": 1762014495,
            "nanos": 154854842
          },
          "visible": {
            "secs": 1762014606,
            "nanos": 600916328
          },
          "ds_visible": {
            "secs": 1762025426,
            "nanos": 314693577
          },
          "rrsig_visible": null,
          "withdrawn": null
        }
      }
    },
    "rollstates": {}
  },
  "dnskey_rrset": [
    "internautique.fr. 3600 IN DNSKEY 256 3 15 /0lRDld3+HcVc2FejLF3zo/UHa+QifvFNNjpau5saWE=",
    "internautique.fr. 3600 IN DNSKEY 257 3 15 HIRZS+arysD2+vcvXGH0cBdXuWMpjsDL/0FhNfgHDY0=",
    "internautique.fr. 3600 IN RRSIG DNSKEY 15 2 3600 1763662931 1762366931 35690 internautique.fr. ICQEXbamentIAmZviskWpJDuyZtjeVmFdarhhi4UlSIK8mu+WufHcw6OCjTlKz5TVHwcS84wuOzJjDtKfWg3Bw=="
  ],
  "ds_rrset": [
    "internautique.fr. 3600 IN DS 35690 15 2 3AE87A0B2B73EAE8A94C2CF7225F0EF857E807543BD870C0D8715F3A746FA0ED"
  ],
  "cds_rrset": [],
  "ns_rrset": [],
  "cron_next": {
    "secs": 1763047213,
    "nanos": 508003529
  },
  "kmip": {
    "servers": {}
  },
  "internal": {}
}

Because of the timing, I wonder if it is related to #331 (but the keys are not yet expired).

May be a zone reload will fix that but I did no do it, in case there are other tests to run.

[ximon18]

Hi @bortzmeyer,

This is unexpected. I'll take a look and identify under what circumstances we stop serving a zone.

[ximon18]

I can't see anywhere that we remove a zone from the set that is published.

NXDOMAIN can be returned either because:

• no zone by the given name or class exists, or
• the query against the zone resulted in NXDOMAIN.

For the first case, the zone would have to have been removed the from published set. The only place that we write to the set of published zones is when handling a zone publication event, where the published zone of a given name is atomically removed and replaced by the new zone for that name. I don't see at a glance how the remove could succeed but the insert fail.

For the second case, it can't be the in-memory zone reading code from domain as since #214 we don't actually create that zone storage type any more in Cascade when signing, instead we create a LightWeightZone, but the zone query code for LWZ is quite dumb and cannot return NXDOMAIN...

Which implies that the issue must either be with an alpha1 version of Cascade (as #214 was included in alpha2) but that seems unlikely, or is indeed somehow a problem with the first case. Even if the zone existed but lacked a SOA RR that should cause a NOERROR response according to the LightWeightZone code.

@bortzmeyer: Can you confirm that this issue occurred with an alpha2 or later version of Cascade?

[ximon18]

Another option, but that seems unlikely too, is that Cascade was restarted and as we have no signed zone persistence at present that would mean that until again signed and approved the zone would not exist at the publication server.

[ximon18]

Given that the reported output includes:

  Loaded at 2025-11-06T15:21:47+00:00 (5days 17h 9m 17s ago)
  Loaded 333 B and 7 records from the filesystem in 0 seconds
✔ Waited for approval to sign <serial number not yet known>
• Approval received to sign <serial number not yet known>, signing requested

I think that loaded zone timestamp is only stored in memory which implies that Cascade was up for at least 5 days, i.e. not restarted.

The serial not yet known doesn't look right either.

[bortzmeyer]

@bortzmeyer: Can you confirm that this issue occurred with an alpha2 or later version of Cascade?

Alpha-3 (which does not log its version number, unfortunately, a cascade version command would be useful.)

[bortzmeyer]

I think that loaded zone timestamp is only stored in memory which implies that Cascade was up for at least 5 days, i.e. not restarted.

Yes (Cascade never crashes):

 % ps auxwww | grep cascade
cascade  3280524  0.0  0.2 296972  8856 ?        Sl   Oct30   2:12 /home/cascade/.cargo/bin/cascaded --daemonize --log-level debug --state=/var/lib/cascade/state.db --config=/etc/cascade/config.toml

[bortzmeyer]

For the first case, the zone would have to have been removed the from published set.

% cascade zone list
xn--potamochre-66a.fr
internautique.fr
rutaba.ga
cours-dns.fr

[ximon18]

@bortzmeyer: Can you confirm that this issue occurred with an alpha2 or later version of Cascade?

Alpha-3 (which does not log its version number, unfortunately, a cascade version command would be useful.)

cascade --version should work.

[bortzmeyer]

cascade --version should work.

Ooops, I missed that:

% cascade --version
cascade 0.1.0-alpha3

[ximon18]

Does cascade zone status now show that the zone has been signed and published, or is it still reporting ready to sign but not signed?

[ximon18]

Is there nothing interesting in the log file?

[ximon18]

For the first case, the zone would have to have been removed the from published set.

% cascade zone list
xn--potamochre-66a.fr
internautique.fr
rutaba.ga
cours-dns.fr

Does it respond to dig for the SOA of the other zones?

[bortzmeyer]

Does cascade zone status now show that the zone has been signed and published, or is it still reporting ready to sign but not signed?

No change. Can/should I zone reload?

% cascade zone status --detailed internautique.fr
Status report for zone 'internautique.fr' using policy 'default'
✔ Waited for a new version of the internautique.fr zone
✔ Loaded <serial number not yet known>
  Loaded at 2025-11-06T15:21:47+00:00 (5days 23h 43m 30s ago)
  Loaded 333 B and 7 records from the filesystem in 0 seconds
✔ Waited for approval to sign <serial number not yet known>
• Approval received to sign <serial number not yet known>, signing requested
DNSSEC keys:
  ZSK tagged 53132:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+53132.key
    Actively used for signing
  KSK tagged 35690:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key
    Actively used for signing
  Details:
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+53132.key expires at 2025-11-13T15:20:13Z
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key expires at 2026-01-01T16:28:15Z
    
% dig @127.0.0.1 -p 8053 internautique.fr

; <<>> DiG 9.20.15-1~deb13u1-Debian <<>> @127.0.0.1 -p 8053 internautique.fr
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58132
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 0
;; QUESTION SECTION:
;internautique.fr.	IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8053(127.0.0.1) (UDP)
;; WHEN: Wed Nov 12 16:05:31 CET 2025
;; MSG SIZE  rcvd: 45