KSK rollover requires a "cascade zone reload" at the end?

[bortzmeyer]

At the end of a KSK rollover, I see this state:

% cascade zone status --detailed internautique.fr
Status report for zone 'internautique.fr' using policy 'default'
✔ Waited for a new version of the internautique.fr zone
✔ Loaded version 2025101721
  Loaded at 2025-10-30T14:38:16+00:00 (19h 49m 11s ago)
  Loaded 333 B and 7 records from the filesystem in 0 seconds
✔ Waited for approval to sign version 2025101721
• Approval received to sign version 2025101721, signing requested
DNSSEC keys:
  KSK tagged 49915:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+49915.key
  KSK tagged 23940:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+23940.key
    Actively used for signing
  ZSK tagged 30906:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+30906.key
    Actively used for signing
  Details:
    KskRoll: Done
    Check that the following RRset has propagated to all name servers:
    internautique.fr. 3600 IN DNSKEY 257 3 15 wEQGubpkbLnvk1bGn+2eG+Nr4ZpTLwI30It+ihnjY44=
    internautique.fr. 3600 IN DNSKEY 256 3 15 BQ9ge7VeiogFmlCxkJliWaxIMyOghwCniwMSS7Sps2g=
    internautique.fr. 3600 IN RRSIG DNSKEY 15 2 3600 1763044701 1761748701 23940 internautique.fr. 6kauoHJtZMZT9qzL9LA/C4+YkxtTDDbEONNuUtshPIzhm7NCy1Uq+rZo6BiaO2h+renqkBijIQ2s61NsOuCmCQ==
    
    For the next step run:
    	cascade keyset internautique.fr ksk roll-done
    	automation is enabled for this step.
    
    Automatic key roll state:
    Roll KskRoll, state Done:
    	Wait until the new DNSKEY RRset has propagated to all nameservers.
    	Try again after 2025-10-31T10:39:56Z
    
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+30906.key expires at 2025-11-03T08:07:27Z
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+49915.key is stale
    this key will be removed automatically after the next key roll
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+23940.key expires at 2025-12-30T06:30:51Z

And it does not move. The old KSK, 49915, is still published so we never reach the desired DNSKEY RRset. Doing a cascade zone reload internautique.fr solves the problem, a new zone is published with the correct DNSKEY RRset. Is it normal that I have to do this extra step?

[Philip-NLnetLabs]

This is strange. The key manager does not change the DNSKEY RRset after a cascade zone reload. So the only explanation would be that Cascade somehow didn't sign a new zone with the new DNSKEY RRset. Which is weird in the current version because Cascade always resigns the zone when the keyset state file is changed.

It is possible to get the zone the in the same state to do some debugging?

[bortzmeyer]

It is possible to get the zone the in the same state to do some debugging?

Rollover started, we'll see.

I suggest that the problem may be because cascade keyset internautique.fr ksk start-roll does not send a NOTIFY to the name server (tested with dnscap -i lo -m n -g). As a result, we have to wait a $refresh time before the name server queries Cascade, which gives the impression that nothing moves on.

cascade zone reload does send a NOTIFY so, this is why it resumes the process.

[Philip-NLnetLabs]

The notify issue is a weird bug. I thought Cascade does not send notifies at the moment. It might worth opening a separate issue about that.

[ximon18]

Cascade sends a NOTIFY (to targets configured in policy, if any) on publication of a zone. See zone_server.rs line 401.

Weirdly it also has NOTIFY receive handling in the zone server which should be removed. See use of NotifyMiddlewareSvc in zone_server.rs.

Update: the NOTIFY receive handling is probably because the same server/interface/port can be used both at the start and the end of the pipeline, and so needs to handle NOTIFY in to know that the upstream zone has changed, but the NotifyMiddlewareSvc is not optionally gated for use only by a publication instance of ZoneServer, so will also react to NOTIFY sent to zone review servers too...

[bortzmeyer]

Anyway, this time, rollover went fine. Cascade sent the NOTIFY:

[73] 2025-11-01 21:30:36.597333 [#18 lo 4095] \
	[127.0.0.1].43435 [127.0.0.1].53  \
	dns NOTIFY,NOERROR,34648, \
	1 internautique.fr.,IN,SOA 0 0 \
	1 .,1400,1400,0,edns0[len=0,UDP=1400,ver=0,rcode=0,DO=0,z=0]

The zone was transferred to the name servers.
And the rollover is complete:

% cascade zone status --detailed  internautique.fr
Status report for zone 'internautique.fr' using policy 'default'
✔ Waited for a new version of the internautique.fr zone
✔ Loaded version 2025101721
  Loaded at 2025-11-01T16:29:59+00:00 (17h 59m 1s ago)
  Loaded 333 B and 7 records from the filesystem in 0 seconds
✔ Waited for approval to sign version 2025101721
✔ Approval received to sign version 2025101721, signing requested
✔ Signed version 2025101721 as version 2025110108
  Signing requested at 2025-11-01T21:30:36+00:00 (12h 58m 25s ago)
  Signing started at 2025-11-01T21:30:36+00:00 (12h 58m 25s ago)
  Signing finished at 2025-11-01T21:30:36+00:00 (12h 58m 25s ago)
  Collected 8 records in 0s, sorted in 0s
  Generated 5 NSEC(3) records in 0s
  Generated 8 signatures in 0s (8 sig/s)
  Inserted signatures in 0s (8 sig/s)
  Took 0s in total, using 1 threads
  Current action: Finished
✔ Waited for approval to publish version 2025110108
✔ Published version 2025110108
  Published zone available on 127.0.0.1:8053
DNSSEC keys:
  KSK tagged 35690:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key
    Actively used for signing
  ZSK tagged 30906:
    Reference: file:///var/db/cascade/keys/Kinternautique.fr.+015+30906.key
    Actively used for signing
  Details:
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+35690.key expires at 2026-01-01T16:28:15Z
    key file:///var/db/cascade/keys/Kinternautique.fr.+015+30906.key expires at 2025-11-03T08:07:27Z
    

The result is OK.