Implement system tests for loading zones.

[ximon18]

• From file, AXFR in and IXFR in. (see Add simple IXFR system test #492 and Add a system test to check that receiving multipart AXFR works as expected. #505)

• For XFR loading should respond to an upstream NOTIFY. (see Add a system test to check that receiving a NOTIFY for an unknown zone is handled correctly. #504)

• We should honour the SOA REFRESH, RETRY and EXPIRE timers.

• Test with all RR types that we support. (see Add a system test that loads a zonefile that includes all RR types that we currently support. #530)

[ximon18]

Regarding testing of IXFR in, we have to ask ourselves how should such testing work, given that from an operator perspective the result is/should be the same irrespective of whether IXFR or AXFR in is used.

We also have to consider what we want to test. Relevant OpenDNSSEC tests for consideration include:

• https://github.com/opendnssec/opendnssec/tree/2.1.14/testing/test-cases.d/signer.adapters.input_basic
• https://github.com/opendnssec/opendnssec/tree/2.1.14/testing/test-cases.d/signer.adapters.input_with_refresh
• https://github.com/opendnssec/opendnssec/tree/2.1.14/testing/test-cases.d/signer.adapters.input_retry_expires
• https://github.com/opendnssec/opendnssec/tree/2.1.14/testing/test-cases.d/signer.adapters.input_ixfr_notimpl

Many things can go wrong here but causing them to go wrong is difficult with a real nameserver such as NSD. For "unhappy" flows such as TCP errors, timeouts, malformed responses, supposedly changed but actually unchanged data, etc. We would have more control if we used something like ldns-testns (which has already been ported, in draft form at least, to dnst, see NLnetLabs/dnst#56). There are also legitimate "unhappy" flows such as fallback from UDP to TCP and fallback from IXFR to AXFR to be tested.

We also need to consider how much control we need to have over the upstream that is serving XFR to Cascade. If the upstream is the primary NSD used by the system test mechanism currently, we can alter the zonefile on disk and issue an nsd reload CLI command to and have NSD update its copy of the zone. It should issue a NOTIFY to Cascade causing it to make a SOA query followed by an IXFR request. With BIND an alternative would be to use DNS UPDATE messages.

A reasonable starting point would be to begin by testing "happy" flows, but even then we have to consider carefully which data sources are available that the test can use to robustly determine if the expected behaviour has occurred. Possible data sources include:

• Logs: A lot of OpenDNSSEC testing relies on parsing logs but parsing logs is brittle as log messages are not guaranteed to remain the same in future, they are not a stable guaranteed API and so tests may fail later if depending on messages that NSD and/or Cascade log today.
• Network captures: tcpdump or tshark could be used to capture network traffic and filter based on the ports in use by the test. We would need a robust way to know when to terminate the capture, running it for N seconds would be brittle.
• CLI status commands: This could be useful for detecting that the SOA serial has changed as expected, but would have to be polled for a period of time and anything time based is brittle. cascade zone status should reveal the current SOA serial of the loaded zone and whether it was received via the network or loaded from a file. It doesn't currently (IIRC) indicate if the update was via AXFR or IXFR however. The zone history command might also be useful.
• Review hooks: An unsigned review hook would be invoked after the zone is updated via XFR in and could check that the SOA serial has changed as expected. It would need a way to signal its conclusions/status in some way to the test script.
• Metrics: The metrics published at present are too generic / high level to be useful for this test. We would need to publish more fine grained metrics such as last SOA serial per zone or I/AXFR specific metrics for metrics to be useful for the testing we need to do.
• DNSTAP: NSD is DNSTAP capable, could that be used to monitor what is served?
• IXFR persisted data: Cascade doesn't currently persist incoming IXFR data so we can't check what it writes to disk.

Also not mentioned yet: multi-part IXFR responses.

Without a mechanism to simulate the advance of time in Cascade any testing of the handling of SOA timers, e.g. retrying a failed XFR or expiring a zone, will need to use low time values and wait for time to progress normally.