Provide a way to swap out an iothr's writer while the thread is running

[edmonds]

It might be desireable to be able to replace a running iothr's writer, without the overhead of doing a full shutdown and bringing up a new iothr (and new input queues, etc.). For instance, runtime reconfiguration of a dnstap-enabled DNS server where the path to a unix writer socket has changed.

This might be possible without any additional overhead by upgrading the shutting_down field (checked in fstrm__iothr_thr()) from a bool to a general purpose control enum, e.g. running, shutting_down, writer_reload_requested. When set to writer_reload_requested, go close and destroy the old writer, and swap in the new writer.

Maybe a signature like:

fstrm_res fstrm_iothr_reload_writer(struct fstrm_iothr *, struct fstrm_writer **);

Probably it should return instantly, with the new writer swapped in asynchronously by the running I/O thread. (Because destroying the old writer potentially requires blocking network I/O, e.g. to close a socket.) It will return with fstrm_res_success if the writer was successfully taken over from the caller, or fstrm_res_failure otherwise, in which case the caller still owns the writer object and must call fstrm_writer_destroy() to dispose of it.

We should add a struct fstrm_writer *new_writer field in struct fstrm_iothr, protected by a mutex. Normally NULL when running ordinarily, but populated when fstrm_iothr_reload_writer() is called. Then fstrm__iothr_thr() is responsible for picking it up and moving it from .new_writer to .writer, under the lock.

Q: If the .new_writer field is not NULL when fstrm_iothr_reload_writer() is called and acquires the lock for updating .new_writer, should we fail and return fstrm_res_failure, or should we quietly call fstrm_writer_destroy() on the not-yet-loaded .new_writer and put the caller's most recent writer into .new_writer? (I lean towards the latter. It should be safe to call fstrm_writer_destroy() on a writer as long as it hasn't been opened yet, which will avoid the possibility of blocking.)