Add support for socket activation

[nicoonoclaste]

• endlessh: Add a new option (-i) to expect a socket on stdin.
• systemd: Add two templated unit files (endlessh@.{service,socket})
  to support socket activation on arbitrary ports.
• endlessh: Error-out when -i is specified on the command-line and a port
  was explicitely configured (either by config file or CLI argument).
• systemd: Rework the default service (endlessh.service) to use socket
  activation, or remove it outright.

Rationale

• Supports listening on port 22 without issue. (see Unable to start endlessh on port 22 #39)
• Lowers overhead, as the service isn't started until the first connection.
• Improves security, as the socket-activated service:
  • runs under a transient user (not root!), even with ports <1024;
  • does not have any access to the network, aside from the socket it was
    started with.

[nicoonoclaste]

@skeeto Does this seems like a reasonable change to you?

[skeeto]

Hmm, I didn't realize systemd supported this kind of socket activation. I thought it only supported "inetd-style" activation on standard input, which is pointless for Endlessh. The official systemd documentation is very unclear, and I'm not convinced that this is actually supported as written. If it *is* supported, I'd like to accept this feature, but the PR needs more work. Have you tested this and does it actually work? Is the listening socket (not the connection socket) *actually* passed as standard input? I can't find documentation saying this is the case. Further, the "@" in the name suggests this is an "Accept=yes" service (multiple instances, inetd-style), but it's actually an "Accept=no" service. I couldn't figure out how to install and run as a systemd service, so I couldn't test it myself. I also don't understand how "ListenStream=%i" works since I can't find any documentation about the "%i". Assuming I can successfully verify that socket activation indeed works correctly as shown, the new option must be better integrated before I'd accept these changes. Rather than a new variable in main(), it should appear in "struct config" along with the other options, accessible as both a command line switch and in the config file (maybe something like SocketActivation yes/no?). Rather than bool — I dislike C99 _Bool / stdbool.h — add "int flags" and designate a flag bit for this option. The new option must also be documented in the man page. But before you worry about any of that, I want to confirm this style of socket activation actually works. Note: Anything to do with sd_listen_fds(3) is a non-starter.

[nicoonoclaste]

The official systemd documentation is very unclear, and I'm not convinced that this is actually supported as written.

I'm quite sure this is an intended functionality (and even the default, as I would have to give Accept=yes to make systemd pass connection sockets to endlessh).

If it is supported, I'd like to accept this feature, but the PR needs more work. Have you tested this and does it actually work?

Yes, it is running on my laptop currently (on port 22). You can see it in action, but here is the gist of it:

[Making sure endlessh is stopped initially]
$ sudo systemctl stop endlessh@22
Warning: Stopping endlessh@22.service, but it can still be activated by:
  endlessh@22.socket
$ ps aux | grep ssh
nicoo    2463981  0.0  0.0   6140  1740 pts/7    SN+  06:20   0:00 grep --color=auto ssh

[A connection causes it to start]
$ ssh localhost
^Z
[1]  + 2464056 suspended  ssh localhost
$ bg
[1]  + 2464056 continued  ssh localhost
$ ps aux | grep ssh
nicoo    2464056  0.0  0.0   9220  4748 pts/7    SN   06:20   0:00 ssh localhost
endlessh 2464057  0.8  0.0   2276   708 ?        Ss   06:20   0:00 /usr/bin/endlessh -i
nicoo    2464127  0.0  0.0   6140  1756 pts/7    SN+  06:20   0:00 grep --color=auto ssh

[A second connection gets handled by the same process]
$ ssh localhost
^Z
[2]  + 2464155 suspended  ssh localhost
$ bg
[2]  - 2464155 continued  ssh localhost
$ ps aux | grep ssh
nicoo    2464056  0.0  0.0   9220  4748 pts/7    SN   06:20   0:00 ssh localhost
endlessh 2464057  0.2  0.0   2276   708 ?        Ss   06:20   0:00 /usr/bin/endlessh -i
nicoo    2464155  0.0  0.0   9220  4704 pts/7    SN   06:20   0:00 ssh localhost
nicoo    2464212  0.0  0.0   6140  1852 pts/7    SN+  06:20   0:00 grep --color=auto ssh

[Stopping the single endlessh process causes both ssh clients to exit]
$ sudo systemctl stop endlessh@22
kex_exchange_identification: read: Connection reset by peer
kex_exchange_identification: read: Connection reset by peer
[2]  - 2464155 exit 255   ssh localhost
[1]  + 2464056 exit 255   ssh localhost
Warning: Stopping endlessh@22.service, but it can still be activated by:
  endlessh@22.socket

Is the listening socket (not the connection socket) actually passed as standard input?

Yes. The example above (with 2 clients handled by the same endlessh process) wouldn't work otherwise.

Further, the "@" in the name suggests this is an "Accept=yes" service (multiple instances, inetd-style), but it's actually an "Accept=no" service.

All that the @ denotes is that this is a templated unit.
In this particular case, I could spin up both endlessh@22 and endlessh@2222 if I wanted 2 instances, on ports 22 and 2222.

It is a bit confusing, because socket units with Accept=yes must be associated with a templated service (otherwise systemd can't figure out how to start multiple instances for multiple connections), but other kinds of units can be templated too; for instance, most distros template the unit getty@.service, which so they can have instances for a bunch of different TTYs out of the same configuration.

I couldn't figure out how to install and run as a systemd service, so I couldn't test it myself.

systemctl enable --now endlessh@22.socket will configure endlessh for socket activation, on port 22, effective immediately and on later boots.

I also don't understand how "ListenStream=%i" works since I can't find any documentation about the "%i".

%i is replaced by the instance name (i.e. the string between @ and .socket in endlessh@22.socket); this is documented in the systemd.unit(5) manpage.

Assuming I can successfully verify that socket activation indeed works correctly as shown, the new option must be better integrated before I'd accept these changes.

Sure, this was only a first go in the hopes of clarifying what's needed for socket activation.
I'm well aware the documentation around that concept is either systemd's (and pretty terrible) or essentially non-existent (in the case of other service managers like s6).

Rather than a new variable in main(), it should appear in "struct config" along with the other options, accessible as both a command line switch and in the config file (maybe something like SocketActivation yes/no?).

I avoided making it settable in the config file, specifically because it only makes sense for the caller of endlessh to set it (only it knows whether it's passing an AF_INET socket as stdin).
Does that make sense?

Note: Anything to do with sd_listen_fds(3) is a non-starter.

I'm... not particularly keen on libsystemd either :/
As I mentionned, this way of doing it also works with other service managers that support socket activation (as in, passing a bound socket, not a connection one).

[skeeto]

Thank you for taking the time to explain all this. Everything is making more sense. The confusing part of the documentation is systemd.socket(5) which says:

Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details) or via the traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).

This is neither sd_listen_fds(3) nor inetd-style, so this sentence in the documentation is incomplete.

So my next question is: How do I use this? Here's what I tried on your branch (Debian Buster):

$ make
$ sudo cp endlessh /usr/bin/
$ sudo cp util/endlessh@.s* /etc/systemd/system/
$ sudo systemctl enable endlessh@2222
$ sudo systemctl start endlessh@2222
Job for endlessh@2222.service failed because of unavailable resources or another system error.
See "systemctl status endlessh@2222.service" and "journalctl -xe" for details.
$ sudo journalctl -u endlessh@2222.service | head -n5
-- Logs begin at Sun 2020-01-05 19:49:36 EST, end at Fri 2020-01-31 10:55:15 EST. --
Jan 31 10:43:55 bee systemd[1]: endlessh@2222.service: Got no socket.
Jan 31 10:43:55 bee systemd[1]: endlessh@2222.service: Failed to run 'start' task: Invalid argument
Jan 31 10:43:55 bee systemd[1]: endlessh@2222.service: Failed with result 'resources'.
Jan 31 10:43:55 bee systemd[1]: Failed to start Endlessh SSH Tarpit.

I'm stumped at this point.

[nicoonoclaste]

Thank you for taking the time to explain all this. Everything is making more sense.

Thanks for being patient about this. I'm glad it makes more sense now.

The confusing part of the documentation is systemd.socket(5) [...] This is neither sd_listen_fds(3) nor inetd-style, so this sentence in the documentation is incomplete.

Yeah, that's indeed pretty confusing. :/

So my next question is: How do I use this? Here's what I tried on your branch (Debian Buster):

$ make
$ sudo cp endlessh /usr/bin/
$ sudo cp util/endlessh@.s* /etc/systemd/system/

At that point, you need systemctl daemon-reload (to make systemd reload the files you just copied) and systemctl enable --now endlessh@2222.socket (to make it bind the socket, now and on later boots).

After that, ssh -p2222 localhost will cause the service to be autostarted (or you can start it manually with systemctl start endlessh@2222, once the socket is already bound)

I should probably add some relationship between endlessh@.socket and .service so enabling the service causes the socket to be enabled too.

$ sudo systemctl enable endlessh@2222
$ sudo systemctl start endlessh@2222
Job for endlessh@2222.service failed because of unavailable resources or another system error.
See "systemctl status endlessh@2222.service" and "journalctl -xe" for details.
$ sudo journalctl -u endlessh@2222.service | head -n5
-- Logs begin at Sun 2020-01-05 19:49:36 EST, end at Fri 2020-01-31 10:55:15 EST. --
Jan 31 10:43:55 bee systemd[1]: endlessh@2222.service: Got no socket.
Jan 31 10:43:55 bee systemd[1]: endlessh@2222.service: Failed to run 'start' task: Invalid argument
Jan 31 10:43:55 bee systemd[1]: endlessh@2222.service: Failed with result 'resources'.
Jan 31 10:43:55 bee systemd[1]: Failed to start Endlessh SSH Tarpit.

Yeah, starting the service when the socket isn't present doesn't make sense (though systemd's error reporting is also pretty unhelpful).

Sorry for the incomplete/confusing instructions on how to test it; since I tested those changes with a local build of the Debian package, things got automatically reloaded and such upon reinstall/upgrade >_>'

[nicoonoclaste]

@skeeto I just updated the feature branch to solve the merge conflict, and while I was at it added some improvements to the systemd unit that I was experimenting with locally.

Were you able to confirm this works as desired?

[nicoonoclaste]

Ping?

[martinetd]

obviously can't speak for @skeeto but nice! better than giving caps. Thanks both!

[martinetd]

fwiw my version of systemd (v245.4) complains about ~socket: /etc/systemd/system/endlessh@.service:42: Failed to parse system call, ignoring: ~socket

[nicoonoclaste]

Good call, thanks for catching this; looks like I cannot combine whitelist and blacklist in this way >_>'

[lucaswerkmeister]

You can, but the ~ operator applies per line, not per whitespace-separated item. This should work better:

SystemCallFilter=@basic-io @file-system @io-event @network-io @signal
SystemCallFilter=arch_prctl brk mprotect
SystemCallFilter=~socket

(Many services shipped by systemd start with an allowlist and then add SystemCallFilter=~@privileged to filter out privileged syscalls again, for instance.)

[martinetd]

fwiw the only thing you need to have systemctl start endlessh@2222 work is Requires=endlessh@%i.socket here -- systemd will automatically pull the socket in then.

[bill-mcgonigle]

This worked well for me and it obviated the need for #21 . ListenStream supports IP's, IP/port, sockets, etc. so it's very flexible. I made an array of the listens I wanted in a puppet class and a loop to deploy them. Slick!

https://www.freedesktop.org/software/systemd/man/systemd.socket.html