Multiplex over ssh #126
Open
Multiplex over ssh #126
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 9, 2020 12:00
I'd like to confirm my understanding of when SSH and dial connections are established with the docker-modem agent for SSH. Call "debug" a key points when connections are established and ended.
When implicitly using the docker-modem agent for SSH by specifying
"protocol" as a connection option, a new agent is created for each
request. However, it is not suitable when passed as the "agent" option
to Modem(), in which case the same agent will be reused across requests.
This is because the docker-modem ssh agent shares one SSH client for all
connections of the HTTP agent whose createConnection method it
overrides. However, the agent appears to be capable of handling
multiple exec calls in parallel. This means that when one stream
closes, the client will be ended, disrupting any other connections still
open. This is fine for short-lived sequential HTTP requests made with
the agent, but if we create and attach to a docker container and then
make another request (for example, to start it), the third request will
disconnect the attached streams from the attach request.
Mon, 09 Nov 2020 19:33:40 GMT modem.ssh1 createConnection
2020-11-09T19:33:40.013Z modem Sending: { path: /containers/create?Image=ubuntu&AttachStdin=true&AttachStdout=true&AttachStderr=true&Tty=true&Cmd=%2Fbin%2Fbash&OpenStdin=true, ... }
Mon, 09 Nov 2020 19:33:40 GMT modem.ssh1 ready
Mon, 09 Nov 2020 19:33:40 GMT modem.ssh1 dialed
2020-11-09T19:33:40.251Z modem Received: {"Id":"d13f119a1a1e60b9b54d6685e3c8e6451221b68fa9baee7f5d9c9b0ac06e52b6","Warnings":[]}
Mon, 09 Nov 2020 19:33:40 GMT modem.ssh1 close
Mon, 09 Nov 2020 19:33:40 GMT modem.ssh1 end
# sleep after request to createContainer
Mon, 09 Nov 2020 19:33:42 GMT modem.ssh2 createConnection
2020-11-09T19:33:42.258Z modem Sending: { path: /containers/d13f119a1a1e60b9b54d6685e3c8e6451221b68fa9baee7f5d9c9b0ac06e52b6/attach?stream=true&stdout=true&stderr=true&stdin=true, ... }
Mon, 09 Nov 2020 19:33:42 GMT modem.ssh2 ready
Mon, 09 Nov 2020 19:33:42 GMT modem.ssh2 dialed
# sleep after request to "attach": no "end" or "close" logged because
# "attach" holds the connection open
Mon, 09 Nov 2020 19:33:44 GMT modem.ssh3 createConnection
2020-11-09T19:33:44.417Z modem Sending: { path: /containers/d13f119a1a1e60b9b54d6685e3c8e6451221b68fa9baee7f5d9c9b0ac06e52b6/start, ... }
# both streams are closed (the one for "attach" and for "start")
Mon, 09 Nov 2020 19:33:44 GMT modem.ssh2 end
Mon, 09 Nov 2020 19:33:44 GMT modem.ssh3 end
Mon, 09 Nov 2020 19:33:44 GMT modem.ssh3 ready
Mon, 09 Nov 2020 19:33:44 GMT modem.ssh3 dialed
2020-11-09T19:33:44.964Z modem Received:
Mon, 09 Nov 2020 19:33:44 GMT modem.ssh3 close
Make the ssh connection local to the http agent, which creates a new,
separate ssh connection each time the http agent requests a new
connection. This allows "attach" to stay attached.
SSH is capable of multiplexing multiple streams over one
ssh-authenticated connection, but we are establishing a new connection
for each http agent connection.
Connect over SSH only once for the modem instance and create new docker
connections by calling dial as many times as needed to open new
multiplexed streams.
In other words, create this situation:
PID USER RSS ARGS
24566 root 8940 \_ sshd: root@notty
24620 root 57600 | \_ docker system dial-stdio
24621 root 56936 | \_ docker system dial-stdio
24622 root 57152 | \_ docker system dial-stdio
not this one:
23901 root 8884 \_ sshd: root@notty
23954 root 58268 | \_ docker system dial-stdio
23911 root 8796 \_ sshd: root@notty
23978 root 58020 | \_ docker system dial-stdio
23924 root 8844 \_ sshd: root@notty
23991 root 58180 | \_ docker system dial-stdio
Trace:
modem.ssh createConnection +0ms
modem Sending: { path: /containers/create?Image=ubuntu&AttachStdin=true&AttachStdout=true&AttachStderr=true&Tty=true&Cmd=%2Fbin%2Fbash&OpenStdin=true, ... } +0ms
modem.ssh ready +52ms
modem.ssh dialed +80ms
modem Received: {"Id":"9d435ecdff50659bc7b602119c2e364ee11e9110bffc3af239d5a21623efec2a","Warnings":[]}
+ +254ms
modem.ssh createConnection +129ms
modem Sending: { path: /containers/9d435ecdff50659bc7b602119c2e364ee11e9110bffc3af239d5a21623efec2a/attach?stream=true&stdout=true&stderr=true&stdin=true, ... } +2ms
modem.ssh close +2ms
modem.ssh dialed +8ms
modem.ssh createConnection +44ms
modem Sending: { path: /containers/9d435ecdff50659bc7b602119c2e364ee11e9110bffc3af239d5a21623efec2a/start, ... } +55ms
modem.ssh dialed +2ms
modem Received: +407ms
modem.ssh close +434ms
Performance when connecting to a host on the other side of the world (a
worst-case scenario) improves from 16-20 sec to 6-8 sec to make these
three calls plus a kill and wait (5 calls total).
|
Would love to see more input about this change. |
|
@apocas: There is some additional detail regarding rationale in each of the commit messages. Does that help meet your request for more input about the change? Also, I'm a little confused about the test suite idea to use dockerode's test suite via SSH. Are you suggesting that docker-modem should gain a dependency on dockerode and run dockerode's suite with a docker-modem configuration that uses SSH? (Normally I would expect people to want to avoid such a circular dependency.) What should the suite connect to over SSH, a docker host configured by Vagrant or something? I noticed the existing test coverage for docker-modem seems to focus mostly on how configuration is applied, not on how docker-modem actually communicates, so following the precedent of tests for the HTTP agents wouldn't seem to accomplish the goal of using dockerode's suite (unless I'm missing part of how docker-modem is tested when I look at the test/ directory of docker-modem?). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Does this seem like a useful change for docker-modem? For my purposes, it means both a performance boost and improved ability to use docker-modem (and dockerode) without running ssh-agent.
I'm building on some ideas from this comment about running without ssh-agent.