isortd server process

[wyuenho]

Starting the python process takes forever, but once it's started, formatting is acceptably fast. I was just wondering if there is any plan to incorporate or write a daemon server ala blackd or eslint_d or prettier_d? A gentleman seems to have already made a start, it would be nice if the two projects could merge too.

urm8/isortd#2

[timothycrosley]

I would love to see this added to isort! @urm8 let me know if you have any interest as well, I can help with quick reviews into the repository, or in any other way you would find helpful.

@wyuenho thanks for highlighting this oppurtunity!

[urm8]

Hi @timothycrosley , that would be great to see isortd as a part of isort project ! But there is one thing I don't like in current isortd implementation(as in blackd impl also) that is very limited in terms of tools devs may use it with. For now that thing is available only for PyCharm users through IsortConnect plugin. Its just opens http api to "isort", and I think, that it would be better to implement some kind of filewatcher service for isort, that will watch for file changes within directory and run isort on em (even brought some pr for that: urm8/isortd#6) than rely on some third party clients to do that. I'd like to hear your opinion on how that better be implemented.

[timothycrosley]

@urm8 a file watcher sounds cool! Another option I could see being interesting, is to write an optimized minimal Python entrypoint, that if it detects a running isortd process uses that, and otherwise reroutes to the existing CLI. As starting python itself is generally fast enough (14ms on my laptop, with a quick test run of time python "exit()". If that could reasonably be pulled off, it would mean most existing tooling (which utilize isort's CLI, for example VSCode) would automatically become much faster if isortd was running and configured. Curious if you have thoughts on this approach?

[disrupted]

@urm8 a file watcher sounds cool! Another option I could see being interesting, is to write an optimized minimal Python entrypoint, that if it detects a running isortd process uses that, and otherwise reroutes to the existing CLI. As starting python itself is generally fast enough (14ms on my laptop, with a quick test run of time python "exit()". If that could reasonably be pulled off, it would mean most existing tooling (which utilize isort's CLI, for example VSCode) would automatically become much faster if isortd was running and configured. Curious if you have thoughts on this approach?

To me this makes much more sense than a file-watcher as I imagine most people invoke isort in their text editor and/or pre-commit hooks.

[wyuenho]

The command to test python start up time is actually time python -c "exit()". It took about 0.2s on my MBP 2014.

The issue a daemon server tries to solve is not just starting the python interpreter, but to also "warm it up", as in, read, parse and execute all the relevant imports, so when input comes in, the program goes straight to formatting.

A really dumb but effective isortd implementation doesn't even need to use aiohttp or any fancy IPC, all it has to do is to make a worker process pool of isort waiting for stdin. When the daemon receives from stdin, it can just send it to a process in the pool and return its stdout. The isort worker process will exit, but the pool will have another one available to accept "requests". The daemon can simply start another worker process once one has exited.

[urm8]

@urm8 a file watcher sounds cool! Another option I could see being interesting, is to write an optimized minimal Python entrypoint, that if it detects a running isortd process uses that, and otherwise reroutes to the existing CLI. As starting python itself is generally fast enough (14ms on my laptop, with a quick test run of time python "exit()". If that could reasonably be pulled off, it would mean most existing tooling (which utilize isort's CLI, for example VSCode) would automatically become much faster if isortd was running and configured. Curious if you have thoughts on this approach?

The command to test python start up time is actually time python -c "exit()". It took about 0.2s on my MBP 2014.

The issue a daemon server tries to solve is not just starting the python interpreter, but to also "warm it up", as in, read, parse and execute all the relevant imports, so when input comes in, the program goes straight to formatting.

A really dumb but effective isortd implementation doesn't even need to use aiohttp or any fancy IPC, all it has to do is to make a worker process pool of isort waiting for stdin. When the daemon receives from stdin, it can just send it to a process in the pool and return its stdout. The isort worker process will exit, but the pool will have another one available to accept "requests". The daemon can simply start another worker process once one has exited.

So the plan is:

1. parse config, launch daemon that holds stdin
2. isort entry point stats some file that indicates, that daemon is running , acquires write lock, and pushes contents to daemon and waits for output
3. daemon receives file contents, and sends it to process pool (mb simple fork suites here better ?), then returns to stdin
4. process spits sorted file to stdout
5. isort takes it, releases the lock and continues execution as usual
   Am I correct ?

[wyuenho]

Something like that, whether the worker process will exit after formatting and requires starting a new one to replenish the pool is an implementation detail that depends on how lazy you want to be lol. A process pool of 2 goes a long way.

[timothycrosley]

I'm starting to wonder if the basis of this issue is fully accurate:

Starting the python process takes forever, but once it's started, formatting is acceptably fast.

tim@pop-os:~/Projects/isort$ time isort . --show-config
real    0m0.111s
user    0m0.095s
sys     0m0.016s

That includes all the setup time for isort (loading configs etc) that could be saved by a daemon. Sure some projects have larger configs, but I'm kinda curious is one tenth of a second really slow? What is an acceptable delay? Is it longer for others? Mostly, I ask, because if this is a goal it will be important to quantify what fast enough is, as that will greatly impact the design of any solution. I also ask, because it's possible, that there is something wrong with individual config options, or similar, causing some users to experience a long delay that is not widely experienced - and if this is the case it may make sense for isort to fix those performance issues separately regardless.

[wyuenho]

Movies play at 24 fps and if you use an old HDMI cable that only supports 30fps to connect your laptop to the screen, the mouse pointer is perceptibly laggy as you move across the screen. Most things on the computer needs at least 60fps to feel smooth. On the same 2014 MBP 15" i7 2.5Ghz, this is the result:

~/Documents/workspace/unicode-fonts
❯ time python -c "exit()"

real	0m0.208s
user	0m0.069s
sys	0m0.090s

~/Documents/workspace/unicode-fonts
❯ time isort ./find_fonts 

real	0m0.260s
user	0m0.203s
sys	0m0.049s

The file has 1463 lines.

60 fps requires isort to finish in about 0.01s and leave 0.004s for the editor to make the request and respond to the response. My isort config only has profile="black" in it. The result above shows 5 frames is required just to rearrange some imports, but starting the interpreter and processing the config effectively lowered formatting opts to only around 4 per second. That's 4 fps. I imagine just formatting can be cut down to perhaps 2-3 frames on newer machines, which is acceptable.

[wyuenho]

As an aside, on my MBP 2020 13" 2GHz machine, there's a file where there are only 204 lines and 9 imports in it. Using the default configuration took 0.4s to format. Yes isort is slow, and there are possibly more than a few ways to improve perf, this is one of them.

[timothycrosley]

I personally, don't think FPS is a reasonable, or even beneficial way to think about time for code formatters. The goal shouldn't be to fool a brain into seeing motion after all... Half a second, honestly, in my opinion is still fast. A very fast human response for any stimuli is around 200ms, anything beyond that and I'm not really sure it's really perceptible unless you are intentionally looking. To me the likely bigger issue is when you combine multiple formatters the fractions of a seconds here and there can add up to multiple seconds, and you actually have a real sense of delay. For that I think a language server is likely the best approach.

[timothycrosley]

Doing some more research, I think in general for actions a user takes, the general consensus is

• <= 100ms is perceived as instant by most if not all people
• <1 second is fast, and doesn't actually impact productivity or actions in any meaningful way
• => 2 < 10 seconds is slow, but will still keep someones attention

See: https://www.nngroup.com/articles/response-times-3-important-limits/

I would like isort to be instant of course, but am okay with it living somewhere between instant and fast.

[wyuenho]

I personally, don't think FPS is a reasonable, or even beneficial way to think about time for code formatters. The goal shouldn't be to fool a brain into seeing motion after all

I beg to differ. Text editors are real time applications, from the moment the user issued the key combo to format source code, to the moment the formatted is seen formatted, the user is blocked from further changes to the source code. 0.4s by default just to rearrange 9 imports on a 2020 machine is unacceptable. The train of thought might not be lost, but the work flow is definitely interrupted.

A very fast human response for any stimuli is around 200ms, anything beyond that and I'm not really sure it's really perceptible unless you are intentionally looking.

Humans can see images in as little as 13ms, that's why 60 fps is the target every video game is trying to hit. Seeing your source code reformatted is equivalent to seeing a new image, seeing it changed is thus perceived as motion, which is required for the user to respond to. That's where the 200ms limit comes in. The time includes receiving stimuli, processing and physically responding. We are talking about 200ms to 500ms for laptops produced in the last 7 years just for isort to produce a stimulus.

For that I think a language server is likely the best approach.

All python languages servers are incredibly slow. Pylance is probably the fastest, but it's not available on editors other than VSCode, and it does not have any sort of server to format source code, it just calls the formatter CLI in a shell like emacs and vim do.

[disrupted]

I'm starting to wonder if the basis of this issue is fully accurate:

Starting the python process takes forever, but once it's started, formatting is acceptably fast.

tim@pop-os:~/Projects/isort$ time isort . --show-config
real    0m0.111s
user    0m0.095s
sys     0m0.016s

That includes all the setup time for isort (loading configs etc) that could be saved by a daemon. Sure some projects have larger configs, but I'm kinda curious is one tenth of a second really slow? What is an acceptable delay? Is it longer for others? Mostly, I ask, because if this is a goal it will be important to quantify what fast enough is, as that will greatly impact the design of any solution. I also ask, because it's possible, that there is something wrong with individual config options, or similar, causing some users to experience a long delay that is not widely experienced - and if this is the case it may make sense for isort to fix those performance issues separately regardless.

In my experience isort takes much longer to start. When I started using isortd (unofficial) I did a small comparison for myself and the difference was astonishing.

I understand this is very personal and non-scientific but the difference for me was 440ms (isort) vs. 20ms (isortd) when running it on a specific file. I also posted about it here: urm8/isortd#2

I basically tested it like this:

hyperfine 'cat subclean/core/parser.py | isort --profile black --stdout -'

# start the daemon:
python isortd
# then in a separate shell:
hyperfine 'cat subclean/core/parser.py | curl -s -X POST "localhost:47393" -H "XX-SRC: $(pwd)" --data-binary @-'

To me this shows the huge potential and having isort run as a daemon especially when you're calling it very frequently through an editor integration (format on save) just makes sense imo.