Performance differences between Net::HTTP and Async::HTTP w/ Threads or Fibers.

[flavio-b]

First of all, thank you for making these gems in the Async family. It's one of the most exciting developments in Ruby 3, in my opinion.

I've been working to replace threads, in several IO-heavy tasks, with Async. For example, downloading up to 500 images in parallel, and saving them to a tempfile:

Sync do |parent|
  urls.each do |url|
    parent.async { download_and_save(url) }
  end
end

The threaded version of this took 0.7s with 500 threads (as an experiment), and 2.5s with 12 threads as we cap to limit resource usage in prod.

download_and_save was written with Net::HTTP. I almost went back to threads because the download time increased to 13s in the Async version. Still better than doing it serially (55s), but much slower than threads.

Finally, I updated download_and_save to use Async::HTTP, and it managed in ~0.6s, with the advantage of consuming less memory and avoiding context switching! I was sold.

This was surprising since I always assumed the Fiber scheduler would work about the same with any library that does IO: that it could capture most blocking operations and pass work to other fibers. But, it sounds like either Net::HTTP has some blocking operations that can't be "seen" by the scheduler, or the HTTP2 aspect of Async::HTTP is making up the difference.

Can you share any comment on this? It look me a while to even think of trying Async::HTTP because I didn't know it would make such a difference.

[ioquatix]

I could not reproduce your results.

Here is a benchmark comparing the 3 scenarios:

      describe Async::HTTP::Client with Thread
        describe Async::HTTP::Client with Thread measure Net::HTTP benchmark/async/http/client.rb:30
          ℹ 26 samples, mean: 216.94ms, standard deviation: 11.02ms, standard error: 2.16ms
      describe Async::HTTP::Client with Async
        describe Async::HTTP::Client with Async measure Net::HTTP benchmark/async/http/client.rb:52
          ℹ 29 samples, mean: 191.85ms, standard deviation: 10.43ms, standard error: 1.94ms
        describe Async::HTTP::Client with Async measure Async::HTTP benchmark/async/http/client.rb:72
          ℹ 18 samples, mean: 168.19ms, standard deviation: 7.03ms, standard error: 1.66ms

Essentially, the performance looks about what I'd expect. Threads are okay, Net::HTTP with Async is better, and Async::HTTP is the best.

Probably the main reason why Async:HTTP has an edge is persistent connections (and connection pooling/multiplexing).

Can you figure out what you are doing that deviates from my benchmark?

https://github.com/socketry/async-http/blob/main/benchmark/async/http/client.rb

You can run the benchmark using bundle exec sus benchmark/async/http/client.rb.

[flavio-b]

Thanks! I cloned the repo and ran bundle install, but when I try to run your benchmark I get LoadError: cannot load such file -- sus/fixtures/benchmark.

In any case, I think it's indeed the multiplexing advantage of HTTP 2, now that I read more about it. All these image requests are to the same CDN host (Shopify's), so Async::HTTP is able to reuse the client for all 500 requests.

I did another quick and dirty test in the middle of the codebase and got a very different result than my first try. I don't know exactly what changed (there have been a few modifications since Friday, and I'm at a different location today), but the new results are more in line with what I'd expect:

Async::HTTP (semaphore with 32 tasks) -> ~0.8s
Async::HTTP (semaphore with 500 tasks) -> ~0.55s
Faraday -> ~2.3s
Net::HTTP/URI -> ~2.75s
12 Threads -> ~2.5s
500 Threads -> ~0.55s

I'm very happy with these and will continue adding more Async in the codebase.

If you don't mind clarifying: How do you "choose" the maximum number of tasks to allow? Do you have any heuristics for that? With Threads, for these specific IO-heavy blocks, I'd normally start with 2 x 3 number of cores and benchmark around that number until an acceptable result. I don't know if with Fibers, the majority of the factor is with the CPU raw core speed, core count, the network interface contention, or the remote host, for example.

[ioquatix]

How do you "choose" the maximum number of tasks to allow?

A good default strategy is to use Async::Idler (as the semaphore). It's going to be a little more conservative than just rapid fire everything, but it will balance load better between different parts of your system.

Thanks! I cloned the repo and ran bundle install, but when I try to run your benchmark I get LoadError: cannot load such file -- sus/fixtures/benchmark.

Should be fixed on main - forgot to commit gems.rb :)

[flavio-b]

Awesome, thanks for the recommendation. I'll give Idler a try, but it seems like it's only looking at CPU utilization. I was wondering about overloading the network interface device, or something like that, but I'm not sure even if that makes sense as concern or if CPU load is a good enough proxy for that.