Rustification - Web Library

[TheTechromancer]

Requirements

• An async client with the following functions:
  • .request() - a one-off request similar to requests
  • .request_batch() - a dedicated pool that, given a list of URLS to visit, yields responses as they happen
  • .request_custom_batch() - similar to request_batch() but each URL can have custom args/kwargs
  • .download() - Download to a local file

Ideally:

• Client is configurable via rate limits, proxy, custom headers, debug logging, etc.
• Benchmarked faster than httpx (especially for request_batch())

TODO

• Research rust web libraries (benchmark performance)
  • Must be async, preferably low enough level to do nasty things
  • Top candidate so far: https://github.com/hyperium/hyper

Basic prototype here:
https://github.com/blacklanternsecurity/rustweb

We could learn a lot from go-retryablehttp, which is the library at the bottom of ProjectDiscovery's tools, including httpx.

• https://github.com/hashicorp/go-retryablehttp

[liquidsec]

merging with #1569

[TheTechromancer]

Neat: https://github.com/lightpanda-io/browser

[liquidsec]

Work has started on this. Will use this comment to document some important features/advantages that have been completed:

• Batch connection reuse
• Rust performance - async I/O,native concurrency etc give significant speed improvements over pure Python HTTP clients
• Direct python bindings - not just shipping back a JSON like project discovery httpx does
• SSL cert info on every request - extracts CN, SANs, and issuer during the TLS handshake that's already happening, eliminating the need for a separate sslcert connection. The sslcert module will be able to just passively monitor for this instead of making separate requests (massive performance savings)
• All TLS ciphers available by default - custom-compiled OpenSSL 3.3.2 with legacy provider baked in (RC4, 3DES, export ciphers, SSLv3) so you can connect to anything
• No cert validation by default - offensive-first: connects to self-signed, expired, and misconfigured TLS without extra config
• HTTP/2 support - automatic via ALPN negotiation, falls back to HTTP/1.1
• Response hashing built-in - MD5, SHA256, and MurmurHash3 computed in Rust for both body and headers. BBOT uses these on the python side, but does the hashing it self. This will happen in rust now which is faster.
• rate limiting mimicing https://github.com/hashicorp/go-retryablehttp algorithm

[liquidsec]

https://github.com/blacklanternsecurity/blasthttp

[liquidsec]

implemented in #2992