Is httpfs support http connection pool？

[naah69]

We use DuckDB and httpfs in json query from s3.
But we found lots of http connection is TIME_WAIT.
We would like to know is there http connection pool in httpfs or any parameter to set keepalive?

root@label-8cfccb645-268xb:/app# netstat -apto | grep 100.125.160.3 | grep WAIT | wc -l
5331

[dentiny]

I'm curious if any one has any idea if this feature could be implemented as an extension?

[resdevd]

There is an extension that may help -> https://github.com/dentiny/duck-read-cache-fs . also try setting something like "SET threads TO 8;" (depending on your cpu)

[dentiny]

I took a quick look (could be wrong, I will revisit later), I think duckdb already supports basic TCP connection pool at FileHandle level, take S3 GET request as an example, the call chain looks like:

• S3 filesystem:

  duckdb-httpfs/extension/httpfs/s3fs.cpp

  Line 710 in cb5b282

  return HTTPFileSystem::GetRequest(handle, http_url, headers);

• HTTP filesystem get http client:

  duckdb-httpfs/extension/httpfs/httpfs.cpp

  Line 160 in cb5b282

  auto http_client = hfh.GetClient();

• which gets a cached http client, or create a new one:

  duckdb-httpfs/extension/httpfs/httpfs.cpp

  Lines 705 to 721 in cb5b282

  	unique_ptr<HTTPClient> HTTPFileHandle::GetClient() {
  	// Try to fetch a cached client
  	auto cached_client = client_cache.GetClient();
  	if (cached_client) {
  	return cached_client;
  	}
  	// Create a new client
  	return CreateClient();
  	}
  	unique_ptr<HTTPClient> HTTPFileHandle::CreateClient() {
  	// Create a new client
  	string path_out, proto_host_port;
  	HTTPUtil::DecomposeURL(path, path_out, proto_host_port);
  	return http_params.http_util.InitializeClient(http_params, proto_host_port);
  	}

Let's take another look at HttpLibClient creation:

• here we create a client: https://github.com/duckdb/duckdb/blob/3de8d76ba83a82fb4337ca7cb8e4de2eab748561/src/main/http/http_util.cpp#L135
• internally it's a wrapper for TCP connection: https://github.com/duckdb/duckdb/blob/3de8d76ba83a82fb4337ca7cb8e4de2eab748561/third_party/httplib/httplib.hpp#L1098

I think there're two places we could improve:

• Make HTTPClientCache sharable by all HTTPFileHandle if possible, meanwhile, need to set expiration time on cached clients
• Implement real IO multiplexing (HTTP/2), current HTTP client works on blocking style:
  • each HTTP request translates to send_ call: https://github.com/duckdb/duckdb/blob/3de8d76ba83a82fb4337ca7cb8e4de2eab748561/third_party/httplib/httplib.hpp#L7592
  • which internally takes a lock: https://github.com/duckdb/duckdb/blob/3de8d76ba83a82fb4337ca7cb8e4de2eab748561/third_party/httplib/httplib.hpp#L6891-L6978, so single TCP connection only serves one request concurrently
  • httplib doesn't have plan to support IO multiplexing: Can you consider adding the implementation of Http2 yhirose/cpp-httplib#666

[dentiny]

There is an extension that may help -> https://github.com/dentiny/duck-read-cache-fs

Thanks for the mention! I'm the author for the extension. :)
One thing could help is to reduce IO access via caching, I haven't any connection pool in the extension (yet).

[dentiny]

There is an extension that may help -> https://github.com/dentiny/duck-read-cache-fs . also try setting something like "SET threads TO 8;" (depending on your cpu)

I didn't implement the connection pool directly in the extension, it's actually httplib / httpfs's responsibility.
Since httpfs does connection pooling at file handle level, and cache-httpfs does cache file handle for read operations,
one workaround with cache-httpfs is to increase the file handle cache size.

[naah69]

@carlopi hello,what do you think about this topic?

[dentiny]

@carlopi hello,what do you think about this topic?

I think duckdb team's moving towards libcurl based solution, which could easily setup connection pool via params like CURLMOPT_MAX_HOST_CONNECTIONS.

[naah69]

@carlopi hello,what do you think about this topic?

I think duckdb team's moving towards libcurl based solution, which could easily setup connection pool via params like CURLMOPT_MAX_HOST_CONNECTIONS.

Okey,this Is a great progress too.
I found that the pr had been merged in 1.4. #96
Do you know when will release 1.4?

[dentiny]

Okey,this Is a great progress too. I found that the pr had been merged in 1.4. #96 Do you know when will release 1.4?

Pretty soon: https://duckdb.org/release_calendar.html

[naah69]

https://duckdb.org/release_calendar.html

👍🏻