Add support for webdav and Hetzner storage boxes #162
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement WebDAV filesystem to support HTTP-based file operations including: - WebDAV filesystem class with PROPFIND, MKCOL, and standard HTTP methods - Authentication via Basic Auth with username/password secrets - File globbing support using PROPFIND for directory listing - Pattern matching for wildcards (*, [..], **) similar to S3 - Support for storagebox:// protocol for Hetzner Storage Box - Custom HTTP method support (PROPFIND) in both curl and httplib clients - Comprehensive secret management for WebDAV credentials The implementation follows S3's globbing pattern, providing consistent behavior across different storage backends. Custom HTTP methods are supported by extending both HTTP clients to check for X-DuckDB-HTTP-Method in extra_headers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add comprehensive test suite for WebDAV filesystem functionality using bytemark/webdav Docker container. The test infrastructure includes: - Docker Compose configuration with separate setup container - Scripts to start, stop, and configure the test server - Comprehensive test suite with 86 assertions covering: - Authentication with username/password - Reading text and CSV files - Globbing patterns (*.csv, **/*.csv) - Hive-style partitioning - Write operations - Complete documentation with troubleshooting guide The test server runs on localhost:9100 with ephemeral storage, eliminating the need for cleanup between test runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Author
|
I tried to initially look for CONTRIBUTING.md and if it prohibits LLM usage. Sorry I didn't notice the CONTRIBUTING.md which probably also applies to this extension. I will close this PR now and look for making a community plugin instead. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey!
I really could use webdav support and prepared this PR together with Claude. Hopefully you don't see this as spam but as something that would expand usage of DuckDB to new places.
I tried to follow the same integration testing patterns as is currently used with S3 and minio.
I'm not super happy on how it currently replaces the http method based on the special header and would be happy to get feedback if there would be a different better way.
Thanks for duckdb and hopefully this is useful addition. Fixes #160.
I tested it and it worked wonderfully with parquet files in Hetzner storage boxes and because WebDAV supports the http range requests it's much better than using Samba or SFTP.