Add Web API support for target data transfer#128
Merged
Conversation
Implement transfer_data_from_uploader_via_webapi function to download target data via Web API as an alternative to rsync. This enables data transfer from environments without SSH access. Key features: - Download and extract ZIP archives from Web API endpoints - Support Bearer token authentication (optional) - SSL certificate verification control (verify_ssl config option) - Automatic detection and skipping of existing directories - Proper handling of empty/invalid upload_ids - Streaming downloads for large files - Comprehensive error handling (HTTP, network, ZIP errors) - Status reporting compatible with rsync version Changes: - Add requests dependency to pyproject.toml - Implement transfer_data_from_uploader_via_webapi in utils.py - Add transfer-targets-api CLI command - Update both transfer functions to create local_dir if not exists - Improve status reporting to distinguish skipped vs failed transfers - Add [webapi] configuration section documentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Break down the main function into smaller helper functions to improve readability and reduce cyclomatic complexity (from 22 to ~7). Changes: - Extract _setup_webapi_config() for configuration initialization - Extract _check_existing_directory() for directory checking logic - Extract _download_zip_from_api() for ZIP download - Extract _extract_and_validate_zip() for extraction and validation - Extract _process_single_upload() to orchestrate single upload processing - Main function now only handles setup, iteration, and status reporting This addresses Codacy complexity warning (MC0001) while maintaining identical functionality. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Configure markdownlint to allow patterns common in technical and auto-generated documentation: - MD013: Allow long lines for CLI docs and URLs - MD014: Allow dollar signs in shell commands - MD033: Allow inline HTML for documentation formatting - MD034: Allow bare URLs - MD041: Don't require first line to be a heading These settings accommodate auto-generated CLI documentation from typer and standard technical documentation practices. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add Web API support for downloading target data as an alternative to rsync-based transfer. This enables data transfer from environments without SSH access by downloading ZIP archives via HTTP/HTTPS.
Changes
New function:
transfer_data_from_uploader_via_webapi()inutils.pyNew CLI command:
transfer-targets-apitransfer-targetsfunctionalityEnhanced error handling:
Configuration: New
[webapi]section in config fileImprovements to both transfer functions:
local_dirif it doesn't existTesting
Tested with production Web API endpoint (
pfs-etc.subaru.nao.ac.jp) with:All transfers completed successfully with proper status reporting.
Documentation
[webapi]configuration section examples🤖 Generated with Claude Code