-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
priority/lowLow priority, nice to haveLow priority, nice to havesize/MMedium: 3-5 daysMedium: 3-5 daystype/refactorCode refactoring without behavior changeCode refactoring without behavior change
Description
Summary
Track code cleanup tasks after the distributed architecture is fully implemented.
Parent Epic
- [Epic] Distributed Roboflow with Alibaba Cloud (OSS + ACK) #9 Distributed Roboflow with TiKV Coordination
When to Execute
After all distributed features are complete and tested (Phase 10+)
Code to Remove
1. TiKV Backup Directory
- Delete
crates/roboflow-distributed/src/tikv_backup/(duplicate oftikv/)
Code to Refactor
2. OssStorage Full-Object Buffering
-
crates/roboflow-storage/src/oss.rslines 213-239 - Replace
bytes().await.to_vec()with streaming reader - Depends on: [Phase 6.1] Add streaming S3 reader with range requests #45 (Streaming S3 reader)
3. Sequential Multipart Uploads
-
crates/roboflow-storage/src/multipart.rslines 281-337 - Implement parallel part uploads
- Depends on: [Phase 6.2] Add parallel multipart uploads #46 (Parallel multipart uploads)
4. Sequential Batch Processing
-
crates/roboflow-pipeline/src/fluent/builder.rslines 439-456 - Replace sequential loop with TiKV job integration
- Depends on: [Phase 4.4] Implement Worker loop with job claiming #43 (Worker loop)
5. Sequential Reader Usage
-
crates/roboflow-pipeline/src/orchestrator/orchestrator.rslines 398-403 - Enable parallel readers
Code to Deprecate
6. Python Examples
- Add deprecation warning to
examples/python/roboflow_utils.py::BatchProcessor - Add deprecation notice to
examples/python/batch_conversion.py - Document migration path to distributed CLI
TODOs to Address
-
crates/roboflow-pipeline/src/fluent/builder.rsline 397 - Standard pipeline -
crates/roboflow-pipeline/src/fluent/builder.rslines 543, 596, 609, 657, 672, 690 -
examples/rust/lerobot_convert.rslines 51, 73, 92 -
tests/kps_integration_tests.rsline 18
Tests to Update
-
tests/sequential_parallel_comparison_tests.rs- Add distributed tests -
tests/pipeline_round_trip_tests.rs- Add job processing tests -
tests/streaming_converter_tests.rs- Add distributed streaming tests
Acceptance Criteria
- No duplicate code remains
- Full-object buffering replaced with streaming
- Sequential uploads replaced with parallel
- Batch processing uses TiKV queue
- Deprecated code has warnings
- All TODOs addressed or tracked
- Tests updated for distributed architecture
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
priority/lowLow priority, nice to haveLow priority, nice to havesize/MMedium: 3-5 daysMedium: 3-5 daystype/refactorCode refactoring without behavior changeCode refactoring without behavior change