Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
da34fcf
fix: adopt robocodec v0.3 breaking API changes
zhexuany Feb 3, 2026
f6f59c9
feat: add async storage trait with security improvements
zhexuany Feb 3, 2026
f528bb8
fix: handle Result from LocalStorage::full_path after path traversal fix
zhexuany Feb 3, 2026
5fae750
fix: correct path validation to avoid false positives on valid local …
zhexuany Feb 3, 2026
65ed834
feat: integrate image decoder into streaming converter
zhexuany Feb 3, 2026
179a0cb
fix: extract S3 object key from cloud URLs in distributed conversion
zhexuany Feb 3, 2026
11a570b
refactor: make feature naming config-driven and fix image decoding
zhexuany Feb 4, 2026
de6d38b
chore: add docker-compose targets to Makefile
zhexuany Feb 4, 2026
91a4326
chore: add dev-clean target to remove all containers and volumes
zhexuany Feb 4, 2026
4a047bf
chore: extend dev-clean to remove local test data
zhexuany Feb 4, 2026
3fe9ada
chore: remove robocodec tests migrated to robocodec repo
zhexuany Feb 4, 2026
9f50f58
fix: adopt robocodec API change for RoboReader::open(&str)
zhexuany Feb 4, 2026
07e2b89
chore: add output directory to .gitignore
zhexuany Feb 4, 2026
3294ebf
fix: resolve duplicate definition and import errors in apple.rs
zhexuany Feb 4, 2026
25fe20b
style: apply cargo fmt formatting to apple.rs
zhexuany Feb 4, 2026
9936897
fix: address code review issues from PR #19
zhexuany Feb 4, 2026
986ced6
fix: remove unnecessary unwrap in alignment.rs
zhexuany Feb 4, 2026
bca1fb4
fix: replace range loop with iterator in test
zhexuany Feb 4, 2026
9021397
refactor: unify jobs and batch system
zhexuany Feb 4, 2026
88ebbe9
add all files
zhexuany Feb 4, 2026
c8eb291
test: add comprehensive integration tests for Phase 2
zhexuany Feb 4, 2026
924fd27
refactor: modularize lerobot/writer.rs into smaller modules
zhexuany Feb 4, 2026
8d7f06c
fix: add proper frame data to dataset writer tests
zhexuany Feb 4, 2026
f623a02
fix: correct image module test assertions
zhexuany Feb 4, 2026
02119ed
refactor: unify writer traits and remove dead code
zhexuany Feb 4, 2026
319739f
eliminate tech debt
zhexuany Feb 4, 2026
e054599
refactor: reduce technical debt by 81%
zhexuany Feb 4, 2026
a608186
refactor: make distributed coordination always-enabled
zhexuany Feb 5, 2026
5110cb2
refactor: improve batch ID format and namespace consistency
zhexuany Feb 5, 2026
2f4f38a
fix: correct URL path extraction to skip past :// protocol marker
zhexuany Feb 5, 2026
b0f86ec
fix: store full S3 source URL in job records for distributed workers
zhexuany Feb 5, 2026
d250080
feat: improve cloud storage URL handling and worker tracing
zhexuany Feb 5, 2026
c22ca9e
fix: handle new FileFormat::Rrd variant in hyper pipeline
zhexuany Feb 5, 2026
f6f213a
docs: address Greptile review comments
zhexuany Feb 5, 2026
b4af445
feat: implement reader stage chunk sending using ParallelReader
zhexuany Feb 5, 2026
79426fc
style: apply cargo fmt formatting fixes
zhexuany Feb 5, 2026
6f8cb35
test: handle missing ffmpeg in frame count increment test
zhexuany Feb 5, 2026
802a46a
docs: update README to reflect current architecture
zhexuany Feb 5, 2026
b0a5226
fix: resolve distributed architecture gaps and improve error handling
zhexuany Feb 5, 2026
43b1181
fix fmt
zhexuany Feb 5, 2026
6030dc9
refactor: remove legacy Job system, migrate to WorkUnit-only architec…
zhexuany Feb 6, 2026
35532fb
refactor: remove storage_prefix, use Batch output_path
zhexuany Feb 6, 2026
16ab937
refactor: remove distributed feature flag
zhexuany Feb 6, 2026
0a1ec81
refactor: replace distributed locks with CAS-based merge coordination
zhexuany Feb 6, 2026
cf3e6b9
docs: update README to reflect CAS-based coordination
zhexuany Feb 6, 2026
39a768b
ci: invalidate rust-cache when robocodec git dependency changes
zhexuany Feb 6, 2026
1c4ae3d
ci: fix cache-key syntax (string not array)
zhexuany Feb 6, 2026
0798ce5
fix: track Cargo.lock for reproducible builds
zhexuany Feb 6, 2026
89c7b6f
ci: use system linker instead of lld
zhexuany Feb 6, 2026
574d4d1
fix: properly format help text with newlines
zhexuany Feb 6, 2026
16d8aaa
fix: implement --help flag for run command
zhexuany Feb 6, 2026
4e91e76
refactor: remove Python bindings support
zhexuany Feb 6, 2026
6d4183e
ci: explicitly set system linker via RUSTFLAGS
zhexuany Feb 6, 2026
ed0cc4f
fix: resolve type inference and lint issues
zhexuany Feb 6, 2026
619daca
docs: remove Python references from documentation
zhexuany Feb 6, 2026
013a630
fix: remove flaky assertion from page_aligned_allocation test
zhexuany Feb 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# CI-specific cargo configuration
[build]
# Use system linker instead of lld (lld crashes on large binaries in CI)
[target.x86_64-unknown-linux-gnu]
linker = "cc"
1 change: 0 additions & 1 deletion .claude/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
"nanobanana-skill@claude-code-settings": true,
"spec-kit-skill@claude-code-settings": true,
"code-simplifier@claude-plugins-official": true,
"rust-analyzer-lsp@claude-plugins-official": true,
"engineering-skills@claude-code-skills": true
}
}
117 changes: 13 additions & 104 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ on:

env:
CARGO_TERM_COLOR: always
RUSTFLAGS: -C linker=cc

jobs:
license:
Expand Down Expand Up @@ -38,16 +39,13 @@ jobs:
components: rustfmt, clippy

- uses: Swatinem/rust-cache@v2

- name: Install ruff
run: pip install ruff
with:
# Invalidate cache when robocodec git dependency changes
cache-key: "robocodec-${{ hashFiles('**/Cargo.lock') }}"

- name: Check formatting (Rust)
run: cargo fmt -- --check

- name: Check formatting (Python)
run: ruff format --check python/

- name: Install HDF5
run: |
if [ "$RUNNER_OS" == "Linux" ]; then
Expand Down Expand Up @@ -76,6 +74,9 @@ jobs:
- uses: dtolnay/rust-toolchain@stable

- uses: Swatinem/rust-cache@v2
with:
# Invalidate cache when robocodec git dependency changes
cache-key: "robocodec-${{ hashFiles('**/Cargo.lock') }}"

- name: Install HDF5
run: |
Expand All @@ -88,101 +89,25 @@ jobs:
fi
shell: bash

# Note: Do NOT use --all-features or --features python here.
# PyO3's extension-module feature prevents linking in standalone test binaries.
# Python bindings are tested separately via maturin in the python job.
- name: Run tests
run: cargo test --features dataset-all

python:
name: Python Tests
needs: rust-test
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
python-version: ['3.12']
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- uses: Swatinem/rust-cache@v2

- name: Create virtual environment
run: |
python -m venv .venv
echo "VIRTUAL_ENV=$PWD/.venv" >> $GITHUB_ENV
echo "$PWD/.venv/bin" >> $GITHUB_PATH

- name: Install dependencies
run: pip install maturin[patchelf] pytest pytest-cov

- name: Build and install Python package
run: maturin develop --release --features python

- name: Run Python tests
run: pytest python/ -v

lint:
name: Python Lint
needs: python
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'pip'

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- uses: Swatinem/rust-cache@v2

- name: Create virtual environment
run: |
python -m venv .venv
echo "VIRTUAL_ENV=$PWD/.venv" >> $GITHUB_ENV
echo "$PWD/.venv/bin" >> $GITHUB_PATH

- name: Install dependencies
run: pip install maturin[patchelf] ruff mypy

- name: Build and install Python package
run: maturin develop --release --features python

- name: Run ruff check (Python linting)
run: ruff check python/

- name: Run mypy
run: mypy python/roboflow

coverage:
name: Coverage
needs: lint
needs: rust-test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'pip'

- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
components: llvm-tools-preview

- uses: Swatinem/rust-cache@v2
with:
# Invalidate cache when robocodec git dependency changes
cache-key: "robocodec-${{ hashFiles('**/Cargo.lock') }}"

- name: Cache cargo-llvm-cov
uses: actions/cache@v4
Expand All @@ -195,32 +120,16 @@ jobs:
if: steps.cache-llvm-cov.outputs.cache-hit != 'true'
run: cargo install cargo-llvm-cov

- name: Create virtual environment
run: |
python -m venv .venv
echo "VIRTUAL_ENV=$PWD/.venv" >> $GITHUB_ENV
echo "$PWD/.venv/bin" >> $GITHUB_PATH

- name: Install dependencies
run: pip install maturin[patchelf] pytest pytest-cov

- name: Install HDF5
run: sudo apt-get update && sudo apt-get install -y libhdf5-dev

# Note: Do NOT use --all-features here - PyO3's extension-module prevents linking.
# Rust coverage excludes python feature; Python coverage is generated separately below.
- name: Generate Rust coverage
run: cargo llvm-cov --features dataset-all --workspace --lcov --output-path lcov.info

- name: Generate Python coverage
run: |
maturin develop --release --features python
pytest python/ --cov=roboflow --cov-report=xml:coverage-python.xml --cov-report=term

- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./lcov.info,./coverage-python.xml
flags: rust,python
files: ./lcov.info
flags: rust
name: codecov-umbrella
109 changes: 0 additions & 109 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,123 +7,14 @@ on:

permissions:
contents: read
id-token: write # Required for trusted publishing

jobs:
build:
name: Build wheels
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
target: [x86_64]
include:
- os: ubuntu-latest
target: x86_64
use-zig: true
- os: windows-latest
target: x86_64
- os: macos-latest
target: x86_64
use-zig: true
- os: macos-latest
target: aarch64
use-zig: true
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Set up Rust
uses: dtolnay/rust-toolchain@stable

- name: Install zig for cross-compilation
if: matrix.use-zig
uses: goto-bus-stop/setup-zig@v2

- name: Install maturin
run: pip install maturin[patchelf]
if: runner.os != 'Windows'

- name: Install maturin (Windows)
run: pip install maturin
if: runner.os == 'Windows'

- name: Build wheels
run: |
TARGET_FLAG=""
if [ "${{ runner.os }}" == "macOS" ]; then
if [ "${{ matrix.target }}" == "x86_64" ]; then
TARGET_FLAG="--target x86_64-apple-darwin"
fi
elif [ "${{ runner.os }}" == "Linux" ] && [ "${{ matrix.target }}" == "aarch64" ]; then
TARGET_FLAG="--target aarch64-unknown-linux-gnu"
fi
maturin build --release --strip --features python --out dist --find-interpreter $TARGET_FLAG
env:
# Enable zig compilation for aarch64
PYO3_CROSS: ${{ matrix.use-zig && '1' || '' }}
PYO3_CROSS_LIB_DIR: ${{ matrix.use-zig && '/usr/aarch64-linux-gnu' || '' }}
shell: bash

- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-${{ matrix.os }}-${{ matrix.target }}
path: dist/

build-manylinux:
name: Build manylinux wheels
runs-on: ubuntu-latest
container: ghcr.io/pyo3/maturin:main
steps:
- uses: actions/checkout@v4

- name: Build wheels
run: maturin build --release --strip --features python --out dist --manylinux 2010+

- name: Upload wheels
uses: actions/upload-artifact@v4
with:
name: wheels-manylinux
path: dist/

publish-pypi:
name: Publish to PyPI
needs: [build, build-manylinux]
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
pattern: wheels-*
path: dist/
merge-multiple: true

- name: List built wheels
run: ls -lh dist/

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
packages-dir: dist/
skip-existing: true
verify-metadata: false
verbose: true

publish-crates:
name: Publish to crates.io
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: '3.11'

- uses: dtolnay/rust-toolchain@stable

- name: Publish to crates.io
Expand Down
38 changes: 4 additions & 34 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,7 @@
/target/
**/*.rs.bk
*.pdb
Cargo.lock

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
.pytest_cache/
.coverage
htmlcov/
.venv/
venv/
ENV/
# Cargo.lock is now tracked for reproducible builds

# IDE
.idea/
Expand All @@ -48,8 +20,9 @@ ENV/
*.log
tests/output/

# ignore pyo3
python/robocodec-0.1.0.data/
# Dataset output
/output/
*.toml.bak

# ignore flamegraph
flamegraph.svg
Expand All @@ -58,6 +31,3 @@ profiles/
# robot data
*.bag
*.mcap

# claude
.claude/
Loading
Loading