Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 6 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,12 @@ version.workspace = true
edition.workspace = true
authors.workspace = true
license.workspace = true
description = "High-performance robotics data codec library for reading, writing, and converting MCAP and ROS1 bag files with support for CDR, Protobuf, and JSON encodings"
description = "High-performance robotics data codec library for reading, writing, and converting MCAP, ROS1 bag, and RRF2 files"
keywords = ["robotics", "mcap", "ros", "rosbag", "codec"]
categories = ["parser-implementations", "encoding", "science::robotics"]
readme = "README.md"
repository = "https://github.com/archebase/robocodec"
documentation = "https://docs.rs/robocodec"

[lib]
crate-type = ["rlib", "cdylib"]
Expand Down Expand Up @@ -57,7 +62,6 @@ bzip2 = "0.4"
crc32fast = "1.4"
mcap = "0.24"
rosbag = "0.6"
bumpalo = "3.16"
bytemuck = "1.15"
chrono = "0.4"
tracing = "0.1"
Expand Down
81 changes: 11 additions & 70 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
- **Auto-Detection** - Format detected from file extension or URL scheme
- **Fast** - Parallel processing with rayon, zero-copy memory-mapped files
- **S3-Native** - First-class support for `s3://` URLs (AWS S3, MinIO, Alibaba OSS, etc.)
- **Transformations** - Topic/type renaming and format conversion built-in
- **Transformations** - Topic/type renaming during file rewriting

## Quick Start

Expand Down Expand Up @@ -122,61 +122,6 @@ export AWS_SECRET_ACCESS_KEY="your-oss-secret-key"

> **Note:** While we use AWS-standard environment variable names for compatibility, robocodec works with any S3-compatible storage service.

### Read from HTTP/HTTPS

Robocodec also supports reading directly from HTTP/HTTPS URLs:

```rust
use robocodec::RoboReader;

// Format detected from URL path, access via HTTP
let reader = RoboReader::open("https://example.com/data.mcap")?;
println!("Found {} channels", reader.channels().len());
```

> **Note:** HTTP reading supports range requests for efficient access to large files.

#### HTTP Authentication

For authenticated HTTP endpoints, robocodec supports Bearer tokens and Basic authentication via `ReaderConfig`:

```rust
use robocodec::io::{RoboReader, ReaderConfig};

// Bearer token (OAuth2/JWT)
let config = ReaderConfig::default().with_http_bearer_token("your-token-here");
let reader = RoboReader::open_with_config("https://example.com/data.mcap", config)?;

// Basic authentication
let config = ReaderConfig::default().with_http_basic_auth("username", "password");
let reader = RoboReader::open_with_config("https://example.com/data.mcap", config)?;
```

Alternatively, you can provide authentication via URL query parameters:

```rust
use robocodec::RoboReader;

// Bearer token via URL
let reader = RoboReader::open("https://example.com/data.mcap?bearer_token=your-token")?;

// Basic auth via URL (user:pass encoded)
let reader = RoboReader::open("https://example.com/data.mcap?basic_auth=user:pass")?;
```

### Write to S3

```rust
use robocodec::RoboWriter;

// Format detected from .mcap extension, S3 from s3:// URL
let mut writer = RoboWriter::create("s3://my-bucket/output.mcap")?;
let channel_id = writer.add_channel("/topic", "MessageType", "cdr", None)?;
// ... write messages ...
writer.finish()?;
}
```

### Custom S3 endpoints

For S3-compatible services with custom endpoints:
Expand All @@ -200,31 +145,25 @@ let reader = RoboReader::open(
)?;
```

### Convert between formats
### Rewrite files with transformations

```rust
use robocodec::RoboRewriter;

let rewriter = RoboRewriter::open("input.bag")?;
rewriter.rewrite("output.mcap")?;
```

### Rename topics during conversion
The rewriter processes files in the same format, optionally applying topic and type transformations:

```rust
use robocodec::{RoboRewriter, TransformBuilder};
use robocodec::{RoboRewriter, TransformBuilder, RewriteOptions};

let transform = TransformBuilder::new()
.with_topic_rename("/old/topic", "/new/topic")
.build();

let rewriter = RoboRewriter::with_options(
"input.mcap",
robocodec::RewriteOptions::default().with_transforms(transform)
)?;
let options = RewriteOptions::default().with_transforms(transform);
let rewriter = RoboRewriter::with_options("input.mcap", options)?;
rewriter.rewrite("output.mcap")?;
```

**Note:** The rewriter preserves the same format. Cross-format conversion is not currently supported.


## Installation

### Rust Users
Expand All @@ -247,6 +186,8 @@ robocodec = { version = "0.1", features = ["jemalloc"] }
| `s3` | S3-compatible storage support (AWS S3, MinIO, etc.) | ✅ Yes |
| `python` | Python bindings | ❌ No |
| `jemalloc` | Use jemalloc allocator (Linux only) | ❌ No |
| `python` | Python bindings | ❌ No |
| `jemalloc` | Use jemalloc allocator (Linux only) | ❌ No |

### Python Users

Expand Down
77 changes: 8 additions & 69 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
- **自动检测** - 从文件扩展名或 URL scheme 自动检测格式
- **高性能** - 使用 rayon 并行处理,零拷贝内存映射文件
- **原生 S3 支持** - 对 `s3://` URL 的一流支持(AWS S3、MinIO、阿里云 OSS 等)
- **数据转换** - 内置主题/类型重命名和格式转换功能
- **数据转换** - 内置主题/类型重命名和文件重写功能

## 快速开始

Expand Down Expand Up @@ -124,60 +124,6 @@ export AWS_SECRET_ACCESS_KEY="your-oss-secret-key"

> **注意:** 虽然我们使用 AWS 标准的环境变量名称以确保兼容性,但 robocodec 可与任何兼容 S3 的存储服务配合使用。

### 从 HTTP/HTTPS 读取

Robocodec 也支持直接从 HTTP/HTTPS URL 读取数据:

```rust
use robocodec::RoboReader;

// 格式从 URL 路径检测,通过 HTTP 访问
let reader = RoboReader::open("https://example.com/data.mcap")?;
println!("找到 {} 个通道", reader.channels().len());
```

> **注意:** HTTP 读取支持范围请求,可高效访问大文件。

#### HTTP 身份验证

对于需要身份验证的 HTTP 端点,robocodec 通过 `ReaderConfig` 支持 Bearer 令牌和基本身份验证:

```rust
use robocodec::io::{RoboReader, ReaderConfig};

// Bearer 令牌(OAuth2/JWT)
let config = ReaderConfig::default().with_http_bearer_token("your-token-here");
let reader = RoboReader::open_with_config("https://example.com/data.mcap", config)?;

// 基本身份验证
let config = ReaderConfig::default().with_http_basic_auth("username", "password");
let reader = RoboReader::open_with_config("https://example.com/data.mcap", config)?;
```

或者,您可以通过 URL 查询参数提供身份验证:

```rust
use robocodec::RoboReader;

// 通过 URL 提供 Bearer 令牌
let reader = RoboReader::open("https://example.com/data.mcap?bearer_token=your-token")?;

// 通过 URL 提供基本身份验证(user:pass 编码)
let reader = RoboReader::open("https://example.com/data.mcap?basic_auth=user:pass")?;
```

### 写入到 S3

```rust
use robocodec::RoboWriter;

// 格式从 .mcap 扩展名检测,S3 从 s3:// URL 检测
let mut writer = RoboWriter::create("s3://my-bucket/output.mcap")?;
let channel_id = writer.add_channel("/topic", "MessageType", "cdr", None)?;
// ... 写入消息 ...
writer.finish()?;
```

### 自定义 S3 端点

对于具有自定义端点的兼容 S3 服务:
Expand All @@ -201,31 +147,24 @@ let reader = RoboReader::open(
)?;
```

### 格式之间转换
### 使用转换重写文件

```rust
use robocodec::RoboRewriter;

let rewriter = RoboRewriter::open("input.bag")?;
rewriter.rewrite("output.mcap")?;
```

### 转换时重命名主题
重写器以相同格式处理文件,可选择应用主题和类型转换:

```rust
use robocodec::{RoboRewriter, TransformBuilder};
use robocodec::{RoboRewriter, TransformBuilder, RewriteOptions};

let transform = TransformBuilder::new()
.with_topic_rename("/old/topic", "/new/topic")
.build();

let rewriter = RoboRewriter::with_options(
"input.mcap",
robocodec::RewriteOptions::default().with_transforms(transform)
)?;
let options = RewriteOptions::default().with_transforms(transform);
let rewriter = RoboRewriter::with_options("input.mcap", options)?;
rewriter.rewrite("output.mcap")?;
```

**注意:** 重写器保持相同的格式。目前不支持跨格式转换。

## 安装

### Rust 用户
Expand Down
24 changes: 12 additions & 12 deletions benches/decoder_bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -180,25 +180,25 @@ fn bench_value_operations(c: &mut Criterion) {

// Benchmark string value access
group.bench_function("access_string_value", |b| {
let value = robocodec::CodecValue::String("test string value".to_string());
let value = robocodec::core::CodecValue::String("test string value".to_string());
b.iter(|| {
if let robocodec::CodecValue::String(s) = black_box(&value) {
if let robocodec::core::CodecValue::String(s) = black_box(&value) {
black_box(s.len());
}
})
});

// Benchmark array value access
group.bench_function("access_array_value", |b| {
let value = robocodec::CodecValue::Array(vec![
robocodec::CodecValue::Int64(1),
robocodec::CodecValue::Int64(2),
robocodec::CodecValue::Int64(3),
robocodec::CodecValue::Int64(4),
robocodec::CodecValue::Int64(5),
let value = robocodec::core::CodecValue::Array(vec![
robocodec::core::CodecValue::Int64(1),
robocodec::core::CodecValue::Int64(2),
robocodec::core::CodecValue::Int64(3),
robocodec::core::CodecValue::Int64(4),
robocodec::core::CodecValue::Int64(5),
]);
b.iter(|| {
if let robocodec::CodecValue::Array(arr) = black_box(&value) {
if let robocodec::core::CodecValue::Array(arr) = black_box(&value) {
black_box(arr.len());
}
})
Expand All @@ -210,13 +210,13 @@ fn bench_value_operations(c: &mut Criterion) {
for i in 0..10 {
fields.insert(
format!("field_{}", i),
robocodec::CodecValue::Int64(i as i64),
robocodec::core::CodecValue::Int64(i as i64),
);
}
let value = robocodec::CodecValue::Struct(fields);
let value = robocodec::core::CodecValue::Struct(fields);

b.iter(|| {
if let robocodec::CodecValue::Struct(fields) = black_box(&value) {
if let robocodec::core::CodecValue::Struct(fields) = black_box(&value) {
black_box(fields.len());
}
})
Expand Down
52 changes: 45 additions & 7 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This directory contains practical examples demonstrating the public API of the r

## Running Examples

Each example accepts a file path as an argument:
### Local Files

```bash
# Inspect a file
Expand All @@ -13,8 +13,25 @@ cargo run --example read_file -- tests/fixtures/robocodec_test_14.mcap
# Decode and display messages
cargo run --example decode_messages -- tests/fixtures/robocodec_test_14.mcap

# Convert between formats
cargo run --example convert_format -- tests/fixtures/robocodec_test_14.mcap output.bag
# Rewrite a file (same format)
cargo run --example convert_format -- tests/fixtures/robocodec_test_14.mcap output.mcap

# Transform topics and types
cargo run --example transform -- tests/fixtures/robocodec_test_14.mcap output.mcap
```

### Remote Files (S3)

```bash
# Set S3 credentials (for AWS S3, MinIO, Alibaba OSS, etc.)
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

# Read from S3
cargo run --example s3_example -- s3://my-bucket/path/to/data.mcap

# Read from S3 with custom endpoint (MinIO, Alibaba OSS, etc.)
cargo run --example s3_example -- "s3://bucket/data.mcap?endpoint=http://localhost:9000"
```

## Examples
Expand All @@ -28,13 +45,15 @@ Demonstrates opening a robotics data file and inspecting its metadata, channels,
- Accessing file metadata
- Listing channels with their properties

### `convert_format.rs` - Format Conversion
### `convert_format.rs` - File Rewriting

Demonstrates converting between MCAP and ROS1 bag formats.
Demonstrates rewriting a robotics data file in the same format.

**What you'll learn:**
- Using `RoboRewriter` for format conversion
- Understanding conversion statistics
- Using `RoboRewriter` to rewrite files
- Understanding rewrite statistics

**Note:** The rewriter preserves the same format as the input file. Cross-format conversion is not currently supported.

### `decode_messages.rs` - Message Decoding

Expand All @@ -44,6 +63,25 @@ Demonstrates iterating through decoded messages with timestamps.
- Using the `decoded()` iterator
- Accessing message data, timestamps, and channel info

### `s3_example.rs` - S3 Remote File Access

Demonstrates reading robotics data files from S3-compatible storage.

**What you'll learn:**
- Reading from S3-compatible storage (AWS S3, MinIO, Alibaba OSS, etc.)
- S3 authentication via environment variables
- Custom S3 endpoints via URL parameters

### `transform.rs` - Topic and Type Transformations

Demonstrates renaming topics and message types during format conversion.

**What you'll learn:**
- Using `TransformBuilder` to create transformation pipelines
- Topic renaming with exact name matching
- Type renaming for schema migration
- Combining transformations with `RoboRewriter`

## Public API

The examples demonstrate the **public API** only:
Expand Down
Loading