Implement MCAP Logging

[EthanMBoos]

Why MCAP?

The current logging writes raw protobuf to 4 separate .bin files with no compression.

The Numbers

I ran a test mission with 10, 50, and 100 entities for 50 sim-seconds at dt=0.1 (500 frames):

Entities	frames.bin	Bytes/entity/frame
10	341 KB	68.2
50	1.6 MB	67.3
100	3.3 MB	67.2

Reproduce These Numbers

source ~/.scrimmage/setup.bash

# 10 entities
sed -i 's/end="100"/end="50"/' missions/straight-no-gui.xml
sed -i 's/<count>1<\/count>/<count>5<\/count>/g' missions/straight-no-gui.xml
scrimmage missions/straight-no-gui.xml
ls -lh ~/.scrimmage/logs/latest/frames.bin

# 50 entities
sed -i 's/<count>[0-9]*<\/count>/<count>25<\/count>/g' missions/straight-no-gui.xml
scrimmage missions/straight-no-gui.xml
ls -lh ~/.scrimmage/logs/latest/frames.bin

# 100 entities
sed -i 's/<count>[0-9]*<\/count>/<count>50<\/count>/g' missions/straight-no-gui.xml
scrimmage missions/straight-no-gui.xml
ls -lh ~/.scrimmage/logs/latest/frames.bin

The per-entity cost is consistent: ~68 bytes per entity per frame.

frames.bin size = entities × timesteps × 68 bytes

So a 1000-entity sim running 30 minutes at 10Hz generates 1.2 GB of frame data alone.

What MCAP Gets You

MCAP chunks messages and compresses them with Zstd. Zstd excels at repetitive structured data — exactly what we have (same field tags, similar values, repeated IDs). Based on Zstd benchmarks and typical robotics log compression ratios, expect 5-7x reduction. MCAP is also self-describing (embedded schema, so tools can introspect without our code) and an industry standard — ROS2, Cruise, and others have adopted it.

Scenario	Current	MCAP (est.)	Saved
100 ent, 5 min	19 MB	~4 MB	80%
500 ent, 10 min	194 MB	~40 MB	80%
1000 ent, 30 min	1.2 GB	~240 MB	80%

Better Replay and Plotting in Scrimmage

With MCAP's index, scrimmage-playback can jump to any timestamp without reading the whole file. Plotting scripts don't need to load everything into memory first — they can stream specific time ranges.

The single-file format also makes log management simpler: one file to copy, share, or archive instead of juggling frames.bin, shapes.bin, contact_visual.bin, etc.

Optional: Foxglove Studio

We can also experiment with Foxglove Studio (free desktop app). Drag an .mcap file into the window and you get timeline scrubbing, plotting, 3D visualization, log filtering, and side-by-side comparison of multiple logs.

Scaling Projections

Typical Missions (30 min - 1 hour)

Entities	Duration	frames.bin	MCAP (est.)
100	30 min	117 MB	~23 MB
100	1 hour	234 MB	~47 MB
500	30 min	584 MB	~117 MB
500	1 hour	1.2 GB	~234 MB
1000	30 min	1.2 GB	~234 MB
1000	1 hour	2.3 GB	~470 MB

Long-Running / Large-Scale Missions

Entities	Duration	frames.bin	MCAP (est.)
1000	4 hours	9.4 GB	~1.9 GB
1000	8 hours	18.7 GB	~3.7 GB
2000	4 hours	18.7 GB	~3.7 GB
2000	8 hours	37 GB	~7.5 GB
5000	4 hours	47 GB	~9.4 GB
5000	8 hours	94 GB	~19 GB

All estimates assume 10Hz (dt=0.1) and ~80% compression with MCAP/Zstd.

Inter-Agent Messaging (msgs.bin)

Note: msgs.bin appears in the codebase but is never actually written. The file descriptor and output stream are declared in Log.h, but open_file() is never called for it in Log.cpp. Only frames.bin, shapes.bin, utm_terrain.bin, and contact_visual.bin are opened.

This may have been intentional due to the potential memory footprint — pubsub messages in a large swarm could easily dwarf frames.bin. Not 100% sure why it was left unimplemented.

If message logging were enabled in the future, the scaling concern would be significant.

At ~50 bytes per message, broadcast with 1000 agents for 1 hour would add ~1.8 GB on top of frames.bin. This is another reason MCAP's compression would be valuable if we ever add message logging.