Add RL/Gymnasium Integration

[EthanMBoos]

Python bindings + Gymnasium wrapper for training RL agents in Scrimmage.

Why

The old OpenAI Gym integration was removed in Nov 2025 (commits 87ffbf1, e140f2a). Looking at what got deleted, I think the issue was embedding Python inside the C++ sim loop — Global Interpreter Lock acquisition every entity every timestep, messy control flow, etc.

Proposed approach

Flip the control: Python drives, C++ executes.

• Use run_single_step() which already exists
• Simple RLAutonomy plugin that's basically just a mailbox for actions
• pybind11 bindings for SimControl/State/Contact
• Gymnasium wrapper on top

The bindings would be opt-in (-DENABLE_RL_BINDINGS=ON), keeping the default build clean.

Scope

Starting small:

• Single agent waypoint navigation
• Maybe 1v1 pursuit evasion
• SAC/TD3 (not PPO, need sample efficiency)

Multi-agent is a stretch goal.

What it'd look like

import gymnasium as gym
import scrimmage_gym
from stable_baselines3 import SAC

env = gym.make("Scrimmage-Waypoint-v0", mission="missions/waypoint.xml")
model = SAC("MlpPolicy", env)
model.learn(100_000)

[EthanMBoos]

Data flow

Python (Gymnasium)
      │
      │ 1. env.step(action)  # action = [desired_heading, desired_speed]
      │    └─► writes to RLAutonomy buffer
      │
      │ 2. sim.run_single_step()
      ▼
┌─────────────────────────────────────────────────────────────────┐
│  SimControl                                                     │
│                                                                 │
│    RLAutonomy ──────► Controller ──────► MotionModel            │
│    (outputs           (PID, etc.)        (physics)              │
│     desired_heading,                          │                 │
│     desired_speed)                            ▼                 │
│                                          Updated State          │
│                                          (pos, vel, quat)       │
│                                               │                 │
│                                               ▼                 │
│                                            Sensor               │
└─────────────────────────────────────────────────────────────────┘
      │
      │ 3. Read state, compute reward
      ▼
return (obs, reward, done, info)