FlyCopter-RL

A reinforcement learning environment inspired by the classic Helicopter Game, built with Gymnasium and trained with Stable Baselines3 PPO.

The agent navigates a helicopter through a procedurally generated cave with narrowing walls and random obstacles, learning to survive as long as possible.

Environment

Detail	Value
Action space	Discrete(2) — 0: gravity, 1: thrust
Observation space	Box(6) — continuous float32 vector
Reward	+1 per frame alive, -10 on collision
Rendering	human (Pygame window) or rgb_array

Observation vector

Index	Description
0	Helicopter Y position
1	Vertical velocity
2	Top wall at helicopter position
3	Bottom wall at helicopter position
4	Distance to next obstacle (X)
5	Y position of next obstacle

Installation

git clone https://github.com/<your-username>/flycopter-rl.git
cd flycopter-rl
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

Usage

Train a new model (PPO)

python train.py

This trains a PPO agent for 100,000 timesteps and saves the model as ppo_copter_model.zip.

Train from scratch (DQN)

python train_scratch.py

Trains a DQN (Deep Q-Network) agent using only PyTorch — no Stable Baselines3 needed. Uses experience replay, epsilon-greedy exploration, and a target network. Trains for 100,000 steps with a replay buffer of 50,000 transitions, saves the model as dqn_copter_model.pt, and runs a demonstration episode with live rendering afterwards.

Watch the trained agent

python run_agent.py

Opens a Pygame window and runs the trained agent for up to 5,000 frames, resetting on collision.

Record a video

python record_video.py

Saves an episode recording as an MP4 in ./copter_videos/.

Project structure

flycopter-rl/
├── flycopter_rl/
│   ├── __init__.py       # Package init, exports FlyTheCopterEnv
│   └── env.py            # Custom Gymnasium environment
├── train.py              # Train the PPO model
├── train_scratch.py      # Train with DQN (from scratch, PyTorch only)
├── record_video_scratch.py  # Record DQN agent episode to video
├── run_agent.py          # Run trained agent with live rendering
├── record_video.py       # Record PPO agent episode to video
├── requirements.txt      # Python dependencies
└── README.md

Dependencies

• Gymnasium — RL environment API
• Stable Baselines3 — PPO implementation
• Pygame — Rendering
• NumPy — Numerical operations

License

MIT