Potential memory leak in training PPO agents #36
Description
Hi,
I followed the tutorial setting up the docker container and ran train.py with all default hyper parameters on tictactoe. Here is my command
sudo docker-compose exec app python3 train.py -r -e tictactoe
I did not use parallelization and I notice that the RAM is growing linearly w.r.t. the training steps. Roughly about 700 MB memory increase after 0.2M steps. After training for ~20M steps my computer's 32GB RAM will be fully occupied.
Has anyone encountered similar issue and is there a way to resolve this?