Imporove apex

[ymd-h]

This PR is for #117

• Update cpprb and use the new class MPPrioritizedReplayBuffer (multi-process version PER), which doesn't requires manual lock
  • MPPrioritizedReplayBuffer doesn't lock whole buffer but critical section only
• Remove busy loop and use efficient waiting by queue's get() method.
• Remove slow proxy, SyncManager
• Reduce access for shared value, trained_steps
  • Fix atomic increment, too. (+= requires manual lock. See doc)
• Run learner at main process instead of background process
• Remove writer.flush() (I just let TensorFlow flush. We might need to adjust flush timing for our needs)
• Remove eps (small value) addition to TD, which is added in buffer.

This improvement have larger effect for small network and/or simple Env.

I tested by running example/run_apex_dqn.py with default "CartPole-v0" on CPU machine.
Please test other Envs and/or on GPU.

P.S.
Weights distribution with multiple queues seems to be inefficient because of multiple copying.
I will continue to consider other solution.

[keiohta]

Hi @ymd-h , thanks for brilliant PR!! I would really appreciate your continuous support!

I checked changes of codes, and I think all of them would contribute improvement of ApeX performance.

P.S.
Weights distribution with multiple queues seems to be inefficient because of multiple copying.
I will continue to consider other solution.

Yeah, this is true. I also consider some workaround. Thanks!

[ymd-h]

@keiohta

I updated the PR.

• Merge master branch to take recent update
• Use newer cpprb, which improved Segment Tree implementation

On my local machine, (a part of) logs at example/run_apex_dqn.py are followings;

Before PR

After PR

It seems that Super Linter v3 is strangely broken and we need to upgrade to v4. (I will add soon.)
https://github.com/github/super-linter/issues/2253