Where is train_utils.py ? I cannod find the script to debug in details #82
Description
I0320 19:20:58.263954 140004008802112 train_utils.py:236] Waiting for wait_predicate_fn. Block execution. Sleeping for 2 seconds.
I0320 19:21:00.266447 140004008802112 train_utils.py:236] Waiting for wait_predicate_fn. Block execution. Sleeping for 2 seconds.
I0320 19:21:02.269329 140004008802112 train_utils.py:236] Waiting for wait_predicate_fn. Block execution. Sleeping for 2 seconds.
I0320 19:21:04.271913 140004008802112 train_utils.py:236] Waiting for wait_predicate_fn. Block execution. Sleeping for 2 seconds.
I0320 19:21:06.276944 140004008802112 train_utils.py:191] Checking for file /workspace/logs/111/policies/collect_policy/saved_model.pb (not found)
I0320 19:21:06.277659 140004008802112 train_utils.py:250] The wait_predicate_fn returned False. Continue execution.
I0320 19:21:11.384629 140004008802112 ppo_reverb_server_lib.py:73] Signature of variables:
{'policy_variables': ListWrapper([TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(17, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(17, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(17, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(12, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(56, 2048), dtype=tf.float32, name=None), TensorSpec(shape=(2048,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 16, 32), dtype=tf.float32, name=None), TensorSpec(shape=(16,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 8, 16), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 4, 8), dtype=tf.float32, name=None), TensorSpec(shape=(4,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 2, 4), dtype=tf.float32, name=None), TensorSpec(shape=(2,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 1, 2), dtype=tf.float32, name=None), TensorSpec(shape=(1,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 1), dtype=tf.float32, name=None), TensorSpec(shape=(1,), dtype=tf.float32, name=None), TensorSpec(shape=(56, 32), dtype=tf.float32, name=None), TensorSpec(shape=(32,), dtype=tf.float32, name=None), TensorSpec(shape=(32, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None)]), 'train_step': TensorSpec(shape=(), dtype=tf.int64, name=None), 'model_id': TensorSpec(shape=(), dtype=tf.int64, name=None)}
I0320 19:21:11.388135 140004008802112 ppo_reverb_server_lib.py:80] Signature of experience:
Trajectory(step_type=TensorSpec(shape=(None,), dtype=tf.int32, name='step_type'), observation={'is_node_placed': BoundedTensorSpec(shape=(None, 3500), dtype=tf.int32, name='observation/is_node_placed', minimum=array(0, dtype=int32), maximum=array(1, dtype=int32)), 'mask': BoundedTensorSpec(shape=(None, 16384), dtype=tf.int32, name='observation/mask', minimum=array(0, dtype=int32), maximum=array(1, dtype=int32)), 'netlist_index': BoundedTensorSpec(shape=(None, 1), dtype=tf.int32, name='observation/netlist_index', minimum=array(0, dtype=int32), maximum=array(0, dtype=int32)), 'locations_y': BoundedTensorSpec(shape=(None, 3500), dtype=tf.float32, name='observation/locations_y', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)), 'current_node': BoundedTensorSpec(shape=(None, 1), dtype=tf.int32, name='observation/current_node', minimum=array(0, dtype=int32), maximum=array(3499, dtype=int32)), 'fake_net_heatmap': BoundedTensorSpec(shape=(None, 16384), dtype=tf.float32, name='observation/fake_net_heatmap', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)), 'locations_x': BoundedTensorSpec(shape=(None, 3500), dtype=tf.float32, name='observation/locations_x', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32))}, action=BoundedTensorSpec(shape=(None,), dtype=tf.int64, name='action', minimum=array(0), maximum=array(16383)), policy_info={'value_prediction': TensorSpec(shape=(None,), dtype=tf.float32, name='value_prediction'), 'dist_params': {'logits': TensorSpec(shape=(None, 16384), dtype=tf.float32, name='GrlPolicyModel_logits')}}, next_step_type=TensorSpec(shape=(None,), dtype=tf.int32, name='step_type'), reward=TensorSpec(shape=(None,), dtype=tf.float32, name='reward'), discount=BoundedTensorSpec(shape=(None,), dtype=tf.float32, name='discount', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)))
[reverb/cc/platform/tfrecord_checkpointer.cc:162] Initializing TFRecordCheckpointer in /tmp/tmpk4u60eow.
[reverb/cc/platform/tfrecord_checkpointer.cc:565] Loading latest checkpoint from /tmp/tmpk4u60eow
[reverb/cc/platform/default/server.cc:71] Started replay server on port 8008