Error in SQuAD-open task

[yoshitomo-matsubara]

Hello,

python -m bert.run_squad_document_full_e2e \ --vocab_file $BERT_DIR/vocab.txt \ --bert_config_file $BERT_DIR/bert_config.json \ --do_predict_open \ --data_dir $DATA_DIR \ --output_dir out/squad_doc/01

As I ran the above program with/without my minor fixes, which is pull-requested here, I faced the following error.

07/08/2019 23:59:15 - INFO - __main__ -   output_dir: out/squad_doc/011                                     
07/08/2019 23:59:17 - INFO - __main__ -   torch_version: 1.1.0 device: cuda n_gpu: 4, distributed training: 
False, 16-bits training: False                                                                              
07/08/2019 23:59:17 - INFO - __main__ -   ***** Preparing model *****                                       
07/08/2019 23:59:23 - INFO - __main__ -   ***** Preparing prediction *****                                  
07/08/2019 23:59:31 - INFO - __main__ -   Loading examples from: data/RE3QA/squad/eval_open_30para
s_examples.pkl
07/09/2019 00:00:54 - INFO - __main__ -   Loading features from: data/RE3QA/squad/eval_open_30para
s_384max_128stride_features.pkl                                                                             
07/09/2019 00:00:54 - INFO - __main__ -   Filtering features randomly                                       
07/09/2019 00:00:54 - INFO - __main__ -   Num orig examples = 10570                                         
07/09/2019 00:00:54 - INFO - __main__ -   Num split features = 444374                                       
07/09/2019 00:00:54 - INFO - __main__ -   Num split filtered features = 84560                               
07/09/2019 00:00:54 - INFO - __main__ -   Batch size for ranker = 64                                        
07/09/2019 00:00:54 - INFO - __main__ -   Batch size for reader = 32                                        
07/09/2019 00:01:06 - INFO - __main__ -   ***** Running ranking prediction *****                            
07/09/2019 00:01:06 - INFO - __main__ -   Processing example: 0                                             
07/09/2019 00:02:36 - INFO - __main__ -   Processing example: 40000                                         
07/09/2019 00:03:38 - INFO - __main__ -   Processing example: 80000                                         
07/09/2019 00:04:47 - INFO - __main__ -   Processing example: 120000                                        
07/09/2019 00:05:49 - INFO - __main__ -   Processing example: 160000                                        
07/09/2019 00:06:58 - INFO - __main__ -   Processing example: 200000                                        
07/09/2019 00:08:08 - INFO - __main__ -   Processing example: 240000                                        
07/09/2019 00:09:10 - INFO - __main__ -   Processing example: 280000                                        
07/09/2019 00:10:27 - INFO - __main__ -   Processing example: 320000                                        
07/09/2019 00:11:29 - INFO - __main__ -   Processing example: 360000                                        
07/09/2019 00:12:31 - INFO - __main__ -   Processing example: 400000                                        
07/09/2019 00:13:40 - INFO - __main__ -   Processing example: 440000                                        
07/09/2019 00:13:51 - INFO - __main__ -   ***** Reconstruct pred data at test_open_30paras_8best.pkl *****  
07/09/2019 00:13:51 - INFO - __main__ -   Filtering features based on: out/squad_doc/011/test_open_30paras_$
best.pkl                                                                                                    
07/09/2019 00:13:52 - INFO - __main__ -   Num orig examples = 10570                                         
07/09/2019 00:13:52 - INFO - __main__ -   Num split features = 444374                                       
07/09/2019 00:13:52 - INFO - __main__ -   Num split filtered features = 84560                               
07/09/2019 00:13:52 - INFO - __main__ -   Batch size for ranker = 64                                        
07/09/2019 00:13:52 - INFO - __main__ -   Batch size for reader = 32
07/09/2019 00:14:03 - INFO - __main__ -   ***** Running reading prediction *****
07/09/2019 00:14:03 - INFO - __main__ -   Processing example: 0
07/09/2019 00:17:03 - INFO - __main__ -   Processing example: 20000
07/09/2019 00:19:57 - INFO - __main__ -   Processing example: 40000
07/09/2019 00:22:59 - INFO - __main__ -   Processing example: 60000
07/09/2019 00:25:54 - INFO - __main__ -   Processing example: 80000
07/09/2019 00:27:37 - INFO - __main__ -   Writing predictions to: out/squad_doc/011/predictions.json
07/09/2019 00:27:37 - INFO - __main__ -   Writing nbest to: out/squad_doc/011/nbest_predictions.json
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/workspace/RE3QA/bert/run_squad_document_full_e2e.py", line 914, in <module>
    main()
  File "/home/ubuntu/workspace/RE3QA/bert/run_squad_document_full_e2e.py", line 910, in main
    eval_read_dataloader, logger, log_path, save_path, type='test_open', write_pred=True)
  File "/home/ubuntu/workspace/RE3QA/bert/run_squad_document_full_e2e.py", line 609, in run_read_eval
    eval_dataloader, logger, write_pred=write_pred, verbose_logging=True)
  File "/home/ubuntu/workspace/RE3QA/bert/run_squad_document_full_e2e.py", line 463, in evaluate_read
    predict_path = os.path.join(args.data_dir, args.predict_file)
  File "/home/ubuntu/.local/share/virtualenvs/RE3QA-pRGEMyAS/lib/python3.6/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/ubuntu/.local/share/virtualenvs/RE3QA-pRGEMyAS/lib/python3.6/genericpath.py", line 149, in _ch
eck_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

Could you please take a look at this problem, and provide the solution?

Thank you!

[huminghao16]

It seems like the output_dir was set as out/squad_doc/01 but it actually turned out to be out/squad_doc/011. I went through my code and I haven't found any modification to output_dir. So could you please take a look at output_dir to see if there is any bug?

[yoshitomo-matsubara]

I intentionally used out/squad_doc/011 to see what files under out/squad_doc/01 are changed and how, so I put a copy of out/squad_doc/01 as out/squad_doc/011, and ran the code.
And it seems that I needed to set predict_file, and that's why I faced the above error.

It would be great if you can update README.md again, and it includes
python -m bert.run_squad_document_full_e2e --vocab_file $BERT_DIR/vocab.txt --bert_config_file $BERT_DIR/bert_config.json --do_predict_open --data_dir $DATA_DIR --predict_file dev-v1.1.json --output_dir out/squad_doc/01

Then the final performance was shown as follows:

Ranker, type: test_open, step: 43646, map: 0.000, mrr: 0.000, top_1: 0.000, top_3: 0.000, top_5: 0.000, top_7: 0.000, retrieval_rate: 0.190 
Reader, type: test_open, step: 43646, em: 40.199, f1: 48.414

Thank you!

[Seohyeong]

Hi @yoshitomo-matsubara, I've seen you nearly reproduced the reported results with training_batch_size of 16 on SQuAD-document test set. I've been trying to reproduce the results using the released code and command lines (which uses batch size of 32) on README but I'm only getting em: 44.664, f1: 67.382 (on SQuAD document test). Is there any detail you used during training the model?

Thanks in advance!

[yoshitomo-matsubara]

Hi @Seohyeong
I've already closed this issue, but it may help