Unable to reproduce results of ASearcher-Web-QwQ on GAIA.

[ligang-cs]

Thank you for your excellent work. I have encountered some difficulties reproducing the results of ASearcher-Web-QwQ on GAIA. I am only able to achieve 41.7% accuracy, which is significantly lower than the 52.8% reported in the paper.
Here is my eval script:

MODEL_PATH=ASearcher-Web-QwQ_inclusionAI
DATA_NAMES=GAIA
AGENT_TYPE=asearcher-reasoning
PROMPT_TYPE=asearcher-reasoning
SEARCH_CLIENT_TYPE=async-web-search-access

python3 search_eval_async.py \
    --data_names ${DATA_NAMES} \
    --model_name_or_path ${MODEL_PATH}  \
    --output_dir results \
    --data_dir ${DATA_DIR} \
    --prompt_type $PROMPT_TYPE \
    --agent-type ${AGENT_TYPE} \
    --search-client-type ${SEARCH_CLIENT_TYPE} \
    --tensor_parallel_size 4 \
    --temperature 0.6 \
    --parallel-mode seed \
    --seed 1 \
    --aggregate-only \
    --use-jina \
    --llm_as_judge \
    --use-openai True \
    --pass-at-k 1 

evaluation results:

JINA and SERPER APIs are correctly set. Thank you for your help!

[samjia2000]

@ligang-cs Thanks for your interest in our work. We will look into this issue.

[ligang-cs]

@samjia2000 https://github.com/inclusionAI/ASearcher/blob/main/agent/asearcher_reasoning.py#L519
I guess, the '10000' character limit may be a mistake, it should be '25000' ?

[GurrenLagann97]

@ligang-cs Hi, we’ve identified the issue: in agent/asearcher_reasoning.py, at line 269, the is_finished method mistakenly included the code used during testing. We’ve already opened a PR to fix this issue, but you can make the modification yourself first to reproduce our results:

you just need to remove line 269 and also delete force_done from the return statement.

We re-ran the GAIA tests and got the following results:
• pass@1: 49.5
• pass@4: 70.4

Note that pass@1 shows relatively high variance, as it’s influenced by multiple factors, so it may fluctuate around the reported value.