Test Configuration Difference

[taodeng009]

Hello , authors! Amazing work! I'm now trying to reproduce your work, but I notice some configuration difference in environment prompt (ALFWORLD) and hyperparameter comparing the logs you given and github code. I think this may lead to lower success rate. In the logs you given, the ALFWORLD prompt allows 8 actions, yet the prompt in github code adds "think: xxx". And, the logs didn't mention whether the "use_projector" is turned on or not. Could you please specify the configurations? It would help me a lot, thanks!

[xinding-sys]

Hello, authors! I also want to know more details! I cannot reproduce the result in this paper.

[bingreeky]

@taodeng009 We also suspect that the difference might be related to the additional action. However, in principle it should not cause a large performance change, since the few-shot examples provided in ALFWorld already include cases where “think” appears as a separate action.

That said, it is also possible that explicitly adding “think” as an available action encourages the agent to overuse thinking steps, which could affect performance. In our original experiments, we did not explicitly include “think” as a selectable action in the action space.

We are currently testing this factor as well. If it is convenient, could you share your logs with us? That would help us better understand the discrepancy.

Thanks again for your interest and for trying to reproduce the results.

[taodeng009]

Thank you for your response. I actually obtain a success rate of "0.8881" in ALFWORLD using QWEN2.5-14B-INSTRUCT after I turn on the "use_projector" and modify the bugs mentioned in issue #14. However, the results of QWEN2.5-7B-INSTRUCT in ALFWORLD are very bad, demonstrating a substantial gap to the results in the paper. In addtion, the average rewards of GMemory are even worse than Empty in the PDDL tasks, both in QWEN-7B and QWEN-14B cases. You can check the logs through https://github.com/taodeng009/GMemory-Reproducing-Logs. Thank you for your assistance.