[Bug] mmlu_pro数据集评测精度结果都是0

[zhangguinan]

操作系统及版本

Ubuntu 22.04.5 LTS

安装工具的python环境

在anaconda/miniconda创建的python虚拟环境

python版本

3.11

AISBench工具版本

3.11.10

AISBench执行命令

ais_bench --models vllm_api_general_chat --datasets mmlu_pro_gen_5_shot_str --debug

模型配置文件或自定义配置文件内容

root@glmTester:/home# cat /home/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content

models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr='vllm-api-general-chat',
path="/data/Qwen3-32B",
model="qwen3",
request_rate = 0,
retry = 2,
host_ip = "100.100.*.**",
host_port = 8011,
max_out_len = 8000,
batch_size=16,
trust_remote_code=False,
generation_kwargs = dict(
temperature = 0,
ignore_eos = False
)
)
]

预期行为

测试结果正常

实际行为

结果都是0

前置检查

• 我已读懂主页文档的快速入门，无法解决问题
• 我已检索过FAQ，无重复问题
• 我已搜索过现有Issue，无重复问题
• 我已更新到最新版本，问题仍存在

[github-actions]

🌟欢迎给AISBench benchmark测评工具提issue，您可以先尝试以下途径寻求issue的解决方法：

1. 【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给回答，deepwiki包含了和工具相关的所有知识库
2. 在 工具历史issue 中通过问题的关键日志片段或关键词检索相似issue
3. 在 官方文档 的搜索栏中通过问题的关键日志片段或关键词检索
4. 寻求仓库维护者的帮助（issue评论区中 @SJTUyh）

[github-actions]

🤖 基于AI机器人的issue内容完整性检查结果:

问题内容检查结果

检查状态

PASS

详细检查

• 操作系统及版本: COMPLETE
• 安装工具的python环境: COMPLETE
• python版本: COMPLETE
• AISBench工具版本: COMPLETE
• AISBench执行命令: COMPLETE
• 模型配置文件或自定义配置文件内容: COMPLETE
• 实际行为: COMPLETE

改进建议

所有内容已完备！

补充说明

该Issue内容完整，提供了所有必需的信息，有助于问题的排查和解决。

👉 如果想重新检查，在评论区@issue_checker即可。

【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给回答，deepwiki包含了和工具相关的所有知识库

[SJTUyh]

看一下输出结果中predictions/vllm-api-general-chat/*.jsonl文件，随便列出一条推理结果看看

[SJTUyh]

mmlu_pro_gen_5_shot_str这种数据集配置文件是为Base模型准备的（即通过类似续写的能力生成答案，而非chat的方式），提取答案的方式非常固化和原始（看代码逻辑是按\n split，提取第一个字符串和答案比较），完全不适用于Qwen3这种在think后通过chat方式回答的模型

[SJTUyh]

目前mmlu_pro还没有提供chat的数据集配置模板

[zhangguinan]

看一下输出结果中predictions/vllm-api-general-chat/*.jsonl文件，随便列出一条推理结果看看

"0": {
"origin_prompt": "Answer the following multiple choice question about math, and give your answer option directly.\nQuestion:\nThe symmetric group $S_n$ has $\n\factorial{n}$ elements, hence it is not true that $S_{10}$ has 10 elements.\nFind the characteristic of the ring 2Z.\nOptions:\nA. 0\nB. 30\nC. 3\nD. 10\nE. 12\nF. 50\nG. 2\nH. 100\nI. 20\nJ. 5\nAnswer: A\nQuestion:\nLet V be the set of all real polynomials p(x). Let transformations T, S be defined on V by T:p(x) -> xp(x) and S:p(x) -> p'(x) = d/dx p(x), and interpret (ST)(p(x)) as S(T(p(x))). Which of the following is true?\nOptions:\nA. ST + TS is the identity map of V onto itself.\nB. TS = 0\nC. ST = 1\nD. ST - TS = 0\nE. ST = T\nF. ST = 0\nG. ST = TS\nH. ST - TS is the identity map of V onto itself.\nI. TS = T\nJ. ST = S\nAnswer: H\nQuestion:\nLet A be the set of all ordered pairs of integers (m, n) such that 7m + 12n = 22. What is the greatest negative number in the set B = {m + n : (m, n) \in A}?\nOptions:\nA. -5\nB. 0\nC. -3\nD. -7\nE. -4\nF. -6\nG. -1\nH. -2\nI. -9\nAnswer: E\nQuestion:\nA tank initially contains a salt solution of 3 grams of salt dissolved in 100 liters of water. A salt solution containing 0.02 grams of salt per liter of water is sprayed into the tank at a rate of 4 liters per minute. The sprayed solution is continually mixed with the salt solution in the tank, and the mixture flows out of the tank at a rate of 4 liters per minute. If the mixing is instantaneous, how many grams of salt are in the tank after 100 minutes have elapsed?\nOptions:\nA. 3 + e^-2\nB. 2 - e^-4\nC. 2 - e^-2\nD. 3 + e^-4\nE. 2 + e^-3\nF. 2 - e^-3\nG. 3 - e^-2\nH. 2 + e^-2\nI. 2 + e^-4\nJ. 2\nAnswer: I\nQuestion:\nA total of 30 players will play basketball at a park. There will be exactly 5 players on each team. Which statement correctly explains how to find the number of teams needed?\nOptions:\nA. Multiply 5 by 5 to find 25 teams.\nB. Divide 30 by 5 to find 6 teams.\nC. Add 5 to 30 to find 35 teams.\nD. Subtract 30 from 5 to find -25 teams.\nE. Divide 5 by 30 to find 0.1667 teams.\nF. Add 5 to 30 then divide by 2 to find 17.5 teams.\nAnswer: B\nQuestion:\nStatement 1 | A ring homomorphism is one to one if and only if the kernel is {0}. Statement 2 | Q is an ideal in R.\nOptions:\nA. True, False\nB. Not Given, Not Given\nC. False, False\nD. Not Given, True\nE. True, Not Given\nF. Not Given, False\nG. True, True\nH. False, True\nI. False, Not Given\nAnswer: ",
"prediction": "\nOkay, let's tackle this problem step by step. The question is about two statements related to ring theory. Let me first recall what each statement is saying and then determine their truth values.\n\nStatement 1: A ring homomorphism is one to one if and only if the kernel is {0}.\n\nHmm, I remember that in group theory, a group homomorphism is injective (one-to-one) if and only if its kernel is trivial, i.e., contains only the identity element. For rings, the kernel of a ring homomorphism is the set of elements that map to zero. So, if the kernel is {0}, then the homomorphism is injective. Conversely, if the homomorphism is injective, then the kernel must be {0} because the only element that maps to zero is the additive identity. So Statement 1 seems true. That would make the first part of the answer "True".\n\nStatement 2: Q is an ideal in R.\n\nWait, Q here probably refers to the rational numbers and R to the real numbers. An ideal in a ring must satisfy two conditions: it's a subgroup under addition, and for any element in the ideal and any element in the ring, their product is in the ideal. So, is Q an ideal in R? Let's think. Take a rational number q and a real number r. The product qr is a real number, but is it necessarily in Q? No. For example, take q=1 (which is in Q) and r=√2 (which is in R but not in Q). Then 1√2 = √2, which is not in Q. Therefore, Q is not closed under multiplication by elements of R. Hence, Q is not an ideal in R. So Statement 2 is false. Therefore, the second part is "False".\n\nPutting it all together, the answer should be "True, False", which corresponds to option A. Let me check the options again to confirm. The options are A. True, False. Yes, that's correct.\n\n\nA. True, False",
"gold": "H. False, True\n"
},

[zhangguinan]

目前mmlu_pro还没有提供chat的数据集配置模板
那测试mmlu_pro数据集推荐用什么模版和命令？