the result of Ultravox-v0.5-LLaMA-3.1-8B

[Gpwner]

I have test Ultravox-v0.5-LLaMA-3.1-8B too.but My test results are slightly different from yours, especially on the sdqa dataset.

	AlpacaEval	CommonEval	SD-QA	OpenBookQA	IFEval	AdvBench
Open-Ended QA	Open-Ended QA	Reference-Based QA	Multiple-Choice QA	Instruction Following	Safety
samples	199	200	553	455	345	520
Ultravox0.5 LLama3.1 8B Instruct	4.75	4.08	72.42	69.01	68.05	98.84

[Gpwner]

the last point of the sd-qa is(panda+gpt)/2,right?

[MatthewCYM]

Hi, I just notice that using qa_metrics==0.2.17 (as specified in the requirements) results in a Panda score of 47.74, which aligns with the current performance reported on the leaderboard. However, when using the latest version, qa_metrics==0.2.30, the Panda score unexpectedly jumps to 74.50. I'm currently investigating this discrepancy.

[MatthewCYM]

Could you please check if you observe the same discrepancy?

[Gpwner]

I am using qa_metrics==0.2.24. When i install qa_metrics==0.2.17 and run the evaluator i got this error:

    panda_results = [self.pedant.evaluate([item['reference'].lower()], item['response'].lower(), item['prompt'].lower()) for item in data]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/extendisk1/usr/bencheval/VoiceBench/src/evaluator/qa.py", line 22, in <listcomp>
    panda_results = [self.pedant.evaluate([item['reference'].lower()], item['response'].lower(), item['prompt'].lower()) for item in data]
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/usr/miniconda3/envs/vllm/lib/python3.11/site-packages/qa_metrics/pedant.py", line 267, in evaluate
    output = self.model.predict(result)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/usr/miniconda3/envs/vllm/lib/python3.11/site-packages/sklearn/linear_model/_base.py", line 451, in predict
    scores = self.decision_function(X)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/usr/miniconda3/envs/vllm/lib/python3.11/site-packages/sklearn/linear_model/_base.py", line 432, in decision_function
    X = self._validate_data(X, accept_sparse="csr", reset=False)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/usr/miniconda3/envs/vllm/lib/python3.11/site-packages/sklearn/base.py", line 626, in _validate_data
    self._check_n_features(X, reset=reset)
  File "/home/usr/miniconda3/envs/vllm/lib/python3.11/site-packages/sklearn/base.py", line 415, in _check_n_features
    raise ValueError(
ValueError: X has 82735 features, but SGDClassifier is expecting 82768 features as input.

[Gpwner]

by the way,when i evaluate ifeval,does the last point cal by (strict-prompt+strict-instruction+loose-prompt+loose-instruction)/4 ? thanks

[MatthewCYM]

Yes, it's calculated as (strict-prompt+strict-instruction+loose-prompt+loose-instruction)/4.

[MatthewCYM]

According to zli12321/qa_metrics#2, the panda score calculation method changed in version 0.2.17 and later, explaining the observed score discrepancy.

[zli12321]

The panda score is tuned to be more correlated with GPT-4 in version 0.2.17 and later (0.2.30 is currently the latest). The jump of score is because pandas gives a more relaxed match than the older version.
Old panda accuracy 0.47 and GPT-4 is 0.68. New panda accuracy 0.76 and is closer to GPT-4 accuracy.

[Gpwner]

so,will the leadboard update to use the latest qa_metrics == 0.2.30 ? @MatthewCYM

[zli12321]

The correlation is closer to LLM evaluation, but since the score averages out with get, the overall rankings for existing leaderboard will mostly remain the same.