Skip to content

Add Qwen3 LLM #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sarapapi wants to merge 55 commits into
base: main
Choose a base branch
from
Open

Add Qwen3 LLM #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
28e7453
Add Qwen3 ACL6060 outputs for Canary
sarapapi Feb 13, 2026
1dbd8dd
Add Qwen3 ACL6060 outputs for remaining models
sarapapi Feb 15, 2026
7f63a00
Add Qwen3 ACL6060 long outputs
sarapapi Feb 15, 2026
cec638e
add acl6060 long and short evals
JAVI897 Feb 16, 2026
eee7c7d
qwen3 LLM outputs for wmt
Feb 17, 2026
585d334
add wmt evals
JAVI897 Feb 17, 2026
52ebfe2
Add Qwen3 MCIF long outputs
sarapapi Feb 18, 2026
80acd8c
Add Qwen3 MCIF short outputs
sarapapi Feb 18, 2026
03cfc07
add mcif-long and mcif-short evals
JAVI897 Feb 18, 2026
daa8eb7
Add Qwen3 commonAccent outputs
sarapapi Feb 23, 2026
3f2c257
Add Qwen3 cs_fleurs outputs
sarapapi Feb 23, 2026
916598b
add commonAccent evals
JAVI897 Feb 24, 2026
ffe7929
add cs-fleurs evals
JAVI897 Feb 24, 2026
f3bd56d
Add Qwen3 libristutter outputs
sarapapi Mar 2, 2026
541df1c
Add Qwen3 emotiontalk outputs
sarapapi Mar 2, 2026
321378c
Add Qwen3 cs-dialogue outputs
sarapapi Mar 2, 2026
ba7cc24
add emotiontalk evals
JAVI897 Mar 5, 2026
e3ccbe3
add cs-dialogue evals
JAVI897 Mar 5, 2026
f06a5f3
add libristutter evals
JAVI897 Mar 6, 2026
a8df473
Add Qwen3 mandi outputs
sarapapi Mar 6, 2026
1cea5c9
Add Qwen3+Whisper europarl_st outputs
sarapapi Mar 12, 2026
7d49608
Add OWSM+Qwen3 europarl_st outputs
sarapapi Mar 14, 2026
75190f6
adding europarl outputs for qwen3+ canary and seamless
Mar 14, 2026
cf32164
adding fleurs outputs for everything but owsm
Mar 14, 2026
e5ca60d
adding fleurs with owsm
Mar 15, 2026
8e62087
add fleurs and europarl evals
JAVI897 Mar 16, 2026
7e14415
covost2 with canary
Mar 17, 2026
cc1c2e7
Merge branch 'qwen3_LLM' of github.com:sarapapi/hearing2translate int…
Mar 17, 2026
07df7e2
adding qwen3+canary covost2 it-en and pt-en
Mar 18, 2026
674a8d2
add qwen3_canary-v2 evals
JAVI897 Mar 18, 2026
531d8e7
Add Whisper+Qwen3 covost2 en-it, en-nl partial outputs
sarapapi Mar 18, 2026
c9bd697
Add Whisper+Qwen3 covost2 zh-en output
sarapapi Mar 19, 2026
6c4bfc8
upload covos for qwen3_owsm4..0-ctc for en-fr, en-it, en-nl, en-pt, e…
JAVI897 Mar 19, 2026
4ba691b
update en-it, en-nl covos for qwen3_whisper
JAVI897 Mar 19, 2026
4f3f545
Add Whisper+Qwen3 covost2 en-pt, de-en, en-de, en-es, and en-fr parti…
sarapapi Mar 19, 2026
402ed51
Add Whisper+Qwen3 covost2 es-en and pt-en outputs
sarapapi Mar 20, 2026
0fe302f
add Whisper+Qwen3 covost2 en-pt, de-en, en-de, en-es, and en-fr
JAVI897 Mar 20, 2026
cda0665
covost 2 in progress for seamless and owsm
Mar 20, 2026
839c845
Add Whisper+Qwen3 covost2 it-en output
sarapapi Mar 21, 2026
76b35b7
upload current evals
JAVI897 Mar 21, 2026
981640f
upload it-en qwen3+whisper covos eval
JAVI897 Mar 23, 2026
d186dae
covost2 owsm and seamless in progress
Mar 23, 2026
f03a8b1
Add Whisper+Qwen3 noisy_fleurs_ambient outputs
sarapapi Mar 23, 2026
7b5d5a4
upload en-de qwen3_owsm4.0 and es-en qwen3_seamlessm4t evals
JAVI897 Mar 23, 2026
358c764
Merge branch 'qwen3_LLM' of https://github.com/sarapapi/hearing2trans…
JAVI897 Mar 23, 2026
7491eb5
300 lines left on en-it, rest of covost is done
Mar 24, 2026
7436cdf
upload de-en evals for qwen3_owsm4.0-ctc and en-zh qwen3_seamless evals
JAVI897 Mar 24, 2026
8a3f0bb
last missing covost file
Mar 25, 2026
b4eada7
upload en-it evals for covos - qwen3_seamless
JAVI897 Mar 25, 2026
5ec83e6
qwen3+canary noisy fleurs ambient
Mar 26, 2026
1ce88b1
Add OWSM+Qwen3 noisy_fleurs_ambient outputs
sarapapi Mar 26, 2026
a983d2f
Add OWSM+Qwen3 noisy_fleurs_babble outputs
sarapapi Mar 27, 2026
82ae769
Add Whisper+Qwen3 noisy_fleurs_babble outputs
sarapapi Mar 29, 2026
75bcf91
qwen3 canary noisy fleurs babble
Mar 30, 2026
c3d76ad
Add Seamless+Qwen3 noisy_fleurs_ambient outputs
sarapapi Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_de/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_de/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 28.6467, "chrF": 59.6531, "XCOMET": 0.8971, "XCOMET-QE": 0.8941, "RefMetricX_24": 3.0574, "QEMetricX_24": 2.9048, "LinguaPy": 0.9615, "RefMetricX_24-Strict-linguapy": 3.2573, "QEMetricX_24-Strict-linguapy": 3.1207, "XCOMET-Strict-linguapy": 0.8916, "XCOMET-QE-Strict-linguapy": 0.8899}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_fr/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_fr/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 31.3671, "chrF": 60.886, "XCOMET": 0.7475, "XCOMET-QE": 0.7651, "RefMetricX_24": 4.6501, "QEMetricX_24": 4.3637, "LinguaPy": 0.7212, "RefMetricX_24-Strict-linguapy": 4.7918, "QEMetricX_24-Strict-linguapy": 4.5301, "XCOMET-Strict-linguapy": 0.744, "XCOMET-QE-Strict-linguapy": 0.7614}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_pt/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_pt/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 40.1187, "chrF": 67.1928, "XCOMET": 0.8237, "XCOMET-QE": 0.8301, "RefMetricX_24": 5.1319, "QEMetricX_24": 4.9653, "LinguaPy": 1.2019, "RefMetricX_24-Strict-linguapy": 5.2777, "QEMetricX_24-Strict-linguapy": 5.1141, "XCOMET-Strict-linguapy": 0.8188, "XCOMET-QE-Strict-linguapy": 0.8257}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_zh/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_canary-v2/en_zh/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 33.2536, "chrF": 29.3356, "XCOMET": 0.8031, "XCOMET-QE": 0.769, "RefMetricX_24": 3.2641, "QEMetricX_24": 3.022, "LinguaPy": 0.4808, "RefMetricX_24-Strict-linguapy": 3.3613, "QEMetricX_24-Strict-linguapy": 3.1165, "XCOMET-Strict-linguapy": 0.7993, "XCOMET-QE-Strict-linguapy": 0.7652}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_de/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_de/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 29.1002, "chrF": 60.2182, "XCOMET": 0.8891, "XCOMET-QE": 0.8848, "RefMetricX_24": 3.2626, "QEMetricX_24": 3.0823, "LinguaPy": 0.9615, "RefMetricX_24-Strict-linguapy": 3.464, "QEMetricX_24-Strict-linguapy": 3.2884, "XCOMET-Strict-linguapy": 0.883, "XCOMET-QE-Strict-linguapy": 0.8807}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_fr/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_fr/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 30.9067, "chrF": 60.241, "XCOMET": 0.723, "XCOMET-QE": 0.741, "RefMetricX_24": 4.8006, "QEMetricX_24": 4.4668, "LinguaPy": 0.7212, "RefMetricX_24-Strict-linguapy": 4.9015, "QEMetricX_24-Strict-linguapy": 4.5935, "XCOMET-Strict-linguapy": 0.72, "XCOMET-QE-Strict-linguapy": 0.7382}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_pt/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_pt/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 37.6221, "chrF": 65.9293, "XCOMET": 0.8102, "XCOMET-QE": 0.8138, "RefMetricX_24": 5.2143, "QEMetricX_24": 5.0654, "LinguaPy": 0.9615, "RefMetricX_24-Strict-linguapy": 5.3662, "QEMetricX_24-Strict-linguapy": 5.2319, "XCOMET-Strict-linguapy": 0.8065, "XCOMET-QE-Strict-linguapy": 0.8113}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_zh/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_owsm4.0-ctc/en_zh/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 32.9853, "chrF": 29.0827, "XCOMET": 0.7817, "XCOMET-QE": 0.7397, "RefMetricX_24": 3.4663, "QEMetricX_24": 3.2441, "LinguaPy": 0.2404, "RefMetricX_24-Strict-linguapy": 3.5099, "QEMetricX_24-Strict-linguapy": 3.2922, "XCOMET-Strict-linguapy": 0.7799, "XCOMET-QE-Strict-linguapy": 0.7382}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_de/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_de/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 29.5416, "chrF": 59.3449, "XCOMET": 0.8694, "XCOMET-QE": 0.8652, "RefMetricX_24": 3.7395, "QEMetricX_24": 3.4778, "LinguaPy": 0.9615, "RefMetricX_24-Strict-linguapy": 3.9039, "QEMetricX_24-Strict-linguapy": 3.6543, "XCOMET-Strict-linguapy": 0.8649, "XCOMET-QE-Strict-linguapy": 0.8622}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_fr/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_fr/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 31.8824, "chrF": 60.1084, "XCOMET": 0.7307, "XCOMET-QE": 0.7447, "RefMetricX_24": 5.221, "QEMetricX_24": 5.0352, "LinguaPy": 1.2019, "RefMetricX_24-Strict-linguapy": 5.4085, "QEMetricX_24-Strict-linguapy": 5.2485, "XCOMET-Strict-linguapy": 0.7268, "XCOMET-QE-Strict-linguapy": 0.7411}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_pt/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_pt/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 38.7942, "chrF": 65.6305, "XCOMET": 0.8127, "XCOMET-QE": 0.8144, "RefMetricX_24": 5.5666, "QEMetricX_24": 5.5571, "LinguaPy": 1.9231, "RefMetricX_24-Strict-linguapy": 5.8082, "QEMetricX_24-Strict-linguapy": 5.7852, "XCOMET-Strict-linguapy": 0.8039, "XCOMET-QE-Strict-linguapy": 0.8077}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_zh/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-long/qwen3_whisper/en_zh/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 31.8265, "chrF": 28.3392, "XCOMET": 0.7593, "XCOMET-QE": 0.7246, "RefMetricX_24": 3.6935, "QEMetricX_24": 3.4646, "LinguaPy": 0.2404, "RefMetricX_24-Strict-linguapy": 3.7103, "QEMetricX_24-Strict-linguapy": 3.4725, "XCOMET-Strict-linguapy": 0.759, "XCOMET-QE-Strict-linguapy": 0.7242}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-short/qwen3_canary-v2/en_de/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-short/qwen3_canary-v2/en_de/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 32.6663, "chrF": 61.9666, "XCOMET": 0.8693, "XCOMET-QE": 0.8685, "RefMetricX_24": 3.8832, "QEMetricX_24": 3.9234, "LinguaPy": 0.4808, "RefMetricX_24-Strict-linguapy": 3.9615, "QEMetricX_24-Strict-linguapy": 3.9757, "XCOMET-Strict-linguapy": 0.8669, "XCOMET-QE-Strict-linguapy": 0.8663}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-short/qwen3_canary-v2/en_fr/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-short/qwen3_canary-v2/en_fr/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 40.0969, "chrF": 66.1331, "XCOMET": 0.7239, "XCOMET-QE": 0.7252, "RefMetricX_24": 5.8585, "QEMetricX_24": 5.82, "LinguaPy": 0.4808, "RefMetricX_24-Strict-linguapy": 5.9782, "QEMetricX_24-Strict-linguapy": 5.9383, "XCOMET-Strict-linguapy": 0.7191, "XCOMET-QE-Strict-linguapy": 0.7204}
416 changes: 416 additions & 0 deletions evaluation/output_evals/acl6060-short/qwen3_canary-v2/en_pt/results.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evaluation/output_evals/acl6060-short/qwen3_canary-v2/en_pt/results_summary.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"SacreBLEU": 42.1653, "chrF": 68.2418, "XCOMET": 0.7886, "XCOMET-QE": 0.7897, "RefMetricX_24": 6.3239, "QEMetricX_24": 6.4495, "LinguaPy": 0.4808, "RefMetricX_24-Strict-linguapy": 6.355, "QEMetricX_24-Strict-linguapy": 6.4832, "XCOMET-Strict-linguapy": 0.7873, "XCOMET-QE-Strict-linguapy": 0.7886}
Loading