Understanding Metabuli scores on real metagenomic datasets

[taorong007]

Hi Metabuli team,

I'm evaluating Metabuli on real metagenomic samples and noticed significantly
lower scores compared to synthetic benchmarks. I'm trying to understand if this
is expected and how to optimize for real-world data.

Score comparison across datasets:

Dataset	MMseqs2 confidence	Metabuli score	Sample type
CAMI-Marine	0.93	0.81	Synthetic
IBS	0.89	0.45	Real (gut)
Freshwater	0.60	0.05	Real (environmental)

My concerns:

1. The dramatic score drop on real data makes it difficult to set confidence
   thresholds for filtering
2. Many assignments that look taxonomically reasonable have very low scores
3. It's unclear whether low scores indicate:
   • True uncertainty (novel/divergent organisms)
   • Database coverage issues
   • Algorithm behavior on fragmented/noisy real data

Questions:

1. Is this score pattern expected for real metagenomic data?
2. What factors most influence Metabuli's scoring on real vs. synthetic data?
3. Are there parameters I should adjust for real samples? (e.g., --min-score,
   --min-sp-score)
4. How do you recommend filtering classifications from real data - by score
   threshold or other metrics?
5. Would using a more recent GTDB version significantly improve scores on
   real samples?

My setup:

• Database: GTDB r214
• Metabuli version: 1.1.0

I've attached example outputs showing the score distribution. Any guidance
on interpreting and optimizing for real data would be greatly appreciated!

Thanks!
metabuli on freshwater-ERR4195020:

mmseq2 on freshwater-ERR4195020:

[jaebeom-kim]

Hi! Thank you for reporting this.

Metabuli extracts k-mers (which hold both DNA and AA information) from reads and searches the database to find AA-level exact matches.
For each read, some of its k-mers have matches, while others do not.
For each read, Metabuli counts the number of identical nucleotides within the regions where k-mer matches occurred.
This number is divided by the read length to get the score.

So,
score = (# of identical DNA within k-mer matches) / (read length)

It seems that contigs were used in your example, so read lengths were large, which resulted in small scores.
If the contigs were from species included in the Metabuli DB, they would have larger scores.
I guess the species in your sample are very divergent from those in the DB.

Could you please share few reads you used? I want to check if the scores are calcuated correctly

Would you try using the DB from https://hulk.mmseqs.com/jaebeom/gtdb226db/ ?
I guess it includes 10 times more species than what you used.
Metabuli might be able to find better matches in this larger DB.

I’m currently trying to report E-values for classification; I think they will be useful for filtering spurious hits. I will leave a comment here when this is ready.

---

1. Is this score pattern expected for real metagenomic data?

• When species in a sample are very divergent from those in the DB, small scores are expected.

2. What factors most influence Metabuli’s scoring on real vs. synthetic data?

• The divergence between species in the sample and the DB is the most influential factor.

3. Are there parameters I should adjust for real samples? (e.g., --min-score, --min-sp-score)

• If you want to filter out results that are likely to be false positives and the quality of your reads is good, try --min-score 0.07 --min-sp-score 0.3.
  Please see Supplementary Figure 5 in the paper to understand how these values were set.

4. How do you recommend filtering classifications from real data—by score threshold or other metrics?

• Please try the parameters above. It will decrease the number of classifications, but remaining more confident ones.

5. Would using a more recent GTDB version significantly improve scores on real samples?

• The more diverse the species in the DB, the more the scores will improve. Please try the larger DB above.

[taorong007]

Thank you for the detailed explanation! This makes much more sense now.

Yes, I used assembled contigs, which explains the lower scores due to longer
sequence lengths.

Clarification on my workflow:
For all datasets (CAMI-Marine, IBS, and Freshwater), I used the same pipeline:
Raw reads → MetaSPAdes Assembly → Contigs → Metabuli/MMseqs2 annotation

Since all used assembled contigs, the score difference is likely due to:

• CAMI-Marine species being well-represented in GTDB
• Real samples containing more divergent/novel organisms

I will try:

1. Test with the GTDB r226 database you provided
2. Apply the filtering parameters (--min-score 0.07 --min-sp-score 0.3)

Regarding contig examples:
Here are 5 representative contigs from the freshwater dataset (ERR4195020).
These contigs have high MMseqs2 confidence scores but very low Metabuli scores.

I've included the first three longest assembled contigs in the attached file.
Below are two shorter contig examples:

>NODE_3999178_length_214_cov_1.125786
GGAATAAAAAAAAGAATATCAATATTGCAGCCCAAAGAATATATATCGGAGACAAGGTGCAGCTCGGGAAGAACAGTTCAATCAAGGTGAGGGGTGAATTCAAGCTTGGAAGTTACAGCCGGCTAGGGGATGATGTAAACATCAGAGGAAACAATGTTAGTCTTGGTGAGCATCTCTACCATTCCTTCGGATTACGAGTAGGTGGGGGAGGCTC

MMseqs2: species, ID 19384, confidence: 1
Metabuli: no rank, score: 0

>NODE_2824617_length_244_cov_0.962963
TTCACTGTCAGGGTCAGTTCGGCCTTCAACAGGTCGTCCGTGGGCTGGTCGATGTTGTAGCGCTGGAATATCTTGGCGCGCTTGGCGGCGATTTTCATCACTACCTGGTCGGTTTCGTAGGCCTGCTCCATTTGCACCAATTGGCGGAGAACGGGCTGGACGAAGGTCTCGACGTAGGTGCGGATCAGGTATTCGACCAGCGTCCCGGATGCGTTGGAGAGCATGGCCATGTTCCGAGCCGGAG

MMseqs2: species, ID 76352, confidence: 1
Metabuli: no rank, score: 0

Attached file contents:

The three larger contigs in the file show:

• NODE_1_length_557759_cov_21.547228: MMseqs2 (species 16263, confidence: 0.850), Metabuli (1 NODE_1_length_557759_cov_21.547228 192287 557757 0.00113939 genus)
• NODE_2_length_301862_cov_27.617507: MMseqs2 (species 3689, confidence: 0.84), Metabuli (1 NODE_2_length_301862_cov_27.617507 246208 301860 0.00673988 order)
• NODE_3_length_269606_cov_27.064500: MMseqs2 (species 3689, confidence: 0.880), Metabuli (1 NODE_3_length_269606_cov_27.064500 246208 269604 0.00825748 order)

The complete raw sequencing dataset was retrieved from the NCBI Sequence Read
Archive (SRA) and is available at:
https://trace.ncbi.nlm.nih.gov/Traces/?view=run_browser&acc=ERR4195020

Looking forward to the E-value feature - that would be very helpful for
downstream analysis!

Thanks again!

ERR4195020_first3contigs.contigs.fasta.txt

[jaebeom-kim]

Great! Could you tell me which MMseqs2 database you used?

…

2026. 1. 19. 오후 9:21, taorong007 ***@***.***> 작성: taorong007 left a comment (steineggerlab/Metabuli#171) <#171 (comment)> Thank you for the detailed explanation! This makes much more sense now. Yes, I used assembled contigs, which explains the lower scores due to longer sequence lengths. Clarification on my workflow: For all datasets (CAMI-Marine, IBS, and Freshwater), I used the same pipeline: Raw reads → MetaSPAdes Assembly → Contigs → Metabuli/MMseqs2 annotation Since all used assembled contigs, the score difference is likely due to: CAMI-Marine species being well-represented in GTDB Real samples containing more divergent/novel organisms I will try: Test with the GTDB r226 database you provided Apply the filtering parameters (--min-score 0.07 --min-sp-score 0.3) Regarding contig examples: Here are 5 representative contigs from the freshwater dataset (ERR4195020). These contigs have high MMseqs2 confidence scores but very low Metabuli scores. I've included the first three longest assembled contigs in the attached file. Below are two shorter contig examples: >NODE_3999178_length_214_cov_1.125786 GGAATAAAAAAAAGAATATCAATATTGCAGCCCAAAGAATATATATCGGAGACAAGGTGCAGCTCGGGAAGAACAGTTCAATCAAGGTGAGGGGTGAATTCAAGCTTGGAAGTTACAGCCGGCTAGGGGATGATGTAAACATCAGAGGAAACAATGTTAGTCTTGGTGAGCATCTCTACCATTCCTTCGGATTACGAGTAGGTGGGGGAGGCTC MMseqs2: species, ID 19384, confidence: 1 Metabuli: no rank, score: 0 >NODE_2824617_length_244_cov_0.962963 TTCACTGTCAGGGTCAGTTCGGCCTTCAACAGGTCGTCCGTGGGCTGGTCGATGTTGTAGCGCTGGAATATCTTGGCGCGCTTGGCGGCGATTTTCATCACTACCTGGTCGGTTTCGTAGGCCTGCTCCATTTGCACCAATTGGCGGAGAACGGGCTGGACGAAGGTCTCGACGTAGGTGCGGATCAGGTATTCGACCAGCGTCCCGGATGCGTTGGAGAGCATGGCCATGTTCCGAGCCGGAG MMseqs2: species, ID 76352, confidence: 1 Metabuli: no rank, score: 0 Attached file contents: The three larger contigs in the file show: NODE_1_length_557759_cov_21.547228: MMseqs2 (species 16263, confidence: 0.850), Metabuli (1 NODE_1_length_557759_cov_21.547228 192287 557757 0.00113939 genus) NODE_2_length_301862_cov_27.617507: MMseqs2 (species 3689, confidence: 0.84), Metabuli (1 NODE_2_length_301862_cov_27.617507 246208 301860 0.00673988 order) NODE_3_length_269606_cov_27.064500: MMseqs2 (species 3689, confidence: 0.880), Metabuli (1 NODE_3_length_269606_cov_27.064500 246208 269604 0.00825748 order) The complete raw sequencing dataset was retrieved from the NCBI Sequence Read Archive (SRA) and is available at: https://trace.ncbi.nlm.nih.gov/Traces/?view=run_browser&acc=ERR4195020 Looking forward to the E-value feature - that would be very helpful for downstream analysis! Thanks again! ERR4195020_first3contigs.contigs.fasta.txt <https://github.com/user-attachments/files/24715215/ERR4195020_first3contigs.contigs.fasta.txt> — Reply to this email directly, view it on GitHub <#171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQK2QJJTIWONVF3Q23KFI334HTD5NAVCNFSM6AAAAACSDWCHMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTONRYGA3DMNJZG4>. You are receiving this because you commented.

[taorong007]

I used GTDB R214 as the MMseqs2 database.

It was downloaded using:

 mmseqs databases GTDB GTDB tmp

with MMseqs2 version 17.b804f.

The database contains 437,238,178 protein sequences (confirmed by wc -l GTDB.lookup).

[taorong007]

Hi Metabuli team,

I tried the GTDB r226 database from the link you provided (https://hulk.mmseqs.com/jaebeom/gtdb226db/) on cami-marine sample 0, but getting unexpected results.

Issue:

GTDB r214: 99.24% classified normally
GTDB r226: 100% unclassified (all sequences marked as "unclassified")
Key difference in logs:

r214: 1,635,759,520 matches found
r226: 122,208,180 matches found (93% reduction)
Command used (same for both databases):

metabuli classify --seq-mode 1 --threads 52 \
    marine-sample-0.contigs.fasta \
    /path/to/gtdb_database/ \
    output_dir \
    sample0_contigs

Questions:

• Is my command correct for using the r226 database?
• Did I download the database correctly? I used wget on all files from your link.
• Is there a known issue with the r226 database, or do I need to rebuild it?

Input file: CAMI-Marine sample 0 (402,046 contigs, same file works fine with r214)

Attached screenshots show the complete output for both runs.

Thanks!

GTDB r214:

GTDB r226:

[jaebeom-kim]

Thank you for reporting this issue.
Sorry I forgot to say that the latest commit is required.
Please find a binary for your server in the link below.
https://dev.mmseqs.com/metabuli/
(metabuli-linux-avx2.tar.gz is suitable in most linux servers)
Since it is a version under active development, there might be other issues.
Please kindly report any problems you meet. It would greatly help us to maintain this software :)

[taorong007]

Hi,

Thank you for your quick response and for providing the precompiled binary!

I'm trying to use it to work with GTDB r226 database, but I'm on
Oracle Linux 8.10 with GLIBC 2.28.

The precompiled binary from https://dev.mmseqs.com/metabuli/ requires
GLIBC 2.29-2.34. Could you:

1. Provide a version compiled for GLIBC 2.28?
2. Or provide the source code corresponding to the precompiled version
   on https://dev.mmseqs.com/metabuli/ so I can compile it myself?

I need the version that supports GTDB r226.

Thank you again for your help!

[jaebeom-kim]

The precompiled binaries are corresponding to the latest commit in this repo.
You can clone this repo and compile yourself :)
please follow this.
Looking forward hearing from you.

[taorong007]

Hi Metabuli team,

Thank you so much for your help! I successfully compiled the latest version from the repository and ran it with the GTDB r226 database on CAMI-Marine sample 0.

My setup:

• Metabuli Version (commit): 22e7026
• Database: GTDB_R226_Rep_COMP70_CONTAM5 (created 2025-7-28)
• Command: metabuli classify --seq-mode 1 --threads 52

The classification is working much better now, but I noticed something in the output:

Observation:

In both output files, I see many entries labeled as "no rank":

In report.tsv:

1.4921  5999    4       species 720715107                     Vibrio nigripulchritudo
1.4911  5995    5995    no rank 1959996476                      GCF_000222685.1
0.7795  3134    4       species 1219554346                    Vibrio natriegens
0.7785  3130    3130    no rank 1536327903                      GCF_035621455.1

In classifications.tsv:

1    k141_38720     2137988105    315    0.790476    no rank    2137988105:24
1    k141_348437    460831477     306    0.769608    no rank    460831477:23

Question:

Are these "no rank" entries (like GCF_000222685.1, GCF_035621455.1) representing subspecies/strain-level assignments below the species level? The GCF_ identifiers suggest these are genome assemblies, and they appear as children under species nodes in the taxonomic tree.

Thanks again for all your support!

[jaebeom-kim]

Great to hear that it works now! Yes that's correct! 'no rank' can appear when a rank name is not assigned. In the GTDB case, they are all assemblies of species. But in the case of NCBI taxonomy-based database, there are diverse cases with 'no rank'. It is just a place holder. 2026년 1월 29일 (목) 오후 3:59, taorong007 ***@***.***>님이 작성:

…

*taorong007* left a comment (steineggerlab/Metabuli#171) <#171 (comment)> Hi Metabuli team, Thank you so much for your help! I successfully compiled the latest version from the repository and ran it with the GTDB r226 database on CAMI-Marine sample 0. *My setup*: - Metabuli Version (commit): 22e7026 <22e7026> - Database: GTDB_R226_Rep_COMP70_CONTAM5 (created 2025-7-28) - Command: metabuli classify --seq-mode 1 --threads 52 The classification is working much better now, but I noticed something in the output: *Observation*: In both output files, I see many entries labeled as "no rank": In report.tsv: 1.4921 5999 4 species 720715107 Vibrio nigripulchritudo 1.4911 5995 5995 no rank 1959996476 GCF_000222685.1 0.7795 3134 4 species 1219554346 Vibrio natriegens 0.7785 3130 3130 no rank 1536327903 GCF_035621455.1 In classifications.tsv: 1 k141_38720 2137988105 315 0.790476 no rank 2137988105:24 1 k141_348437 460831477 306 0.769608 no rank 460831477:23 Question: Are these "no rank" entries (like GCF_000222685.1, GCF_035621455.1) representing subspecies/strain-level assignments below the species level? The GCF_ identifiers suggest these are genome assemblies, and they appear as children under species nodes in the taxonomic tree. Thanks again for all your support! 2026-01-29.14.52.08.png (view on web) <https://github.com/user-attachments/assets/d5871ce6-ff54-4294-96e3-e65f5241ff92> 2026-01-29.14.51.59.png (view on web) <https://github.com/user-attachments/assets/1eff7e05-bcc2-4df9-affa-1d3ed475ba8c> 2026-01-29.14.56.58.png (view on web) <https://github.com/user-attachments/assets/ca923928-c505-4d9b-9978-f63aa087d8bf> — Reply to this email directly, view it on GitHub <#171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQK2QJOWUAVLUQSLL7UX3TT4JGVTZAVCNFSM6AAAAACSDWCHMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQMJVHA2DGMBQGI> . You are receiving this because you commented.Message ID: ***@***.***>