Large peptide ID number differences between v14.6 and v15.1

[zix-xiao]

Hi there,

I have noticed an interesting discrepancy between v0.14.6 and v0.15.0-beta.1. The search results for a short LC gradient (5 minute active gradient) timstof ddaPASEF data differ quite a lot in terms of PSM as well as uniquely ID'ed peptide numbers at 1%FDR.

This is the all unique unmodified peptides from psm

And this is the unique unmodified peptides identified at 1%FDR

However, the search results on a 30min LC gradient timstof ddaPASEF are much more consistent across the versions:

The same config file is used for both versions, except for as suggested in the change log the LFQ config on v14 will have no effect default_config.json

Is there some changes from v14 to v15 that could potentially lead to this performance change?

[theGreatHerrLebert]

Hey everyone,

Since this effect has now also been observed on real-world data, I wanted to share some findings from our recent tests on simulated DDA-PASEF data, where the same pattern emerged.

We’re currently developing a simulator that can generate synthetic DDA-PASEF datasets and export them to Bruker TDF format. Our main goals include:

• Benchmarking proteomics tools
• Supporting software development
• Enabling robust unit testing

We think this simulator might also be useful to the SAGE community as part of a future testing ecosystem, once the tool is stable enough.

Observations

Here is a comparison of identifications using SAGE 14.07 vs. SAGE 15.01, on the same synthetic dataset, both with and without a peptide-q filter (q ≤ 0.01), on ion- and peptide-level:

We also ran the same dataset across all available SAGE versions that support the TDF reader:

There appears to be a significant drop in identifications in SAGE 15.01, compared to previous versions, despite identical input and configuration. This is consistent across both ions and peptides on this simulated dataset.

If you’re interested, I can provide, the simulated raw data (.d folders) and ground truth peptides and ions used to generate the dataset. Here also the exact parameter file used for all runs: parameters_sim_tdf.json

Let me know if you’d like access to the files, and of course, no problem at all if this type of data isn’t currently relevant for your work.

Best,

David

[lazear]

Thanks for the details - having the simulated raw data, and the raw data from @zix-xiao would be very useful for debugging. There have been quite a few changes since v14.7 to the Bruker side of things, so hopefully we can pin down where the regression is.

It's a beta version for a reason, thanks for helping to test it out!

[theGreatHerrLebert]

Cool!

I’ve uploaded the raw data to Zenodo, you can (hopefully) access it here.

The archive contains:

• raw-dda-hela-200K.zip: the synthetic raw data in Bruker .d format
• synthetic_data-hela-200K.zip: the corresponding ground truth as a SQLite database

If you’re curious about how the simulation works under the hood, we’ve also prepared Jupyter notebooks showing the setup for both:

• DDA-PASEF simulation
• DIA-PASEF simulation

Let me know if I can be of any help,

Best,
David

[zix-xiao]

Thanks for the speedy response, much appreciated! I just asked for my collaborators' consent to share the data. I will share them as soon as I get a green light.

After a second try of the testing, this behavior does not seem to be necessarily, or only, a result of the LC gradient length, but rather more related to whether the data are acquired with "denoise" on or off.

Bests,
Zixuan

[sander-willems-bruker]

We will also look into ths internally. Quick note on it being either LC or denoise related, I perosnally think it is neither actually. The config states to use "max_peaks": 150. I hypothesize that short gradients have much more peaks per spectrum. The same is true for spectra which are not denoised. As a result, you might miss a bunch of peaks for identification if they are of lower intensity.

[zix-xiao]

We tested several different cases, including chaning max_peaks, and summarize them here https://docs.google.com/presentation/d/1xN2dtMREGvTbvAv0MWhp_ONxJG4s-MpCYFxve9a1imA/edit?usp=sharing. We will keep it updated in case we find more. On denoised 7min gradient the large discrepancy seems to be not observed.

Hope it helps!

Bests,
Zixuan

[zix-xiao]

Thanks for the details - having the simulated raw data, and the raw data from @zix-xiao would be very useful for debugging. There have been quite a few changes since v14.7 to the Bruker side of things, so hopefully we can pin down where the regression is.

It's a beta version for a reason, thanks for helping to test it out!

Hi @lazear, here is the data that generates the result from the first two figures https://syncandshare.lrz.de/getlink/fiKZ61tX3JzvWDDD5UEUZ2/ In the figures the searches were done on multiple technical replicates, here for convenience I only uploaded one replicate and the fasta file. Please let me know if you want more.

Bests,
Zixuan

[theGreatHerrLebert]

Hello again,

I gave it a try and increased the number of peaks allowed per fragment spectrum (now 1500), as suggested by @sander-willems-bruker. Unfortunately, this didn’t resolve the issue. I was somewhat expecting that, since the number of peaks per spectrum in version 0.14.7 was configured the same way.

Could this perhaps be related to how the data is read in or how the query is prepared? I ran the same search using a sagepy build based on SAGE-V0.15.1-beta, where we perform some custom read-in and pre-processing. Using the dataset provided by @zix-xiao, I got the following:

We tried to match the hyperparameters of the SAGE search as closely as possible.

Hope this helps!

Best,
David

[sander-willems-bruker]

In this particular use case (i.e. NOT APPLICABLE TO EVERY OTHER SAMPLE), the quick fix is to add the following to your sage config:

  "bruker_config": {
    "ms2": {
      "spectrum_processing_params": {
        "smoothing_window": 1,
        "centroiding_window": 1,
        "calibration_tolerance": 0.1,
        "calibrate": true //NOTE THAT THIS IS THE ONLY NON_DEFAULT PARAM
      },
      "frame_splitting_params": {
        "Quadrupole": {
          "Even": 1
        }
      }
    },
    "ms1": {
      "mz_ppm": 5.0,
      "ims_pct": 3.0
    }
  },

There is quite some context to be given here, but this requires some internal discussion first. Just though I should not keep you waiting if we can resolve the particular issue at hand already:)

[theGreatHerrLebert]

Thanks a lot for looking into it, this indeed sloved the issue 😊! We now have:

We can indeed use it for now!

[zix-xiao]

In this particular use case (i.e. NOT APPLICABLE TO EVERY OTHER SAMPLE), the quick fix is to add the following to your sage config:

"bruker_config": {
"ms2": {
"spectrum_processing_params": {
"smoothing_window": 1,
"centroiding_window": 1,
"calibration_tolerance": 0.1,
"calibrate": true //NOTE THAT THIS IS THE ONLY NON_DEFAULT PARAM
},
"frame_splitting_params": {
"Quadrupole": {
"Even": 1
}
}
},
"ms1": {
"mz_ppm": 5.0,
"ims_pct": 3.0
}
},
There is quite some context to be given here, but this requires some internal discussion first. Just though I should not keep you waiting if we can resolve the particular issue at hand already:)

Thanks a lot for the quick fix, does solved the problem! It would be quite interesting to learn about the context once it's decided that it can be shared, so we also know what to do depending on how the data is acquired:)