Problem using quantms outputs with ibaqpy

[MathiasJoensson]

Hi, I have used quantms to process a small proteomics experiment.
nextflow run bigbio/quantms -r dev -profile singularity --input data/processed/run2/sdrf.tsv --database data/external/combined_proteome.fasta --outdir data/processed/run2/

I would like to use ibaqpy to estimate the absolute quantification of the proteins. However, to run 'quantmsioc convert-diann', 'ibaqpyc features2peptides', and 'ibaqpyc peptides2protein' I need to do some processing of various files. I have attached the command line outputs, and the python scripts I used to process the files.

Specifically, I need to:

1. Remove sequences containing 'X' from diann_report.tsv.
2. Rename the column 'precursor_mz' to 'observed_mz' for all parquet files in the mzmlstatistics folder.
3. After running quantmsioc convert-diann, I need to explode the intensities column in the feature.parquet file.

Please let me know if I am using the wrong input files, or how to streamline the process going from quantms outputs to ibaqpyc peptides2protein outputs.

Many thanks,
Mathias

ibaqpy_terminal_and_output.pdf
absolute_prot_quant.pdf

[enryH]

I encountered also some issues with DDA outputs: bigbio/qpx#97

Could you copy the output into blocks of code it would ease the discussion here (using code blocks, see here

[MathiasJoensson]

Hi, thanks for the response! Here's the first error I get.

(bigbio-quantms) [HPC]$ quantmsioc convert-diann --report_path data/processed/run2/diannsummary/diann_report.tsv --qvalue_threshold 0.05 --mzml_info_folder data/processed/run2/mzmlstatistics/ --sdrf_path data/processed/run2/pipelineinfo/sdrf.tsv --output_folder data/processed/run2/ --output_prefix file diannconvert

/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pyopenms/__init__.py:41: UserWarning: Warning: OPENMS_DATA_PATH environment variable already exists. pyOpenMS will use it (/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/share/OpenMS/) to locate data in the OpenMS share folder (e.g., the unimod database), instead of the default (/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pyopenms/share/OpenMS).
  warnings.warn(
INFO:root:duckdb uses 805.5 GiB threads.
INFO:root:duckdb uses 128 of memory.
100% ▕████████████████████████████████████████████████████████████▏
INFO:root:Time to create duckdb database 4.7445056438446045 seconds
Traceback (most recent call last):
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/bin/quantmsioc", line 8, in <module>
    sys.exit(quantmsio__main())
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/quantmsioc.py", line 79, in quantmsio_main
    cli()
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/commands/diann_diann.write_command.py", line 112, in diann_convert_to_parquet
    feature_to_file(
      qvalue_threshold=qvalue_threshold,
      ...
      protein_file=protein_file,
    )
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/core/diann.py", line 383, in write_feature_to_file
    for report in self.generate_feature(
      qvalue_threshold, mzml_info_folder, file_num, protein_str
    ):
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/core/diann.py", line 342, in generate_feature
    for report in self.main_report_df(
      qvalue_threshold, mzml_info_folder, file_num, protein_str
    ):
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/core/diann.py", line 199, in main_report_df
    masses_map, modifications_map = self.get_masses_and_modifications_map()
  File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/core/diann.py", line 75, in get_masses_and_modifications_map
    masses_map = {k: AASequence.fromString(k).getMonoWeight() for k in uniq_p}
  File "pyopenms/_pyopenms_5.pyx", line 1051, in pyopenms._pyopenms_5.AASequence.getMonoWeight
  File "pyopenms/_pyopenms_5.pyx", line 1020, in pyopenms._pyopenms_5.AASequence._getMonoWeight0
RuntimeError: the value 'XVEGWILFVTGVHEEATEEDIHDK' was used but is not valid; Cannot get weight of sequence with unknown AA 'X' with unknown mass.

And I solved it by doing this:

diann_report = pd.read_csv("../data/processed/run2/diannsummary/diann_report.tsv", sep="\t)
filtered = diann_report[~diann_report['Stripped.Sequence'].str.contains('X')]
filtered.to_csv("../data/processed/run2/diannsummary/diann_report_noX.tsv", sep="\t", index=False)

[enryH]

ok. so this on is actually a quantms.io issue. So DIANN can deal with X as an placeholder for amino acids, but quantmsioc cannot.

[jpfeuffer]

This is just a weird nomenclature of Diann.
X is selenomethionine or something like that.
It needs to be replaced with that mass. See code in Diann convert in quantms.
Not sure why this is still there after diannconvert. I feel we should not have X in our outputs even if that is how DIANN calls it.

bigbio/quantms#368

[MathiasJoensson]

Thanks for the response! I had some other issues with ipaqpy, please let me know if you would like me to open new issues instead of discussing here.

After I removed the AA sequences with 'X' in them I ran into the problem of a missing column in the diann data:

(bigbio-quantms) [HPC]$ quantmsioc convert-diann --report
_path
data/processed/run2/diannsummary/diann
_
report
_
noX.tsv --qvalue
threshold 0.05 --mzml
info
folder
_
_
_
data/processed/run2/mzmlstatistics/
--sdrf
_path data/processed/run2/pipeline
_
info/sdrf.tsv --output
_
folder data/processed/run2/
--output
_prefix
file diann
convert
_
_
/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pyopenms/
__
init
__
.py:41: UserWarning: Warning: OPENMS
DATA
_
_
PATH environment variable already exists. pyOpenMS
will use it (/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/share/OpenMS/) to locate
data in the OpenMS share folder (e.g., the unimod database), instead of the default
(/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pyopenms/s
hare/OpenMS).
warnings.warn(
INFO:root:duckdb uses 805.5 GiB threads.
INFO:root:duckdb uses 128 of memory.
100% ▕████████████████████████████████████████████████████████████▏
INFO:root:Time to create duckdb database 4.7528839111328125 seconds
INFO:root:Time to load report 0.5624732971191406 seconds
Traceback (most recent call last):
File "/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/bin/quantmsioc"
<module>
sys.exit(quantms
io
_
_
main())
~~~~~~~~~~~~~~~^^
, line 8, in
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/
quantmsioc.py"
, line 79, in quantms
io
main
_
_
cli()
~~~^^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core
.py"
, line 1442, in
call
__
__
return self.main(*args,
**kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core
.py"
, line 1363, in main
rv = self.invoke(ctx)
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core
.py"
, line 1830, in invoke
return
_process
_
result(sub
_
ctx.command.invoke(sub
_
ctx))
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core
.py"
, line 1226, in invoke
return ctx.invoke(self.callback,
**ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/click/core
.py"
, line 794, in invoke
return callback(*args,
**kwargs)
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/
commands/diann
_
dia
nn.write
command.py"
, line 112, in diann
convert
to
_
_
_parquet
feature
to
_
_
_
_
file(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
qvalue
_
threshold=qvalue
_
threshold,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
protein
_
file=protein
_
file,
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/
core/diann.py"
, line 383, in write
feature
to
file
_
_
_
for report in self.generate
_
feature(
~~~~~~~~~~~~~~~~~~~~~^
qvalue
_
threshold, mzml
info
_
_
folder, file
_
num, protein
str
_
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/
core/diann.py"
, line 342, in generate
feature
_
for report in self.main
_
report
_
df(
~~~~~~~~~~~~~~~~~~~^
qvalue
_
threshold, mzml
info
_
_
folder, file
_
num, protein
str
_
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/
core/diann.py"
File
, line 231, in main
_
report
df
_
results = list(executor.map(integrate
_
msg, refs))
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/concurrent/futures/
_
.py"
, line 619, in result
iterator
_
yield
result
or
_
_
_
cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
base
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/concurrent/futures/
_
base
.py"
, line 317, in
result
or
_
_
_
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
cancel
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/concurrent/futures/
_
.py"
, line 456, in result
return self.
__get
_
result()
~~~~~~~~~~~~~~~~~^^
base
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/concurrent/futures/
_
.py"
, line 401, in
raise self.
__get
result
_
_
exception
base
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/concurrent/futures/threa
d.py"
, line 59, in run
result = self.fn(*self.args,
**self.kwargs)
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/quantmsio/
core/diann.py"
, line 181, in integrate
_
msg
target = pd.read
_parquet(
ms
info
_
_
file, columns=["rt"
,
"scan"
,
"observed
_
mz"]
)
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pandas/io/
parquet.py"
, line 669, in read
_parquet
return impl.read(
~~~~~~~~~^
path,
^^^^^
...<6 lines>...
**kwargs,
^^^^^^^^^
)
^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pandas/io/
parquet.py"
, line 265, in read
pa
_
table = self.api.parquet.read
_
table(
path
or
_
_
handle,
...<3 lines>...
**kwargs,
)
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pyarrow/pa
rquet/core.py"
, line 1824, in read
table
_
return dataset.read(columns=columns, use
threads=use
_
_
threads,
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
use
_pandas
metadata=use
_
_pandas
_
metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/data/gpfs/projects/punim2537/anaconda3/envs/bigbio-quantms/lib/python3.13/site-packages/pyarrow/pa
rquet/core.py"
table = self.
, line 1475, in read
dataset.to
_
_
table(
columns=columns, filter=self.
filter
_
_
expression,
use
threads=use
threads
_
_
)
File "pyarrow/
_
dataset.pyx"
File "pyarrow/
_
dataset.pyx"
File "pyarrow/
_
dataset.pyx"
File "pyarrow/
_
dataset.pyx"
File "pyarrow/
_
dataset.pyx"
File "pyarrow/error.pxi"
, line 579, in pyarrow.
dataset.Dataset.to
table
_
, line 415, in pyarrow.
_
dataset.Dataset.scanner
_
, line 3676, in pyarrow.
dataset.Scanner.from
_
, line 3589, in pyarrow.
dataset.Scanner.
_
_
, line 3539, in pyarrow.
dataset.
_
_populate
_
, line 92, in pyarrow.lib.check
status
_
dataset
_
make
scan
_
_
options
builder
pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(observed
_
mz) in scan: string
ms
level: int32
_
num
_peaks: int32
base
_peak
_
intensity: double
summed
_peak
intensities: double
_
rt: double
precursor
_
charge: int32
precursor
mz: double
_
precursor
rt: double
_
precursor
_
intensity: double
precursor
total
_
_
intensity: double
acquisition
_
datetime: string
__
fragment
index: int32
_
batch
index: int32
__
last
_
in
__
_
_
fragment: bool
__
filename: string

I solved this by renaming the precursor_mz columns in the mzmlstatistics/*.parquet files to 'observed_mz':

parquet_files = glob.glob("../data/processed/run2/mzmlstatistics/*.parquet")
for file in parquet_files:
    df = pd.read_parquet(file)
    if 'precursor_mz' in df.columns and 'observed_mz' not in df.columns:
        df = df.rename(columns={'precursor_mz': 'observed_mz'})
        df.to_parquet(file)
        print(f"Renamed 'precursor_mz' to 'observed_mz' in {file}")
    else:
        print(f"No change needed for {file}")

Appreciate any help, thanks!