Skip to content

[UPDATE] export parquet, include score tables into export if present #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
singjc merged 18 commits into from
Oct 29, 2025
Merged

[UPDATE] export parquet, include score tables into export if present #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
a9eb975
Initial plan
Copilot Oct 29, 2025
7af902f
Initial exploration of pyprophet export parquet code
Copilot Oct 29, 2025
12c3ba0
Add support for SCORE_ tables and include_transition_data flag in par…
Copilot Oct 29, 2025
9fea41f
Refactor: extract helper method to reduce code duplication
Copilot Oct 29, 2025
1104aae
Fix comment to match code behavior
Copilot Oct 29, 2025
77fb510
Fix Python 3.9 compatibility: use Tuple instead of tuple for type hints
Copilot Oct 29, 2025
adac489
Fix: filter out columns without types when building temp table schema
Copilot Oct 29, 2025
7066857
Fix: add RUN_ID to score peptide/protein views for proper joining
Copilot Oct 29, 2025
28fd32e
Fix: use LEFT JOIN for score views to avoid filtering out rows withou…
Copilot Oct 29, 2025
6687a89
Add tests for parquet export with SCORE_ tables and include_transitio…
Copilot Oct 29, 2025
1a31938
update: regtest for new test
singjc Oct 29, 2025
3a6bab4
fix tests
singjc Oct 29, 2025
811462a
Merge pull request #3 from singjc/copilot/update-export-parquet-method
singjc Oct 29, 2025
2938add
Initial plan
Copilot Oct 29, 2025
14aefb8
Add SCORE_IPF table to OSW parquet export
Copilot Oct 29, 2025
589d694
Add test for SCORE_IPF columns in parquet export
Copilot Oct 29, 2025
c6f274c
add: regtest output
singjc Oct 29, 2025
bd43a91
Merge pull request #4 from singjc/copilot/fix-osw-export-score-ipf-table
singjc Oct 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pyprophet/_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,7 @@ class ExportIOConfig(BaseIOConfig):
compression_level: int = 11
split_transition_data: bool = True
split_runs: bool = False
include_transition_data: bool = True # Whether to include transition data in parquet export

# SqMass: Export to parquet
pqp_file: Optional[str] = None # Path to PQP file for precursor/transition mapping
Expand Down
9 changes: 9 additions & 0 deletions pyprophet/cli/export.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -553,6 +553,13 @@ def export_library(
type=int,
help="Compression level to use for parquet file.",
)
@click.option(
"--include_transition_data/--no-include_transition_data",
"include_transition_data",
default=True,
show_default=True,
help="Include transition data in the exported parquet file(s). When disabled, only precursor-level data is exported.",
)
@measure_memory_usage_and_time
def export_parquet(
infile,
Expand All @@ -565,6 +572,7 @@ def export_parquet(
split_runs,
compression,
compression_level,
include_transition_data,
):
"""
Export OSW or sqMass to parquet format
Expand Down Expand Up @@ -600,6 +608,7 @@ def export_parquet(
split_runs=split_runs,
compression_method=compression,
compression_level=compression_level,
include_transition_data=include_transition_data,
)

writer = WriterDispatcher.get_writer(config)
Expand Down
198 changes: 165 additions & 33 deletions pyprophet/io/export/osw.py
View file
Open in desktop

Large diffs are not rendered by default.

726 changes: 383 additions & 343 deletions pyprophet/scoring/_optimized.c
View file
Open in desktop

Large diffs are not rendered by default.

13 changes: 13 additions & 0 deletions tests/_regtest_outputs/test_pyprophet_export.test_parquet_export_no_transition_data.out
View file
Open in desktop

Large diffs are not rendered by default.

13 changes: 13 additions & 0 deletions tests/_regtest_outputs/test_pyprophet_export.test_parquet_export_scored_osw.out
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions tests/_regtest_outputs/test_pyprophet_export.test_parquet_export_split_format.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Precursor data: 3410 rows with 79 columns
Transition data: 96259 rows with 23 columns
Precursor score columns: ['SCORE_MS2_PEAK_GROUP_RANK', 'SCORE_MS2_PEP', 'SCORE_MS2_P_VALUE', 'SCORE_MS2_Q_VALUE', 'SCORE_MS2_SCORE', 'SCORE_PEPTIDE_GLOBAL_PEP', 'SCORE_PEPTIDE_GLOBAL_P_VALUE', 'SCORE_PEPTIDE_GLOBAL_Q_VALUE', 'SCORE_PEPTIDE_GLOBAL_SCORE', 'SCORE_PROTEIN_GLOBAL_PEP', 'SCORE_PROTEIN_GLOBAL_P_VALUE', 'SCORE_PROTEIN_GLOBAL_Q_VALUE', 'SCORE_PROTEIN_GLOBAL_SCORE']
Precursor data sample:
DELTA_RT EXP_IM EXP_RT FEATURE_ID FEATURE_MS1_APEX_INTENSITY FEATURE_MS1_AREA_INTENSITY FEATURE_MS1_VAR_ISOTOPE_CORRELATION_SCORE FEATURE_MS1_VAR_ISOTOPE_OVERLAP_SCORE FEATURE_MS1_VAR_MASSDEV_SCORE FEATURE_MS1_VAR_XCORR_COELUTION FEATURE_MS1_VAR_XCORR_SHAPE FEATURE_MS2_APEX_INTENSITY FEATURE_MS2_AREA_INTENSITY FEATURE_MS2_VAR_BSERIES_SCORE FEATURE_MS2_VAR_DOTPROD_SCORE FEATURE_MS2_VAR_ELUTION_MODEL_FIT_SCORE FEATURE_MS2_VAR_INTENSITY_SCORE FEATURE_MS2_VAR_ISOTOPE_CORRELATION_SCORE FEATURE_MS2_VAR_ISOTOPE_OVERLAP_SCORE FEATURE_MS2_VAR_LIBRARY_CORR FEATURE_MS2_VAR_LIBRARY_DOTPROD FEATURE_MS2_VAR_LIBRARY_MANHATTAN FEATURE_MS2_VAR_LIBRARY_RMSD FEATURE_MS2_VAR_LIBRARY_ROOTMEANSQUARE FEATURE_MS2_VAR_LIBRARY_SANGLE FEATURE_MS2_VAR_LOG_SN_SCORE FEATURE_MS2_VAR_MANHATTAN_SCORE FEATURE_MS2_VAR_MASSDEV_SCORE FEATURE_MS2_VAR_MASSDEV_SCORE_WEIGHTED FEATURE_MS2_VAR_NORM_RT_SCORE FEATURE_MS2_VAR_SONAR_LAG FEATURE_MS2_VAR_SONAR_LOG_DIFF FEATURE_MS2_VAR_SONAR_LOG_SN FEATURE_MS2_VAR_SONAR_LOG_TREND FEATURE_MS2_VAR_SONAR_RSQ FEATURE_MS2_VAR_SONAR_SHAPE FEATURE_MS2_VAR_XCORR_COELUTION FEATURE_MS2_VAR_XCORR_COELUTION_WEIGHTED FEATURE_MS2_VAR_XCORR_SHAPE FEATURE_MS2_VAR_XCORR_SHAPE_WEIGHTED FEATURE_MS2_VAR_YSERIES_SCORE FILENAME GENE_DECOY GENE_ID GENE_NAME IPF_PEPTIDE_ID LEFT_WIDTH MODIFIED_SEQUENCE NORM_RT PEPTIDE_DECOY PEPTIDE_ID PRECURSOR_CHARGE PRECURSOR_DECOY PRECURSOR_GROUP_LABEL PRECURSOR_ID PRECURSOR_LIBRARY_DRIFT_TIME PRECURSOR_LIBRARY_INTENSITY PRECURSOR_LIBRARY_RT PRECURSOR_MZ PRECURSOR_TRAML_ID PROTEIN_ACCESSION PROTEIN_DECOY PROTEIN_ID RIGHT_WIDTH RUN_ID SCORE_MS2_PEAK_GROUP_RANK SCORE_MS2_PEP SCORE_MS2_P_VALUE SCORE_MS2_Q_VALUE SCORE_MS2_SCORE SCORE_PEPTIDE_GLOBAL_PEP SCORE_PEPTIDE_GLOBAL_P_VALUE SCORE_PEPTIDE_GLOBAL_Q_VALUE SCORE_PEPTIDE_GLOBAL_SCORE SCORE_PROTEIN_GLOBAL_PEP SCORE_PROTEIN_GLOBAL_P_VALUE SCORE_PROTEIN_GLOBAL_Q_VALUE SCORE_PROTEIN_GLOBAL_SCORE UNMODIFIED_SEQUENCE
0 -85.0733 NaN 1923.17 483971408708572459 192394.8869 935372.0 0.9919 0.1179 3.1724 0.0000 0.9713 61269.0 321681.0 5.0 0.7829 NaN 0.3415 0.9953 0.0000 0.9944 0.9786 0.2234 0.0893 0.1044 0.2119 3.7524 0.7034 2.8770 1.0938 0.0237 NaN NaN NaN NaN NaN NaN 0.0000 0.0000 0.9660 0.9924 6.0 napedro_L120420_010_SW.mzXML.gz NaN NaN NaN 522 1898.67 GIGDWSDSK(UniMod:259) 7.0277 0 523 2 0 AQUA4SWATH_HMLangeF_GIGDWSDSK(UniMod:259)/2 119 NaN NaN 9.4 486.7293 AQUA4SWATH_HMLangeF_GIGDWSDSK(Label:13C(6)15N(... AQUA4SWATH_HMLangeF NaN 5 1946.4600 -8670811102654834151 1 0.0031 0.0029 0.0033 4.6997 NaN NaN NaN NaN NaN NaN NaN NaN GIGDWSDSK
1 -55.2126 NaN 1953.03 6854889104354289238 5696.7273 45882.6 0.2886 1.6923 20.0342 5.1458 0.5950 7999.0 45147.0 3.0 0.5879 NaN 0.0479 0.6040 0.6485 0.9860 0.9809 0.2040 0.0854 0.1024 0.2154 1.8184 0.8426 7.5457 10.7048 0.0151 NaN NaN NaN NaN NaN NaN 3.4670 1.4506 0.6956 0.8139 3.0 napedro_L120420_010_SW.mzXML.gz NaN NaN NaN 522 1946.46 GIGDWSDSK(UniMod:259) 7.8936 0 523 2 0 AQUA4SWATH_HMLangeF_GIGDWSDSK(UniMod:259)/2 119 NaN NaN 9.4 486.7293 AQUA4SWATH_HMLangeF_GIGDWSDSK(Label:13C(6)15N(... AQUA4SWATH_HMLangeF NaN 5 1977.1899 -8670811102654834151 4 1.0000 0.4692 0.4692 -1.7930 NaN NaN NaN NaN NaN NaN NaN NaN GIGDWSDSK
2 9.7944 NaN 2018.03 2696300170322160855 17401.1825 95751.8 0.9301 0.5084 13.5151 0.7500 0.6966 2243.0 13809.0 4.0 0.5880 NaN 0.0147 -0.0614 0.1439 -0.4161 0.7216 0.6824 0.3240 0.4081 1.2232 0.0221 0.9147 2.4447 2.0283 0.0038 NaN NaN NaN NaN NaN NaN 1.3498 0.4384 0.8111 0.8839 4.0 napedro_L120420_010_SW.mzXML.gz NaN NaN NaN 522 2001.09 GIGDWSDSK(UniMod:259) 9.7785 0 523 2 0 AQUA4SWATH_HMLangeF_GIGDWSDSK(UniMod:259)/2 119 NaN NaN 9.4 486.7293 AQUA4SWATH_HMLangeF_GIGDWSDSK(Label:13C(6)15N(... AQUA4SWATH_HMLangeF NaN 5 2024.9800 -8670811102654834151 2 1.0000 0.4692 0.4692 -0.3786 NaN NaN NaN NaN NaN NaN NaN NaN GIGDWSDSK
3 -130.8641 NaN 1877.37 8207933629855485114 6239.5198 48788.5 -0.5293 1.7491 3.3984 3.2500 0.5946 3336.0 36324.0 3.0 0.4316 NaN 0.0386 0.1794 0.2909 -0.3937 0.8019 0.6135 0.2909 0.3399 1.0151 0.6018 1.1139 5.2642 1.9825 0.0370 NaN NaN NaN NaN NaN NaN 2.2472 0.8549 0.7655 0.8558 4.0 napedro_L120420_010_SW.mzXML.gz NaN NaN NaN 522 1857.70 GIGDWSDSK(UniMod:259) 5.7000 0 523 2 0 AQUA4SWATH_HMLangeF_GIGDWSDSK(UniMod:259)/2 119 NaN NaN 9.4 486.7293 AQUA4SWATH_HMLangeF_GIGDWSDSK(Label:13C(6)15N(... AQUA4SWATH_HMLangeF NaN 5 1898.6700 -8670811102654834151 3 1.0000 0.4692 0.4692 -1.5525 NaN NaN NaN NaN NaN NaN NaN NaN GIGDWSDSK
4 -268.6805 NaN 1739.56 745237666153652118 6493.7774 66798.4 -0.4011 0.7500 16.0151 4.5817 0.6404 7539.0 53232.0 3.0 0.5633 NaN 0.0565 0.3089 0.5266 -0.7130 0.6301 0.8349 0.3552 0.4363 1.3366 1.4128 0.9671 6.3286 10.9637 0.0770 NaN NaN NaN NaN NaN NaN 2.8670 1.1513 0.7267 0.8314 3.0 napedro_L120420_010_SW.mzXML.gz NaN NaN NaN 522 1717.73 GIGDWSDSK(UniMod:259) 1.7038 0 523 2 0 AQUA4SWATH_HMLangeF_GIGDWSDSK(UniMod:259)/2 119 NaN NaN 9.4 486.7293 AQUA4SWATH_HMLangeF_GIGDWSDSK(Label:13C(6)15N(... AQUA4SWATH_HMLangeF NaN 5 1762.1100 -8670811102654834151 5 1.0000 0.4692 0.4692 -3.2559 NaN NaN NaN NaN NaN NaN NaN NaN GIGDWSDSK
14 changes: 14 additions & 0 deletions tests/_regtest_outputs/test_pyprophet_export.test_parquet_export_with_ipf.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
Exported 97965 rows with 103 columns
SCORE_IPF columns found: ['SCORE_IPF_PEP', 'SCORE_IPF_PRECURSOR_PEAKGROUP_PEP', 'SCORE_IPF_QVALUE']
Sample data with IPF scores:
FEATURE_ID SCORE_IPF_PEP SCORE_IPF_PRECURSOR_PEAKGROUP_PEP SCORE_IPF_QVALUE
0 4.8397e+17 0.0000e+00 2.1927e-07 0.0000e+00
1 1.0824e+18 0.0000e+00 9.9998e-08 0.0000e+00
2 -1.1854e+18 0.0000e+00 4.5323e-08 0.0000e+00
3 6.8070e+18 0.0000e+00 1.5978e-09 0.0000e+00
4 7.1486e+18 0.0000e+00 1.1662e-08 0.0000e+00
5 9.0780e+17 2.5734e-10 1.6434e-04 9.0990e-13
6 2.4200e+18 0.0000e+00 1.7179e-08 0.0000e+00
7 -1.4753e+18 0.0000e+00 1.5978e-09 0.0000e+00
8 5.4169e+18 0.0000e+00 4.0794e-08 0.0000e+00
9 -3.0355e+17 0.0000e+00 2.0475e-08 0.0000e+00
Loading