Inference on macOS Produces Different Scores from Reference Dataset

[stuartmac]

I managed to get protgps running on macOS (M4). Inference completes successfully, but the scores I obtain differ from those in disease_mutations_reference_set_preds.csv from the Zenodo dataset.

• Is this expected due to model nondeterminism, floating-point differences, or another factor?
• Are there any test sequences available to validate the installation and confirm expected outputs?

To get the code running, I removed build-specific constraints from the Conda specification and dropped the CUDA dependencies. Here’s my fork: stuartmac/protgps.

Let me know if you’d be interested in a PR for these changes!

Thanks for your work on this project!

[pgmikhael]

Hi,

Thanks for adapting it to macOS! I'm not exactly sure, but I wouldn't be surprised if there are nondeterministic, floating-point differences.

For testing purposes, the sequences in notebook/Predict.ipynb should have the following scores:

O15116
--------
sequence MNYMPGTASLIEDIDKKHLVLLRDGRTLIGFLRSIDQFANLVLHQTVERIHVGKKYGDIPRGIFVVRGENVVLLGEIDLEKESDTPLQQVSIEEILEEQRVEQQTKLEAEKLKVQALKDRGLSIPRADTLDEY
NUCLEAR_SPECKLE_Score 0.0
P-BODY_Score 0.9990000128746033
PML-BDOY_Score 0.0
POST_SYNAPTIC_DENSITY_Score 0.0
STRESS_GRANULE_Score 0.0
CHROMOSOME_Score 0.0
NUCLEOLUS_Score 0.0010000000474974513
NUCLEAR_PORE_COMPLEX_Score 0.0010000000474974513
CAJAL_BODY_Score 0.0
RNA_GRANULE_Score 0.0
CELL_JUNCTION_Score 0.0
TRANSCRIPTIONAL_Score 0.0

P38432
--------
sequence MAASETVRLRLQFDYPPPATPHCTAFWLLVDLNRCRVVTDLISLIRQRFGFSSGAFLGLYLEGGLLPPAESARLVRDNDCLRVKLEERGVAENSVVISNGDINLSLRKAKKRAFQLEEGEETEPDCKYSKKHWKSRENNNNNEKVLDLEPKAVTDQTVSKKNKRKNKATCGTVGDDNEEAKRKSPKKKEKCEYKKKAKNPKSPKVQAVKDWANQRCSSPKGSARNSLVKAKRKGSVSVCSKESPSSSSESESCDESISDGPSKVTLEARNSSEKLPTELSKEEPSTKNTTADKLAIKLGFSLTPSKGKTSGTTSSSSDSSAESDDQCLMSSSTPECAAGFLKTVGLFAGRGRPGPGLSSQTAGAAGWRRSGSNGGGQAPGASPSVSLPASLGRGWGREENLFSWKGAKGRGMRGRGRGRGHPVSCVVNRSTDNQRQQQLNDVVKNSSTIIQNPVETPKKDYSLLPLLAAAPQVGEKIAFKLLELTSSYSPDVSDYKEGRILSHNPETQQVDIEILSSLPALREPGKFDLVYHNENGAEVVEYAVTQESKITVFWKELIDPRLIIESPSNTSSTEPA
NUCLEAR_SPECKLE_Score 0.0
P-BODY_Score 0.0
PML-BDOY_Score 0.003000000026077032
POST_SYNAPTIC_DENSITY_Score 0.0
STRESS_GRANULE_Score 0.0
CHROMOSOME_Score 0.0
NUCLEOLUS_Score 0.004000000189989805
NUCLEAR_PORE_COMPLEX_Score 0.0020000000949949026
CAJAL_BODY_Score 0.9919999837875366
RNA_GRANULE_Score 0.0
CELL_JUNCTION_Score 0.0
TRANSCRIPTIONAL_Score 0.0

[TomoyaKanno]

Hi, I also got this to work on mac by disabling or changing a few dependencies in the environment file. The results I got from the two test sequences look like this (very similar):

protgps_results-3.xlsx

I'm also working to turn this into an basic interactive web app in Streamlit so it's easier for people to use. I'll post a fork when it's ready.

[stuartmac]

Thanks for the test data @pgmikhael

For those, I get the same results 👍

However, I ran a more extensive comparison with the disease_mutations_reference_set_preds.csv data from Zenodo, and in this case, the predictions I got were different. Would you expect this?