error running compute_distance_matrix_zernike3deep.py

[geoffwoollard]

The file ref_2_16x16x16.npy is shape (2,16**3).

The error seems to be coming from the xmipp ImageHandler, ih.scaleSplines(data=volume, ...).

Probably a version issue.

(flexutils-tensorflow) (base) [gwoollard@workergpu004 Cryo-EM-Heterogeneity-Challenge-1]$ compute_distance_matrix_zernike3deep.py --references_file tmp_zernike/ref_2_16x16x16.npy --targ
ets_file tmp_zernike/ref_2_16x16x16.npy --out_path tmp_zernike/ --gpu 0 --thr 20
2024-11-14 14:56:42.032833: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-11-14 14:56:42.071005: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-11-14 14:56:42.071033: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-11-14 14:56:42.072179: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-14 14:56:42.078630: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-14 14:56:42.853031: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Projecting volumes: 0%| | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/bin/compute_distance_matrix_zernike3deep.py", line 8, in
sys.exit(main())
File "/mnt/ceph/users/gwoollard/repos/Flexutils-Toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 164, in main
compute_distance_matrix(**inputs)
File "/mnt/ceph/users/gwoollard/repos/Flexutils-Toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 61, in compute_distance_matrix
volume = ih.scaleSplines(data=volume, finalDimension=64)
TypeError: scaleSplines() got an unexpected keyword argument 'data'

[geoffwoollard]

(flexutils-tensorflow) (base) [gwoollard@workergpu004 Cryo-EM-Heterogeneity-Challenge-1]$ pip freeze
absl-py==2.1.0
addict==2.4.0
astunparse==1.6.3
attrs==24.2.0
bibtexparser==1.2.0
biopython==1.76
blinker==1.8.2
cachetools==5.5.0
certifi==2024.8.30
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
ConfigArgParse==1.7
contourpy==1.3.0
cycler==0.12.1
dash==2.18.1
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
decorator==5.1.1
dill==0.3.9
dm-tree==0.1.8
emfile==0.3.0
emtable==0.0.14
exceptiongroup==1.2.2
fastjsonschema==2.20.0
Flask==3.0.3
flatbuffers==24.3.25
fonttools==4.54.1
future==1.0.0
gast==0.6.0
google-auth==2.35.0
google-auth-oauthlib==1.2.1
google-pasta==0.2.0
grpcio==1.66.2
h5py==3.12.1
idna==3.10
imageio==2.35.1
importlib_metadata==8.5.0
importlib_resources==6.4.5
iniconfig==2.0.0
itsdangerous==2.2.0
Jinja2==3.1.4
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter_core==5.7.2
keras==2.15.0
kiwisolver==1.4.7
libclang==18.1.1
Markdown==3.7
MarkupSafe==2.1.5
matplotlib==3.8.2
ml-dtypes==0.3.2
morphsnakes==2.0.1
mrcfile==1.4.3
multiprocess==0.70.17
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.2.1
numpy==1.23.5
nvidia-cublas-cu12==12.2.5.6
nvidia-cuda-cupti-cu12==12.2.142
nvidia-cuda-nvcc-cu12==12.2.140
nvidia-cuda-nvrtc-cu12==12.2.140
nvidia-cuda-runtime-cu12==12.2.140
nvidia-cudnn-cu12==8.9.4.25
nvidia-cufft-cu12==11.0.8.103
nvidia-curand-cu12==10.3.3.141
nvidia-cusolver-cu12==11.5.2.141
nvidia-cusparse-cu12==12.1.2.141
nvidia-nccl-cu12==2.16.5
nvidia-nvjitlink-cu12==12.2.140
oauthlib==3.2.2
open3d @ file:///mnt/home/gwoollard/ceph/repos/Open3D/build/lib/python_package/pip_package/open3d-0.18.0%2Bdd0d35956-cp39-cp39-manylinux_2_28_x86_64.whl#sha256=c744177f77b4ea2a3ea1f3f6a627be24dbb14327aec4dceddf9f0d917cc2f711
opt_einsum==3.4.0
packaging==24.1
pandas==2.1.4
pathos==0.3.3
Pillow==10.1.0
platformdirs==4.3.6
plotly==5.24.1
pluggy==1.5.0
pox==0.3.5
ppft==1.7.6.9
ProDy==2.4.1
protobuf==4.23.4
psutil==5.8.0
pyarrow==17.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pyparsing==3.1.4
pyquaternion==0.9.9
pytest==8.3.3
python-dateutil==2.9.0.post0
pytz==2024.2
PyWavelets==1.6.0
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1695373447169/work
referencing==0.35.1
requests==2.32.3
requests-oauthlib==2.0.0
retrying==1.3.4
rpds-py==0.20.0
rsa==4.9
scikit-image==0.19.3
scikit-learn==1.3.2
scipy==1.11.4
seaborn==0.13.2
six==1.16.0
starfile==0.5.8
tenacity==9.0.0
tensorboard==2.15.2
tensorboard-data-server==0.7.2
tensorflow==2.15.1
tensorflow-addons==0.23.0
tensorflow-estimator==2.15.0
tensorflow-io-gcs-filesystem==0.37.1
tensorflow-probability==0.23.0
-e git+https://github.com/I2PC/Flexutils-Toolkit.git@8f7b695520d5b67e86c39efc6915b81c2b31874e#egg=tensorflow_toolkit
termcolor==2.4.0
tf-siren==0.0.5
threadpoolctl==3.5.0
tifffile==2021.3.5
tkcolorpicker==2.1.3
tomli==2.0.2
tqdm==4.66.5
traitlets==5.14.3
typeguard==2.13.3
typing_extensions==4.12.2
tzdata==2024.2
urllib3==2.2.3
Werkzeug==3.0.4
wrapt==1.14.1
xmipp_metadata @ git+https://github.com/DavidHerreros/xmipp_metadata.git@fb37e4a1386706257d956d4f0bfff426ed35b82d
yapf==0.40.2
yellowbrick==1.5
zipp==3.20.2

[geoffwoollard]

@DavidHerreros did you see this?

[DavidHerreros]

Hi @geoffwoollard,

Thanks for reporting this bug!

Yes, it seems the package responsible for scaling the volumes is missing the parameter data. This new argument was part of a new release that has not been uploaded to Pypi yet.

I have now released the package with this new argument. You will need to update it in your environment by running pip install -U xmipp_metadata.

Let me know if this fixes the error!

[geoffwoollard]

Ok made some progress, but hit another bug:

(flexutils-tensorflow) (base) [gwoollard@workergpu005 Cryo-EM-Heterogeneity-Challenge-1]$ compute_distance_matrix_zernike3deep.py --references_file tmp_zernike/reference_maps.npy  --targets_file tmp_zernike/target_maps.npy --out_path tmp_zernike/ --gpu 0 --thr 20
Projecting volumes:   0%|                                                                                                                                                                           | 0/10 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/bin/compute_distance_matrix_zernike3deep.py", line 8, in <module>
    sys.exit(main())
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 166, in main
    compute_distance_matrix(**inputs)
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 62, in compute_distance_matrix
    volume = ih.scaleSplines(data=volume, finalDimension=64)
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/xmipp_metadata/image_handler/image_handler.py", line 228, in scaleSplines
    new_sr = self.getSamplingRate() / scaleFactor
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

The issue is that self.getSamplingRate() returns None

[geoffwoollard]

@DavidHerreros small bug.

The dictionary inputs should contain "numProjections": args.num_projections

https://github.com/I2PC/Flexutils-Toolkit/blob/zernike3deep_dm_fixes/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py#L163

[DavidHerreros]

@DavidHerreros small bug.

The dictionary inputs should contain "numProjections": args.num_projections

https://github.com/I2PC/Flexutils-Toolkit/blob/zernike3deep_dm_fixes/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py#L163

Thanks for the report @geoffwoollard!

I have fixed the typo in the dictionary.

Since everything is now merged in the Flatiron repo, I have merged the PR with the fixes we have been solving these past weeks, so everything is included in the upcoming new release of this package.

Thanks for all the help!!