Fatal Python error: Aborted

[TakashiKoyama]

Hello,
I successfully installed pyrho using conda as described in README but got error when running tests.py.
The error messages are as below.
Could anyone help me solving this problem?
Thanks.

$ python -m pytest pyrho/tests/tests.py 
============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.8.11, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/samba/Suijitsu_Public/script/python/pyrho
plugins: anyio-4.3.0
collected 24 items                                                                                                                                                                                                                                                             

pyrho/tests/tests.py ..Fatal Python error: Aborted

Current thread 0x00007f52fcfb9740 (most recent call first):
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/tables/file.py", line 750 in __init__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/tables/file.py", line 300 in open_file
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pandas/io/pytables.py", line 731 in open
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pandas/io/pytables.py", line 579 in __init__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pandas/io/pytables.py", line 420 in read_hdf
  File "/home/samba/Suijitsu_Public/script/python/pyrho/tests/tests.py", line 63 in test_downsample
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/python.py", line 1632 in runtest
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/runner.py", line 341 in from_call
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/main.py", line 339 in _main
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 87 in _run_code
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 194 in _run_module_as_main
Aborted (core dumped)

[jeffspence]

Hmmm -- that's a tough one to debug. Do you get errors when you run the example.sh script in the example directory?

[jeffspence]

Could also be a caching issue with numba --- which version of numba do you have (e.g., by running pip freeze | grep numba)?

[TakashiKoyama]

@jeffspence
Thank you for your reply.
example.sh does not work properly. The end of stdout say HDF version is wrong.
Am I correct?

INFO:root:Finished likelihoods in 0.723522 seconds for rho=0.000000
INFO:root:Cleaning up results...
INFO:root:Computed lookup table in 152.569932 seconds 
INFO:root:	...complete
INFO:root:Downsampling
INFO:root:Downsampling...  Currently at n = 25
INFO:root:Downsampling...  Currently at n = 24
INFO:root:Downsampling...  Currently at n = 23
INFO:root:Downsampling...  Currently at n = 22
INFO:root:Downsampling...  Currently at n = 21
INFO:root:	...complete
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.12.2, library is 1.14.2
        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
                   HDF5 Version: 1.14.2
                  Configured on: Tue Mar 26 22:23:22 JST 2024
                  Configured by: root@XW-E5S.suijitsu.utokyo.jp
                    Host system: x86_64-pc-linux-gnu
              Uname information: Linux XW-E5S.suijitsu.utokyo.jp 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
                       Byte sex: little-endian
             Installation point: /usr/local/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: 
                     H5_LDFLAGS: 
                     AM_LDFLAGS: 
                Extra libraries: -lpthread -lz -ldl -lm 
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /opt/rh/devtoolset-8/root/usr/bin/gcc ( gcc (GCC) 8.3.1 20190311 )
                       CPPFLAGS: 
                    H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L   -DNDEBUG -UH5_DEBUG_API -I/root/Downloads/hdfsrc/src/H5FDsubfiling
                    AM_CPPFLAGS: 
                        C Flags: 
                     H5 C Flags:  -std=c99  -Wall -Wcast-qual -Wconversion -Wextra -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-include-dirs -Wshadow -Wundef -Wwrite-strings -pedantic -Wno-c++-compat -Wlarger-than=2560 -Wlogical-op -Wframe-larger-than=16384 -Wpacked-bitfield-compat -Wsync-nand -Wno-unsuffixed-float-constants -Wdouble-promotion -Wtrampolines -Wstack-usage=8192 -Wmaybe-uninitialized -Wdate-time -Warray-bounds=2 -Wc99-c11-compat -Wduplicated-cond -Whsa -Wnormalized -Wnull-dereference -Wunused-const-variable -Walloca -Walloc-zero -Wduplicated-branches -Wformat-overflow=2 -Wformat-truncation=1 -Wattribute-alias -Wcast-align=strict -Wshift-overflow=2 -fstdarg-opt  -s  -Wbad-function-cast -Wcast-align -Wformat -Wimplicit-function-declaration -Wint-to-pointer-cast -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-sign -Wpointer-to-int-cast -Wredundant-decls -Wstrict-prototypes -Wswitch -Wunused-but-set-variable -Wunused-variable -Wunused-function -Wunused-parameter -Wincompatible-pointer-types -Wint-conversion -Wshadow -Wrestrict -Wcast-function-type -Wmaybe-uninitialized -Wno-aggregate-return -Wno-inline -Wno-missing-format-attribute -Wno-missing-noreturn -Wno-overlength-strings -Wno-jump-misses-init -Wstrict-overflow=2 -Wno-suggest-attribute=const -Wno-suggest-attribute=noreturn -Wno-suggest-attribute=pure -Wno-suggest-attribute=format -Wno-suggest-attribute=cold -Wno-suggest-attribute=malloc -O3
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                     Parallel HDF5: no
  Parallel Filtered Dataset Writes: no
                Large Parallel I/O: no
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: yes
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib)
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
            Function stack tracing: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no
Bye...
example.sh: line 30: 12929 Aborted                 (core dumped) pyrho make_table -n 20 -N 25 --mu 1.25e-8 --logfile . --outfile ACB_n_20_N_40_lookuptable.hdf --approx --smcpp_file ACB_pop_sizes.csv --decimate_rel_tol 0.1
Traceback (most recent call last):
  File "/usr/local/src/pyenv/versions/3.8.11/bin/pyrho", line 8, in <module>
    sys.exit(main())
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pyrho/frontend.py", line 67, in main
    func(args)
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pyrho/hyperparameter_optimizer.py", line 194, in _main
    table = read_hdf(args.tablefile, 'ldtable')
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pandas/io/pytables.py", line 418, in read_hdf
    raise FileNotFoundError(f"File {path_or_buf} does not exist")
FileNotFoundError: File ACB_n_20_N_40_lookuptable.hdf does not exist
INFO:root:Reading data from ACB_chr_1_subset.vcf.gz
INFO:root:	Done!
INFO:root:Loading lookup table...
Traceback (most recent call last):
  File "/usr/local/src/pyenv/versions/3.8.11/bin/pyrho", line 8, in <module>
    sys.exit(main())
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pyrho/frontend.py", line 67, in main
    func(args)
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pyrho/optimizer.py", line 350, in _main
    table = read_hdf(args.tablefile, 'ldtable')
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pandas/io/pytables.py", line 418, in read_hdf
    raise FileNotFoundError(f"File {path_or_buf} does not exist")
FileNotFoundError: File ACB_n_20_N_40_lookuptable.hdf does not exist
Traceback (most recent call last):
  File "/usr/local/src/pyenv/versions/3.8.11/bin/pyrho", line 8, in <module>
    sys.exit(main())
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pyrho/frontend.py", line 67, in main
    func(args)
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pyrho/compute_r2.py", line 97, in _main
    table = DataFrame(read_hdf(args.tablefile, 'ldtable'))
  File "/usr/local/src/pyenv/versions/3.8.11/lib/python3.8/site-packages/pandas/io/pytables.py", line 418, in read_hdf
    raise FileNotFoundError(f"File {path_or_buf} does not exist")
FileNotFoundError: File ACB_n_20_N_40_lookuptable.hdf does not exist

[TakashiKoyama]

@jeffspence
The version of number is numba==0.58.1.

[jeffspence]

Hi @TakashiKoyama -- yes, that looks like some sort of mismatch in hdf5 versioning (and not a pyrho problem, per se). This SO post suggests that maybe resinstalling h5py could help: https://stackoverflow.com/questions/45411700/warning-hdf5-library-version-mismatched-error-python-pandas-windows. It might also be necessary to update or downgrade the version of hfd5 on your system.