Blas GEMM launch failed

[Zyj061]

When test the scirtp `generate_encoding.py', it always interrupt with the error: Blas GEMM launch failed: a.shape=(56, 512), b.shape](512,512), m=56, k=512. We test the code with GPU 3090 that has 24G free memory, so I think it is not due to the out of memory.

[brucejwittmann]

Could you provide a little more information on the way you are running the script? I haven't seen this error before, so it is likely a problem with one of the dependencies (tensorflow most likely). Information that would be helpful for me to assist:

1. Conda environment used to run the code (including CUDA version)
2. Specific commands sent to the script
3. The full traceback

[Zyj061]

Thank you for your reply. Please see below for information about the script we run:

1. The Conda environment is mlde, which is created by your tutorial using mlde.yml.
2. The comman is

python generate_encoding.py transformer GB1_T2Q --fasta
./code/validation/basic_test_data/2GI9.fasta --positions V39 D40 --batches 1

3. The full trackback:
   
   
   

[brucejwittmann]

Ok, so it looks like the error begins while calling run_embed.py from TAPE. The error that you're seeing is well documented on the tensorflow GitHub (Linke here). The most recommended solutions seem to be:

1. Restarting the computer you're seeing the issue on.
2. Closing all other programs that are using the same GPU. You can check what other programs are using your GPU by calling nvidia-smi.
3. Allowing GPU memory to grow during code execution.

I always find that 1 is a good strategy whenever something strange is going on with CUDA, pytorch, or tensorflow. I do not believe that 2 and 3 should be a problem in MLDE and TAPE (otherwise many other people would have run into this problem already), but adding code to allow memory growth is always an option to try.

Some other tests that you can run to try and narrow down where the problem is coming from:

1. MLDE should have produced the fasta files for embedding before it crashed. Try running tape-embed outside of MLDE to see if you get the same error. Instructions for TAPE and the tape-embed command are here. You should run this in the mlde environment.
2. Try embedding with different TAPE models. See if you get the same error for all models.
3. Try embedding with the ESM models instead. ESM relies on torch, so you would not expect to see any problems if tensorflow is indeed the culprit. To do this, run in the mlde2 environment and run

python generate_encoding.py esm1_t6_43M_UR50S GB1_T2Q --fasta ./code/validation/basic_test_data/2GI9.fasta --positions V39 D40

[tortoisetsunami]

@brucejwittmann I ran into the same error.

This is my trackback:

(mlde) tamima@LAPTOP-V0IMUSP0:~/MLDE/MLDE$ python generate_encoding.py transformer GB1_T2Q --fasta ./code/validation/basic_test_data/2GI9.fasta --positions V39 D40 --batches 1
Batch#:   0%|                                                                    | 0/1 [00:00<?, ?it/s]usage: tape-embed [-h] [--model_config_file MODEL_CONFIG_FILE]
                  [--vocab_file VOCAB_FILE] [--output_dir OUTPUT_DIR]
                  [--no_cuda] [--seed SEED] [--local_rank LOCAL_RANK]
                  [--tokenizer {iupac,unirep}] [--num_workers NUM_WORKERS]
                  [--log_level {DEBUG,INFO,WARN,WARNING,ERROR,10,20,30,40}]
                  [--debug] [--batch_size BATCH_SIZE] [--full_sequence_embed]
                  model_type data_file out_file from_pretrained
tape-embed: error: the following arguments are required: from_pretrained
Batch#:   0%|                                                                    | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "generate_encoding.py", line 65, in <module>
    main()
  File "generate_encoding.py", line 61, in main
    batch_size = args.batch_size)
  File "/mnt/c/Users/tamim/OneDrive/Documents/CS/Cambronne/MLDE/MLDE/code/encode/encoding_generator.py", line 332, in generate_encodings
    unnormalized_embeddings = self._generate_tape(n_batches)
  File "/mnt/c/Users/tamim/OneDrive/Documents/CS/Cambronne/MLDE/MLDE/code/encode/encoding_generator.py", line 284, in _generate_tape
    with open(temp_filename, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/c/Users/tamim/OneDrive/Documents/CS/Cambronne/MLDE/MLDE/20241004-134729/Encodings/GB1_T2Q_transformer_TempOutputs.pkl'

Restarting the computer did not change my error message.

I tried to run #1 of your tests with tape-embed and got this error:

(mlde) tamima@LAPTOP-V0IMUSP0:~/MLDE/MLDE$ tape-embed ./code/validation/basic_test_data/2GI9.fasta --model transformer --load-from transformer_weights.h5
usage: tape-embed [-h] [--model_config_file MODEL_CONFIG_FILE]
                  [--vocab_file VOCAB_FILE] [--output_dir OUTPUT_DIR]
                  [--no_cuda] [--seed SEED] [--local_rank LOCAL_RANK]
                  [--tokenizer {iupac,unirep}] [--num_workers NUM_WORKERS]
                  [--log_level {DEBUG,INFO,WARN,WARNING,ERROR,10,20,30,40}]
                  [--debug] [--batch_size BATCH_SIZE] [--full_sequence_embed]
                  model_type data_file out_file from_pretrained
tape-embed: error: argument --model_config_file: File path: transformer is not a valid file

I tried to run it with a different model and it had the same error:

(mlde) tamima@LAPTOP-V0IMUSP0:~/MLDE/MLDE$ tape-embed ./code/validation/basic_test_data/2GI9.fasta --model georgiev --load-from transformer_weights.h5       
usage: tape-embed [-h] [--model_config_file MODEL_CONFIG_FILE]
                  [--vocab_file VOCAB_FILE] [--output_dir OUTPUT_DIR]
                  [--no_cuda] [--seed SEED] [--local_rank LOCAL_RANK]
                  [--tokenizer {iupac,unirep}] [--num_workers NUM_WORKERS]
                  [--log_level {DEBUG,INFO,WARN,WARNING,ERROR,10,20,30,40}]
                  [--debug] [--batch_size BATCH_SIZE] [--full_sequence_embed]
                  model_type data_file out_file from_pretrained
tape-embed: error: argument --model_config_file: File path: georgiev is not a valid file

I ran the code you gave for #3 in the mlde2 environment, and it worked! Does this mean I downloaded the wrong tensorflow packages?
However when I checked, it said I have the correct tensorflow-gpu version:

(mlde) tamima@LAPTOP-V0IMUSP0:~/MLDE/MLDE$ pip show tensorflow-gpu
Name: tensorflow-gpu
Version: 1.13.1
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /home/tamima/anaconda3/envs/mlde/lib/python3.7/site-packages
Requires: absl-py, astor, gast, grpcio, keras-applications, keras-preprocessing, numpy, protobuf, six, tensorboard, tensorflow-estimator, termcolor, wheel
Required-by:

What else could be wrong for the GB1_T2Q_transformer_TempOutputs.pkl file to not generate?

[brucejwittmann]

Hi @tortoisetsunami, it looks like you might have installed the wrong TAPE package. Did you install MLDE with the --recurse-submodules flag and then build the mlde env from the provided yml file? Based on the docstring from the TAPE command you're using, it looks like you have tape and not tape-neurips2019.

[tortoisetsunami]

Hi @brucejwittmann, thanks for your quick response. I installed MLDE using the input and flags in the MLDE installation guidelines:
git clone --recurse-submodules https://github.com/fhalab/MLDE.git

I downloaded tape-neurips2019 using the same code from your guidelines:

cd ./MLDE
conda env create -f mlde.yml
conda env create -f mlde2.yml
conda env create -f deep_sequence.yml

I also checked tape-neurips2019 folder under the pretrained models folder and it has these h5 files:
bepler_multitask_pretrain_weights.h5
bepler_unsupervised_pretrain_weights.h5
lstm_weights.h5
resnet_weights.h5
transformer_weights.h5
unirep_weights.h5

[brucejwittmann]

@tortoisetsunami odd. Are you looking in the MLDE/code/tape-neurips2019 folder? Can you add a screenshot of exactly what you see in there?

Your installation is using the wrong TAPE version for some reason. Do you have the other TAPE repository installed somewhere else on your computer? If so, I suppose it's possible that that's the one being used.

[tortoisetsunami]

This is a screenshot of my tape-neurips2019 folder:

This screenshot shows the files under pretrained models:

This screenshot shows the files under the tape folder:

[brucejwittmann]

@tortoisetsunami It looks like there's an MLDE folder in the tape-neurips folder. Is it possible that you were in the wrong directory when you built the environments? You should be in the root directory of the MLDE repository when you build the conda environments.

I think next step would be to try a fresh install. Delete the downloaded MLDE repository and reclone it. Then remove the conda environments that you installed and rebuild them. If it doesn't work after that, please copy in the console log from the conda installation process (I'd only need to see it for the mlde env, not mlde2, as mlde seems to be the problem).

[tortoisetsunami]

I deleted the MLDE repository and removed all environments, and reinstalled everything using the installation guide on the MLDE github.

This is the new error I get for generate_encoding.py when I try to use the transformer, resnet, bepler, unirep, and lstm encodings (georgiev & onehot work fine) in the mlde environment.

(mlde) tamima@LAPTOP-V0IMUSP0:/mnt/c/Users/tamim/OneDrive/Documents/CS/Cambronne/MLDE/MLDE$ python generate_encoding.py transformer GB1_T2Q --fasta ./code/validation/basic_test_data/2GI9.fasta --positions V39 D40 --batches 1
Batch#:   0%|                                                                                         | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "generate_encoding.py", line 65, in <module>
    main()
  File "generate_encoding.py", line 61, in main
    batch_size = args.batch_size)
  File "/mnt/c/Users/tamim/OneDrive/Documents/CS/Cambronne/MLDE/MLDE/code/encode/encoding_generator.py", line 332, in generate_encodings
    unnormalized_embeddings = self._generate_tape(n_batches)
  File "/mnt/c/Users/tamim/OneDrive/Documents/CS/Cambronne/MLDE/MLDE/code/encode/encoding_generator.py", line 281, in _generate_tape
    "--output", temp_filename])
  File "/home/tamima/anaconda3/envs/mlde/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/tamima/anaconda3/envs/mlde/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/home/tamima/anaconda3/envs/mlde/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'tape-embed': 'tape-embed'

I checked my new tape-neurips2019 folder, and it doesn't contain another MLDE folder. It also has all of the pretrained models.

Conda log for mlde:

$ conda list
# packages in environment at /home/tamima/anaconda3/envs/mlde:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
_py-xgboost-mutex         2.0                       cpu_0
_pytorch_select           0.1                       cpu_0
_sysroot_linux-64_curr_repodata_hack 3                   haa98f57_10
_tflow_select             2.1.0                       gpu
absl-py                   1.3.0            py37h06a4308_0
astor                     0.8.1            py37h06a4308_0
binutils_impl_linux-64    2.35.1               h27ae35d_9
binutils_linux-64         2.35.1              h454624a_30
biopython                 1.74             py37h7b6447c_0
blas                      1.0                         mkl
brotli-python             1.0.9            py37h6a678d5_7
c-ares                    1.19.1               h5eee18b_0
ca-certificates           2024.9.24            h06a4308_0
certifi                   2022.12.7        py37h06a4308_0
cffi                      1.15.1           py37h5eee18b_3
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.0.4            py37h06a4308_0
cloudpickle               2.0.0              pyhd3eb1b0_0
cudatoolkit               11.8.0               h6a678d5_0
cudnn                     8.9.2.26               cuda11_0
cupti                     11.8.0               he078b1a_0
dataclasses               0.8                pyh6d0b6a4_7
decorator                 5.1.1                    pypi_0    pypi
filelock                  3.9.0            py37h06a4308_0
flit-core                 3.6.0              pyhd3eb1b0_0
future                    0.18.3           py37h06a4308_0
gast                      0.5.3              pyhd3eb1b0_0
gcc_impl_linux-64         7.5.0               h7105cf2_17
gcc_linux-64              7.5.0               h8f34230_30
google-pasta              0.2.0              pyhd3eb1b0_0
grpcio                    1.12.1           py37hdbcaa40_0
gxx_impl_linux-64         7.5.0               h0a5bf11_17
gxx_linux-64              7.5.0               hffc177d_30
h5py                      2.10.0           py37hd6299e0_1
hdf5                      1.10.6               hb1b8bf9_0
huggingface_hub           0.10.1           py37h06a4308_0
hyperopt                  0.2.2                      py_0    conda-forge
idna                      3.4              py37h06a4308_0
importlib-metadata        4.11.3           py37h06a4308_0
intel-openmp              2022.0.1          h06a4308_3633
joblib                    1.1.1            py37h06a4308_0
keras                     2.2.5                    py37_1    conda-forge
keras-applications        1.0.8                      py_1
keras-preprocessing       1.1.2              pyhd3eb1b0_0
kernel-headers_linux-64   3.10.0              h57e8cba_10
ld_impl_linux-64          2.35.1               h7274673_9
libedit                   3.1.20230828         h5eee18b_0
libffi                    3.4.4                h6a678d5_1
libgcc-devel_linux-64     7.5.0               hbbeae57_17
libgcc-ng                 11.2.0               h1234567_1
libgfortran-ng            7.5.0               ha8ba4b0_17
libgfortran4              7.5.0               ha8ba4b0_17
libgomp                   11.2.0               h1234567_1
libgpuarray               0.7.6                h7f8727e_1
libnsl                    2.0.0                h5eee18b_0
libopenblas               0.3.18               hf726d26_0
libprotobuf               3.19.1               h4ff587b_0
libstdcxx-devel_linux-64  7.5.0               hf0c5c8d_17
libstdcxx-ng              11.2.0               h1234567_1
libxgboost                0.90                 he1b5a44_4    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
mako                      1.2.3            py37h06a4308_0
markdown                  3.4.1            py37h06a4308_0
markupsafe                2.1.1            py37h7f8727e_0
mkl                       2019.4                      243
mkl-service               2.3.0            py37he8ac12f_0
mkl_fft                   1.2.0            py37h23d657b_0
mkl_random                1.1.0            py37hd6b4f25_0
mock                      4.0.3              pyhd3eb1b0_0
ncurses                   6.4                  h6a678d5_0
networkx                  2.2                      pypi_0    pypi
ninja                     1.10.2               h06a4308_5
ninja-base                1.10.2               hd09550d_5
nltk                      3.4.4                    py37_0
numpy                     1.21.6                   pypi_0    pypi
openssl                   3.0.15               h5eee18b_0
packaging                 21.3                     pypi_0    pypi
pandas                    0.25.3           py37he6710b0_0
pip                       22.3.1           py37h06a4308_0
protobuf                  3.19.1           py37h295c915_0
psutil                    5.9.0            py37h5eee18b_0
py-xgboost                0.90                     py37_4    conda-forge
pycparser                 2.21               pyhd3eb1b0_0
pygpu                     0.7.6            py37heb32a55_0
pymongo                   3.12.0           py37h295c915_0
pyparsing                 3.1.4                    pypi_0    pypi
pysocks                   1.7.1                    py37_1
python                    3.7.12          hf930737_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd3eb1b0_0
pytorch                   1.5.0           cpu_py37hd91cbb3_0
pytz                      2022.7           py37h06a4308_0
pyyaml                    6.0              py37h5eee18b_1
readline                  8.2                  h5eee18b_0
regex                     2022.7.9         py37h5eee18b_0
requests                  2.28.1           py37h06a4308_0
sacremoses                0.0.43             pyhd3eb1b0_0
scikit-learn              0.21.2           py37hd81dba3_0
scipy                     1.3.0            py37h7c811a0_0
setuptools                65.6.3           py37h06a4308_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.38.2               hc218d9a_0
sysroot_linux-64          2.17                h57e8cba_10
tensorboard               1.13.1           py37hf484d3e_0
tensorflow                1.13.1           py37h90a7d86_1    conda-forge
tensorflow-base           1.14.0          eigen_py37hf4a566f_0
tensorflow-estimator      1.13.0                     py_0
tensorflow-gpu            1.13.1               h0d30ee6_0
termcolor                 2.1.0            py37h06a4308_0
theano                    1.0.5            py37h295c915_1
tk                        8.6.11               h1ccaba5_0
tokenizers                0.10.3           py37hb317417_1
tqdm                      4.32.1                     py_0
transformers              4.14.1             pyhd3eb1b0_0
typing-extensions         4.4.0            py37h06a4308_0
typing_extensions         4.4.0            py37h06a4308_0
urllib3                   1.26.19            pyhd8ed1ab_0    conda-forge
werkzeug                  2.2.2            py37h06a4308_0
wheel                     0.38.4           py37h06a4308_0
wrapt                     1.14.1           py37h5eee18b_0
xgboost                   0.90             py37he1b5a44_4    conda-forge
xz                        5.4.6                h5eee18b_1
yaml                      0.2.5                h7b6447c_0
zipp                      3.11.0           py37h06a4308_0
zlib                      1.2.11            h166bdaf_1014    conda-forge

[brucejwittmann]

Hey a new error! Progress :)

Based on your conda environment output, TAPE is not installed. This tells me that the installation process is failing at the pip install stage for TAPE and that MLDE is falling back to another TAPE install somewhere in your base environment. When you run the conda env create -f mlde.yml command, what is the output? My guess is there's an error somewhere in there about the TAPE build failing.

One quick-fix is to just install the correct tape version manually. In your activated MLDE environment, cd into the tape-neurips2019 folder and run pip install -e .. This is basically what the conda env file does to add TAPE to the mlde environment.

A longer term fix is to determine why TAPE is not being installed in your mlde env. I think next step is to look at the output of the conda build command and see if that yields clues.