Can't install python packages with either pip or conda during Docker build

[jennyfolkesson]

Hi,
I've been trying to build a Docker image on an IBM power9 server using ibmcom/powerai:1.6.0-all-ubuntu18.04-py3 as a base.
I've tried adding python packages using both pip and conda but neither one works, and I'm at a loss. It's hard to find answers by googling the errors since not many use the ppc64le architecture, so any help you can give me would be greatly appreciated.
Thank you.

Below is my Dockerfile for installing packages with conda, which during build hangs with the message:
"Solving environment: ...working... "

`FROM ibmcom/powerai:1.6.0-all-ubuntu18.04-py3

ENV USER_NAME pwrai
ENV WORK_DIR /home/${USER_NAME}/microDL

RUN ["/bin/bash", "-c", "cd /opt/anaconda3/bin && source activate base && IBM_POWERAI_LICENSE_ACCEPT=yes ./accept-powerai-license.sh" ]

RUN sudo /bin/ln -sf /bin/bash /bin/sh

RUN sudo apt-get update && sudo apt-get install -yq --no-install-recommends
software-properties-common &&
add-apt-repository universe
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
&& sudo apt-get update && sudo apt-get install -yq --no-install-recommends
build-essential
apt-utils
libcupti-dev
ca-certificates
cmake
libfreetype6-dev
wget
git
pkg-config
tmux
graphviz
vim

RUN sudo apt-get clean &&
sudo apt-get autoremove &&
sudo rm -rf /var/lib/apt/lists/*

ENV PATH /opt/anaconda3/bin:${PATH}
RUN sudo ln -s /opt/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh
ADD conda_environment_docker.yml /tmp/conda.yml
RUN conda update -n base -c defaults --override-channels conda &&
conda env create --file=/tmp/conda.yml &&
conda install tensorflow-gpu==1.13.1 &&
conda activate micro_dl

WORKDIR ${WORK_DIR}
ENV PYTHONPATH ${WORK_DIR}
RUN sudo chown ${USER_NAME} ${WORK_DIR}
RUN sudo chgrp ${USER_NAME} ${WORK_DIR}

EXPOSE 8888 6006`

[jennyfolkesson]

Here is my Dockerfile for pip installation of packages:

`FROM ibmcom/powerai:1.6.0-all-ubuntu18.04-py3

ENV USER_NAME pwrai
ENV WORK_DIR /home/${USER_NAME}/microDL
USER root
ENV PATH /root/.local/bin:${PATH}

RUN ["/bin/bash", "-c", "cd /opt/anaconda3/bin && source activate base && IBM_POWERAI_LICENSE_ACCEPT=yes ./accept-powerai-license.sh" ]

RUN sudo /bin/ln -sf /bin/bash /bin/sh

RUN apt-get update && apt-get install -y
software-properties-common &&
add-apt-repository universe
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
&& sudo apt-get update && sudo apt-get install -yq --no-install-recommends
build-essential
apt-utils
libcupti-dev
ca-certificates
cmake
libfreetype6-dev
libpng-dev
wget
git
pkg-config
tmux
graphviz
vim
#python3-pip
python3-dev
python3-distutils
python3-venv

RUN sudo apt-get clean &&
sudo apt-get autoremove &&
sudo rm -rf /var/lib/apt/lists/*

RUN echo $PATH
RUN wget https://bootstrap.pypa.io/pip/3.6/get-pip.py &&
python3.6 get-pip.py --target $(python3 -c 'import sys; print(sys.path[-1])') &&
rm get-pip.py && \

python3.6 -m venv env && source ./env/bin/activate &&
sudo python3.6 -m pip install --user --upgrade --force pip setuptools wheel &&
python3.6 -m pip install --user --upgrade Cython==0.29.10 &&
python3.6 -m pip install --user --upgrade numpy &&
python3.6 -m pip install --user --upgrade scipy &&
python3.6 -m pip install --user --upgrade scikit-image &&
python3.6 -m pip install --user --upgrade scikit-learn &&
python3.6 -m pip install --user --upgrade opencv-python &&
python3.6 -m pip install --user --upgrade pandas &&
python3.6 -m pip install --user --upgrade natsort nose protobuf pydot PyYAML testfixtures tqdm zarr && \

python3.6 -m pip install --user --upgrade --no-cache-dir keras==2.1.6 &&
python3.6 -m pip install --user --upgrade --no-cache-dir jupyter &&
python3.6 -m pip install --user --upgrade matplotlib

WORKDIR ${WORK_DIR}
ENV PYTHONPATH ${WORK_DIR}
RUN sudo chown ${USER_NAME} ${WORK_DIR}
RUN sudo chgrp ${USER_NAME} ${WORK_DIR}

EXPOSE 8888 6006`

[jennyfolkesson]

And below is the error message I get trying to install packages with pip:

`Collecting pip<22.0
Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
Collecting setuptools
Downloading setuptools-59.6.0-py3-none-any.whl (952 kB)
Collecting wheel
Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
Successfully installed pip-21.3.1 setuptools-59.6.0 wheel-0.37.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting pip
Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Collecting setuptools
Using cached setuptools-59.6.0-py3-none-any.whl (952 kB)
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
WARNING: The script wheel is installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip, pip3 and pip3.6 are installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Successfully installed pip-21.3.1 setuptools-59.6.0 wheel-0.37.1
Collecting Cython==0.29.10
Downloading https://files.pythonhosted.org/packages/e7/2e/aee8dfff93c7f5d20461ddcefd1da3d43bab18e1666a7777f4d9dbe94065/Cython-0.29.10.tar.gz (2.0MB)
Building wheels for collected packages: Cython
Running setup.py bdist_wheel for Cython: started
Failed building wheel for Cython
Running setup.py bdist_wheel for Cython: finished with status 'error'
Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-x02aauu3/Cython/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp420xt618pip-wheel- --python-tag cp36:
Unable to find pgen, not compiling formal grammar.
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help

error: invalid command 'bdist_wheel'

---

Running setup.py clean for Cython
Failed to build Cython
Installing collected packages: Cython
Running setup.py install for Cython: started
Running setup.py install for Cython: still running...
Running setup.py install for Cython: finished with status 'done'
Successfully installed Cython-0.29.10
Collecting numpy
Downloading https://files.pythonhosted.org/packages/51/60/3f0fe5b7675a461d96b9d6729beecd3532565743278a9c3fe6dd09697fa7/numpy-1.19.5.zip (7.3MB)
Building wheels for collected packages: numpy
Running setup.py bdist_wheel for numpy: started
Failed building wheel for numpy
Running setup.py bdist_wheel for numpy: finished with status 'error'
Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp6uxnrillpip-wheel- --python-tag cp36:
Running from numpy source directory.
Cythonizing sources
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_mt19937.pyx
Traceback (most recent call last):
File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 59, in process_pyx
from Cython.Compiler.Version import version as cython_version
ModuleNotFoundError: No module named 'Cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 235, in
main()
File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 231, in main
find_process_files(root_dir)
File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 222, in find_process_files
process(root_dir, fromfile, tofile, function, hash_db)
File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 188, in process
processor_function(fromfile, tofile)
File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 64, in process_pyx
raise OSError('Cython needs to be installed in Python as a module')
OSError: Cython needs to be installed in Python as a module
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 508, in
setup_package()
File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 488, in setup_package
generate_cython()
File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 285, in generate_cython
raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

---

Running setup.py clean for numpy
Failed cleaning build dir for numpy
Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" clean --all:
Running from numpy source directory.

setup.py clean is not supported, use one of the following instead:

- `git clean -xdf` (cleans all files)
- `git clean -Xdf` (cleans all versioned files, doesn't touch
                    files that aren't checked into the git repo)

Add --force to your command to use it anyway if you must (unsupported).

---

Failed to build numpy
Installing collected packages: numpy
Running setup.py install for numpy: started
Running setup.py install for numpy: finished with status 'error'
Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-wxeof2ob-record/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.6/numpy --user --prefix=:
Running from numpy source directory.

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last NumPy release on PyPi)


Cythonizing sources
numpy/random/_bounded_integers.pxd.in has not changed
Processing numpy/random/_mt19937.pyx
Traceback (most recent call last):
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 59, in process_pyx
    from Cython.Compiler.Version import version as cython_version
ModuleNotFoundError: No module named 'Cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 235, in <module>
    main()
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 231, in main
    find_process_files(root_dir)
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 222, in find_process_files
    process(root_dir, fromfile, tofile, function, hash_db)
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 188, in process
    processor_function(fromfile, tofile)
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 64, in process_pyx
    raise OSError('Cython needs to be installed in Python as a module')
OSError: Cython needs to be installed in Python as a module
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 508, in <module>
    setup_package()
  File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 488, in setup_package
    generate_cython()
  File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 285, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

----------------------------------------

Command "/env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-wxeof2ob-record/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.6/numpy --user --prefix=" failed with error code 1 in /tmp/pip-build-hgmtxb0b/numpy/`

[alexlang74]

Hi @jennyfolkesson ,
to my knowledge, these PowerAI channels are no longer supported . We are building much more recent versions of data science libraries as part of https://github.com/open-ce/open-ce .

Our University Partners at Oregon State University provide conda channels with these packages. See https://osuosl.org/services/powerdev/opence/#current-release on a description how to install these packages

[eric-aquaronne]

Hi @jennyfolkesson , Alex is correct : you should use OpenCE recipes. If you need a pre built set, beside the OSU 'academic build', we have a reliable build for our client made by Rocket: see the RocketCE conda channel.
I would be very interested to know your use case are (what libraries do you use....) we are working to make sure our Power CPUs features (e.g. inside CPU matrix accelerators) are used to the best : can you reach me at eric.aquaronne@fr.ibm.com ? Thank you

[jennyfolkesson]

Hi Eric,
We were stuck with the old docker channels because we had a repo using an old version of TensorFlow that we were using on a power9 server. We're right now in the process of updating the repo to pytorch. After the update, I'll reach out to you if we have issues with using OpenCE. Thank you!