diff --git a/alliance-computing.rst b/alliance-computing.rst index e6cd8dc..a3f626b 100644 --- a/alliance-computing.rst +++ b/alliance-computing.rst @@ -240,8 +240,9 @@ You should be able to follow these instructions to set up on any of the other Al and run NEMO. #. Follow the :ref:`salishseadocs:InstallCommandProcessorPackages` docs to install the - :ref:`salishseacmd:SalishSeaCmdProcessor` and its dependencies in a - :command:`conda` environment. + :ref:`salishseacmd:SalishSeaCmdProcessor` and its dependencies with `Pixi`_. + + .. _Pixi: https://pixi.prefix.dev/latest/ #. Follow the :ref:`BuildXIOS-MEOPAR-nibi` docs to build ``XIOS-2``. diff --git a/getting_started.rst b/getting_started.rst index 36fb23b..db2f9b3 100644 --- a/getting_started.rst +++ b/getting_started.rst @@ -35,7 +35,7 @@ Here's the Getting Started checklist: #. :ref:`SetUpSecureRemoteAccess` #. :ref:`SetUpGit` #. :ref:`SetUpBash` -#. :ref:`InstallMiniforge` +#. :ref:`InstallPixi` #. :ref:`SetUpYourAnalysisRepository` @@ -109,16 +109,15 @@ before you can: on a Waterhole workstation. -.. _InstallMiniforge: +.. _InstallPixi: -Install Miniforge -================= +Install Pixi +============ You will need to: -#. Learn about :ref:`MOAD-CondaPkgAndEnvMgr` -#. :ref:`Install Miniforge` on your laptop -#. :ref:`Install Miniforge` in your workspace on the EOAS Ocean collection of Linux computers +#. Learn about :ref:`MOAD-PixiPkgAndEnvMgr` +#. :ref:`Install Pixi` in your workspace on the EOAS Ocean collection of Linux computers .. _SetUpYourAnalysisRepository: @@ -129,6 +128,6 @@ Set Up Your Analysis Repository You will need to have: #. :ref:`SetUpGit` -#. :ref:`Installed Miniforge` +#. :ref:`Installed Pixi` before you can set up your :ref:`analysis repository`. diff --git a/index.rst b/index.rst index 483dec4..6072691 100644 --- a/index.rst +++ b/index.rst @@ -12,7 +12,12 @@ UBC EOAS MOAD Group Documentation ********************************* -This is documentation by and for members of the MOAD (Mesoscale Ocean and Atmospheric Dynamics) group in the EOAS (Earth, Ocean, and Atmospheric Sciences) department at UBC (the University of British Columbia). +This is documentation by and for members of the MOAD +(Mesoscale Ocean and Atmospheric Dynamics) +group in the EOAS +(Earth, Ocean, and Atmospheric Sciences) +department at UBC +(University of British Columbia). .. toctree:: @@ -24,7 +29,7 @@ This is documentation by and for members of the MOAD (Mesoscale Ocean and Atmosp vcs_repos git_version_control bash_config - conda_pkg_env_mgr + pixi/index analysis_repo python jupyter diff --git a/pixi/index.rst b/pixi/index.rst new file mode 100644 index 0000000..fb04460 --- /dev/null +++ b/pixi/index.rst @@ -0,0 +1,34 @@ +.. Copyright 2018 – present by The UBC EOAS MOAD Group +.. and The University of British Columbia +.. +.. Licensed under a Creative Commons Attribution 4.0 International License +.. +.. https://creativecommons.org/licenses/by/4.0/ + + +.. _MOAD-PixiPkgAndEnvMgr: + +************************************ +Pixi Package and Environment Manager +************************************ + +`Pixi`_ is a tool for managing software packages and environments. +It is designed to be fast, reproducible, and easy to use across different operating systems (Windows, macOS, and Linux). + +.. _Pixi: https://pixi.prefix.dev/latest/ + +For users with minimal experience in Python dependency management, +Pixi simplifies many complex tasks into a single tool. + +`This notebook`_ is from the 16-Apr-2026 group discussion in which Doug introduced Pixi. + +.. _This notebook: https://github.com/UBC-MOAD/PythonNotes/blob/main/pkgs-envs/PythonPkgsEnvsSlides-2026.ipynb + + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + pixi_pkg_env_mgr + pixi_SalishSeaCmd + pixi_analysis_repo diff --git a/pixi/pixi_SalishSeaCmd.rst b/pixi/pixi_SalishSeaCmd.rst new file mode 100644 index 0000000..4c463ef --- /dev/null +++ b/pixi/pixi_SalishSeaCmd.rst @@ -0,0 +1,172 @@ +.. Copyright 2018 – present by The UBC EOAS MOAD Group +.. and The University of British Columbia +.. +.. Licensed under a Creative Commons Attribution 4.0 International License +.. +.. https://creativecommons.org/licenses/by/4.0/ + + +.. _MigratingToPixiPoweredSalishSeaCmd: + +************************************** +Migrating to Pixi-Powered SalishSeaCmd +************************************** + +If you are running NEMO on one of the Alliance HPC clusters, +an easy way to start using Pixi is to migrate your :py:obj:`SalishSeaCmd` installation +and workflow to Pixi. +The steps are: + +#. Install Pixi on the Alliance cluster + (if you have not already done so) +#. Pull the most recent version of :py:obj:`SalishSeaCmd` from GitHub +#. Use Pixi to to create an isolated environment for :py:obj:`SalishSeaCmd` +#. Use :command:`pixi run salishsea run...` to launch a run +#. Clean up the elements of your old conda-powered installation + + +Install Pixi +============ + +.. code-block:: console + + $ curl -fsSL https://pixi.sh/install.sh | bash + +Having Pixi autocomplete its sub-commands, +option flags, +environment names, +etc. +when you hit :kbd:`Tab` is very convenient. +To enable that, +add the following line to your :file:`$HOME/.bashrc` file: + +.. code-block:: bash + + eval "$(pixi completion --shell bash)" + +After installation, please restart your terminal session to finalize the installation. + + +Update :py:obj:`SalishSeaCmd` +============================= + +.. code-block:: console + + $ cd $HOME/SalishSeaCmd/ + $ git pull + +.. note:: + Depending on when you set up :py:obj:`SalishSeaCmd` on the cluster, + it is possible that its path is different to :file:`$HOME/MEOPAR/SalishSeaCmd/`. + The next most likely possibility is :file:`$PROJECT/$USER/MEOPAR/SalishSeaCmd/`. + + +Create the :py:obj:`SalishSeaCmd` Environment +============================================= + +.. code-block:: console + + $ pixi install + + +Use :command:`pixi run salishsea run...` +======================================== + +When you are in the :file:`SalishSeaCmd/` directory +(or a sub-directory) +you can run the :program:`salishsea` command with with the :command:`pixi run` command. +Example: + +.. code-block:: console + + $ pixi run salishsea help + +A common use-case is to execute the :command:`salishsea run` command in the directory containing +your run description YAML file. +To accomplish that, +you have to tell Pixi where to find the :file:`SalishSeaCmd/` directory so that it can use the +correct environment. +You do that by using the ``-m`` or ``--manifest`` option of :command:`pixi run`. +Example: + +.. code-block:: console + + $ cd SS-run-sets/SalishSea/sea/Carbon_v202111/ + $ pixi run -m $HOME/MEOPAR/SalishSeaCmd salishsea run 01jan11_Lb80.yaml \ + /scratch/allen/Carbon/MoreSens/Now/01jan11/ + + +Cleanup +======= + +#. Uninstall :py:obj:`SalishSeaCmd` from :file:`$HOME/.local/bin/` + + .. code-block:: console + + $ conda activate salishsea-cmd + $ python -m pip uninstall SalishSeaCmd + +#. Uninstall :py:obj:`NEMO-Cmd` from :file:`$HOME/.local/bin` + + .. code-block:: console + + $ python -m pip uninstall NEMO-Cmd + +#. Remove your ``salishsea-cmd`` conda environment + + .. code-block:: console + + $ conda deactivate salishsea-cmd + $ conda env remove -n salishsea-cmd + +#. Remove your :py:obj:`NEMO-Cmd` repository clone + + Installation and updating of :py:obj:`NEMO-Cmd` is now handled by Pixi because :py:obj:`NEMO-Cmd` + is now an implicit dependency for :py:obj:`SalishSeaCmd` in its manifest file. + So, + you can remove your :py:obj:`NEMO-Cmd` repository clone: + + .. code-block:: console + + $ rm -rf $HOME/MEOPAR/NEMO-Cmd/ + + .. note:: + Depending on when you set up :py:obj:`SalishSeaCmd` on the cluster, + it is possible that the path is different to :file:`$HOME/MEOPAR/NEMO-Cmd/`. + The next most likely possibility is :file:`$PROJECT/$USER/MEOPAR/NEMO-Cmd/`. + +#. Remove ``NEMO-Cmd`` from the ``vcs revisions:`` sections of your run description YAML files. + + If you don't you will get warnings like: + + .. code-block:: output + :class: no-copybutton + + nemo_cmd.prepare WARNING: revision and status requested for non-existent repo: /home/dlatorne/MEOPAR/NEMO-Cmd + + every time you use :command:`pixi run salishsea run ...` + +#. Remove conda (if the only environment left is ``base``) + + If the output of: + + .. code-block:: console + + $ conda env list + + shows only the ``base`` environment like: + + .. code-block:: output + :class: no-copybutton + + # conda environments: + # + base /home/allen/miniforge3 + + follow the `Uninstall instructions`_ for Miniforge to remove it. + + .. _Uninstall instructions: https://github.com/conda-forge/miniforge#uninstall + + If you have environments in addition to ``base`` left, + migrate them to Pixi before you remove conda. + Please contact Doug if you need help. diff --git a/pixi/pixi_analysis_repo.rst b/pixi/pixi_analysis_repo.rst new file mode 100644 index 0000000..aca9aa4 --- /dev/null +++ b/pixi/pixi_analysis_repo.rst @@ -0,0 +1,178 @@ +.. Copyright 2018 – present by The UBC EOAS MOAD Group +.. and The University of British Columbia +.. +.. Licensed under a Creative Commons Attribution 4.0 International License +.. +.. https://creativecommons.org/licenses/by/4.0/ + + +.. _MigratingYourAnalysisRepositoryToPixi: + +****************************************** +Migrating Your Analysis Repository to Pixi +****************************************** + +The steps top migrate your analysis repository to Pixi are: + +#. Initialize your repository as a Pixi workspace +#. Test a notebook +#. Clean up the default environment +#. Commit and push :file:`pixi.toml`, :file:`pixi.lock`, and removal of :file:`notebooks/environment.yaml` +#. Import any other environment you may have +#. Remove conda if you have removed all your environments + +The steps above assume that you have already :ref:`Installed Pixi` + + +Initialize Your Repository as a Pixi Workspace +============================================== + +These command assume that you are working on ``salish`` or a Waterhole workstation. +They use :file:`analysis-casey/` as the analysis repository. +Yours, +of course, +will use your name. + +.. code-block:: console + + $ cd /ocean/$USER/MOAD/analysis-casey/ + $ pixi init --import notebooks/environment.yaml + $ pixi add \ + --git https://github.com/SalishSeaCast/tools.git \ + --pypi \ + --subdir SalishSeaTools \ + SalishSeaTools + +.. note:: + Depending on when you set up your analysis repository, + it is possible that its path is different to :file:`/ocean/$USER/MOAD/analysis-/`. + + +Test a Notebook +=============== + +If you run Jupyter from the commandline: + +.. code-block:: console + + $ pixi run jupyter lab notebooks/my_notebook.ipynb + +If you us VS Code: + +* Install the `Pixi Code extension`_ + + .. _Pixi Code extension: https://marketplace.visualstudio.com/items?itemName=renan-r-santos.pixi-code + +* The :guilabel:`Select Kernel` button in VS Code should show the kernel in your workspace as a choice, + probably :file:`.pixi/envs/default/bin/python` + + +Clean Up the Default Environment +================================ + +You no longer need the :file:`notebooks/environment.yaml` file, +note the ``analysis-casey`` conda environment. +So, +remove them: + +.. code-block:: console + + $ rm notebooks/environment.yaml + $ conda env remove -n analysis-casey + + +Commit and Push +=============== + +The steps above will have created 3 new files: + +* :file:`pixi.toml` - the Pixi manifest +* :file:`pixi.lock` - the Pixi lockfile +* :file:`.gitattributes` - settings to control how Git manages :file:`pixi.lock` + +removed 1 file: + +* :file:`notebooks/environment.yaml` - conda environment description file + +and modified 1 file: + +* :file:`.gitignore` + +Stage and commit those files in the VS Code Commit side-panel or at the commandline, +and push them to GitHub: + +.. code-block:: console + + $ git add pixi.toml pixi.lock .gitattributes .gitignore notebooks/environment.yaml + $ git commit -m "Change to Pixi for dependency and environment management" + $ git push + + +Import Any Other Environments +============================= + +If the :file:`erddap-obs-matching/environment.yaml` file describes an environment called ``erddap-obs-matching``, +you can import that environment into your workspace with: + +.. code-block:: console + + $ pixi import erddap-obs-matching/environment.yaml + +That command: + * adds a Pixi feature called ``erddap-obs-matching`` + * adds channels and dependencies from the :file:`erddap-obs-matching/environment.yaml` file to the feature + * adds the feature to a Pixi environment called ``erddap-obs-matching`` + +Then, + +.. code-block:: console + + $ pixi install -e erddap-obs-matching + +creates the ``erddap-obs-matching`` environment and installs the dependencies, +updating the Pixi lock file with those details to ensure reproducibility. + +You can run commands in the environment with commands like: + +.. code-block:: console + + $ pixi run -e erddap-obs-matching python -m my_module + +When you are happy that the environment is working as you expect, +clean up by removing the :file:`erddap-obs-matching/environment.yaml` environment description file, +and the :file:`erddap-obs-matching/environment.yaml` conda environment: + +.. code-block:: console + + $ rm erddap-obs-matching/environment.yaml + $ conda env remove -n erddap-obs-matching + +Then commit and push the changes to :file:`pixi.toml` and :file:`pixi.lock`, +and the removal of :file:`erddap-obs-matching/environment.yaml`. + + +Maybe Remove Conda +================== + +If the output of: + +.. code-block:: console + + $ conda env list + +shows only the ``base`` environment like: + +.. code-block:: output + :class: no-copybutton + + # conda environments: + # + base /home/allen/miniforge3 + +follow the `Uninstall instructions`_ for Miniforge to remove it. + +.. _Uninstall instructions: https://github.com/conda-forge/miniforge#uninstall + +If you have environments in addition to ``base`` left, +migrate them to Pixi before you remove conda. +Please contact Doug if you need help. diff --git a/pixi/pixi_pkg_env_mgr.rst b/pixi/pixi_pkg_env_mgr.rst new file mode 100644 index 0000000..1ea3e99 --- /dev/null +++ b/pixi/pixi_pkg_env_mgr.rst @@ -0,0 +1,147 @@ +.. Copyright 2018 – present by The UBC EOAS MOAD Group +.. and The University of British Columbia +.. +.. Licensed under a Creative Commons Attribution 4.0 International License +.. +.. https://creativecommons.org/licenses/by/4.0/ + + +Why Pixi? +========= + +If you have worked with Python before, +you might have heard of tools like ``pip``, ``conda``, or ``venv``. +Pixi provides several advantages over these traditional tools: + +* **Reproducibility**: + Pixi uses a "lock file" (:file:`pixi.lock`) to ensure that the exact same versions of all packages + can be reliably installed by different users on different machines. + This prevents "it works on my machine" bugs. +* **Speed**: + Pixi is built in Rust and is exceptionally fast at resolving and installing packages. +* **Multi-language**: + While excellent for Python, Pixi can also manage packages from other languages and systems (like C++, R, or command-line tools). +* **Isolated Environments**: + Pixi creates a separate environment for each project, + so you don't have to worry about one project's requirements breaking another. + + +.. _InstallingPixi: + +Installing Pixi +=============== + +The easiest way to install Pixi is using their official installer script. + + +On macOS and Linux: +------------------- + +.. code-block:: console + + $ curl -fsSL https://pixi.sh/install.sh | bash + + +On Windows (PowerShell): +------------------------ + +.. code-block:: powershell + + iwr -useb https://pixi.sh/install.ps1 | iex + + +Autocompletion +-------------- + +Having Pixi autocomplete its sub-commands, +option flags, +environment names, +etc. +when you hit :kbd:`Tab` is very convenient. +To enable that for :program:`bash`, +add the following line to your :file:`$HOME/.bashrc` file: + +.. code-block:: bash + + eval "$(pixi completion --shell bash)" + +For other shells, +please see the `Pixi Autocompletion`_ docs. + +.. _Pixi Autocompletion: https://pixi.prefix.dev/latest/installation/#autocompletion + +After installation, please restart your terminal session to finalize the installation. + + +Basic Usage +=========== + +Once Pixi is installed, here are the most common commands you will use. + + +Installing an Environment +------------------------- + +After you have cloned a Pixi-powered repository, navigate to your project folder and run: + +.. code-block:: console + + $ pixi install + +This downloads the packages required by the project and installs them in an isolated environment +for you to use. + + +Adding Packages +--------------- + +To add a package (like ``numpy`` or ``pandas``) to your project: + +.. code-block:: console + + $ pixi add numpy + +This will update your :file:`pixi.toml` and :file:`pixi.lock` files, +and automatically install the package into the project-specific environment. +Please ensure that you commit the changes your :file:`pixi.toml` and :file:`pixi.lock` files, +and push them to GitHub so that you maintain an accurate record of you working environment. + + +Running Commands +---------------- + +To run a script or a command within your Pixi environment: + +.. code-block:: console + + $ pixi run python my_script.py + + +Using the Pixi Shell +-------------------- + +If you are doing a lot of command-line work and don't want to have to type :command:`pixi run` +before every command, +you can work inside the environment interactively by doing: + +.. code-block:: console + + $ pixi shell + +This "activates" the environment in your current terminal session. +You can exit the shell by typing ``exit``. + + +Pixi Details +============ + +Pixi has many more features and capabilities. +Most MOAD users will use Pixi by cloning Pixi-powered repositories from GitHub and +running commands with :command:`pixi run`, +but Pixi also has extensive support for Python package development and other software +development tasks. +The `documentation`_ has excellent "Getting Started" sections and tutorials, +as well as detailed `command-line interface (CLI) reference docs`_. + +.. _documentation: https://pixi.prefix.dev/latest/ +.. _command-line interface (CLI) reference docs: https://pixi.prefix.dev/latest/reference/cli/pixi/ diff --git a/vscode.rst b/vscode.rst index 6b192fc..1fe568e 100644 --- a/vscode.rst +++ b/vscode.rst @@ -89,6 +89,10 @@ Microsoft's Python Extension https://marketplace.visualstudio.com/items?itemName=ms-python.python +Pixi Code + Pixi support for the Python Environments extension + + https://marketplace.visualstudio.com/items?itemName=renan-r-santos.pixi-code Microsoft's Jupyter Notebook Renderers Renderers for various image/graph formats output in Jupyter notebooks diff --git a/conda_pkg_env_mgr.rst b/zzz_archival_docs/conda_pkg_env_mgr.rst similarity index 100% rename from conda_pkg_env_mgr.rst rename to zzz_archival_docs/conda_pkg_env_mgr.rst diff --git a/zzz_archival_docs/index.rst b/zzz_archival_docs/index.rst index fe94461..e149807 100644 --- a/zzz_archival_docs/index.rst +++ b/zzz_archival_docs/index.rst @@ -23,4 +23,5 @@ it will remain available after deletion in the `Git repository`_. :maxdepth: 2 :caption: Contents: + conda_pkg_env_mgr hg_version_control