Skip to content

Switch deployment to mache.deploy#501

Merged
xylar merged 22 commits intoE3SM-Project:mainfrom
xylar:add-mache-deploy
Mar 21, 2026
Merged

Switch deployment to mache.deploy#501
xylar merged 22 commits intoE3SM-Project:mainfrom
xylar:add-mache-deploy

Conversation

@xylar
Copy link
Copy Markdown
Collaborator

@xylar xylar commented Mar 18, 2026

Summary

This branch replaces the legacy self-contained deployment machinery
(configure_polaris_envs.py, deploy/bootstrap.py, deploy/shared.py,
and the associated template/spec files) with the new mache.deploy
framework introduced in mache 3.0.0. The result is a thin, stable
entrypoint in this repo (deploy.py + deploy/) that delegates the
heavy lifting to mache, rather than duplicating it here.


Core change: deploy.py and the deploy/ directory

What was removed

The old deployment approach lived almost entirely inside this repo:

Removed file Purpose
configure_polaris_envs.py Top-level deployment entrypoint (~162 lines)
deploy/bootstrap.py Full bootstrap logic (~1522 lines)
deploy/shared.py Shared deployment helpers (~396 lines)
deploy/default.cfg Deployment defaults config
deploy/conda-dev-spec.template Conda dev spec template
deploy/load_polaris.template Load-script template
deploy/spec-bootstrap.txt Bootstrap conda spec
deploy/albany_supported.txt Albany support list
deploy/petsc_supported.txt PETSc support list
deploy/unsupported.txt Unsupported machine list

In total, roughly 2,700 lines of deployment code are removed from this
repo.

What was added

The new approach provides a lean set of files that configure and invoke
mache.deploy:

New file Purpose
deploy.py Single entrypoint (~472 lines): reads deploy/pins.cfg, builds the CLI from deploy/cli_spec.json, downloads mache/deploy/bootstrap.py for the pinned (or fork/branch) mache version, and delegates to it
deploy/cli_spec.json Declarative CLI argument specification; mache version is pinned here as 3.0.0
deploy/config.yaml.j2 Jinja2 template for the mache deploy config: sets project name/version, machine resolution, pixi channels, spack support flags, and hooks
deploy/hooks.py Polaris-specific deployment hooks (pre_pixi, pre_spack, post_deploy) that resolve the polaris version, select the correct MPI provider per machine, and locate the Spack path
deploy/pixi.toml.j2 Jinja2 template for the pixi workspace generated by mache during deployment
deploy/spack.yaml.j2 Jinja2 template for the Spack environment generated by mache
deploy/load.sh Shell snippet (sourced by the generated load script) exporting NETCDF, NETCDFF, PNETCDF, PIO, METIS_ROOT, PARMETIS_ROOT, and build-flag env vars
deploy/pins.cfg Pinned mache and Python versions used by deploy.py

The design means that future changes to the common deployment logic (new
machines, updated Spack recipes, pixi channel changes, etc.) only require
a mache release and a version bump in deploy/pins.cfg, with no changes
to this repo.


Supporting polaris changes

Machine configs (polaris/machines/)

  • Comments updated from "conda and spack environments" → "polaris
    environments" to reflect the new pixi-based approach.
  • conda-linux.cfg renamed to default-linux-64.cfg.
  • conda-osx.cfg renamed to default-osx-64.cfg.

Environment variable rename (polaris/setup.py)

The environment variable used to locate the active load script was
renamed from LOAD_POLARIS_ENV to POLARIS_LOAD_SCRIPT to match the
convention used by mache.deploy.

Provenance (polaris/provenance.py)

conda list replaced with pixi list in the provenance file written to
work directories, since the environment is now managed by pixi.

pyproject.toml

  • Added termcolor runtime dependency (used by the new deploy tooling).
  • Removed conda* from the ruff exclusion list (no longer relevant).

Documentation updates

The developer and user guide docs are updated throughout to match the new
deployment workflow:

  • docs/developers_guide/quick_start.md — substantially revised to
    describe running deploy.py instead of configure_polaris_envs.py.
  • docs/developers_guide/updating_conda.md — rewritten to cover
    updating the pixi environment spec via deploy/pixi.toml.j2 and
    deploy/pins.cfg.
  • docs/developers_guide/machines/index.md — updated machine
    deployment descriptions.
  • docs/developers_guide/updating_spack/ — various pages updated to
    reflect the new mache.deploy spack workflow.
  • docs/users_guide/machines/ — load-script and setup instructions
    updated for all supported machines (Aurora, Chrysalis, Frontier,
    Perlmutter).
  • Miscellaneous minor fixes across overview, troubleshooting, and
    tutorial docs.

Checklist

  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
  • Testing comment in the PR documents testing used to verify the changes

@xylar xylar added enhancement New feature or request deployment Changes relate to creating conda and Spack environments, and creating a load script labels Mar 18, 2026
@xylar xylar self-assigned this Mar 18, 2026
@xylar xylar mentioned this pull request Mar 18, 2026
34 tasks
@xylar xylar force-pushed the add-mache-deploy branch from f76407f to ca52922 Compare March 20, 2026 09:04
@xylar xylar force-pushed the add-mache-deploy branch from 78220dd to 091d514 Compare March 20, 2026 15:18
@xylar xylar force-pushed the add-mache-deploy branch from 091d514 to feae065 Compare March 20, 2026 16:31
@xylar
Copy link
Copy Markdown
Collaborator Author

xylar commented Mar 20, 2026

Testing

This was thoroughly tested as part of #502

@xylar
Copy link
Copy Markdown
Collaborator Author

xylar commented Mar 20, 2026

Will merge this with #502 because it requires dependency updates in that branch for successful testing.

@xylar xylar force-pushed the add-mache-deploy branch from 02c5199 to 9c92dbd Compare March 21, 2026 08:03
@xylar xylar merged commit f946d25 into E3SM-Project:main Mar 21, 2026
1 of 6 checks passed
@xylar xylar deleted the add-mache-deploy branch March 21, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment Changes relate to creating conda and Spack environments, and creating a load script enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant