Update spack-stack-dev from spack develop 2026/01/09 by climbfuji · Pull Request #574 · JCSDA/spack

climbfuji · 2026-01-09T21:18:38Z

Description

Note. This PR must be merged as a regular merge, not as a squashed merge!

This PR updates spack-stack-dev from the upstream Spack develop branch. Because spack-stack-dev was previously based on Spack release v1.0.1 with a few modifications cherry-picked from develop, and because the Spack developers no longer maintain mergeability between the releases and the develop branch, I had to resolve several merge conflicts.

One of the perks of this update is that we can reenable most GitHub actions tests, which resolves #573.

Because we previously cherry-picked some commits from develop into the spack-stack-dev branch, we now have fewer differences:

spack v1.0.1 vs spack-stack-dev before this PR: https://github.com/spack/spack/compare/v1.0.1...JCSDA:spack:spack-stack-dev?expand=1

spack develop as of 2026/01/09 vs this PR: https://github.com/spack/spack/compare/develop...climbfuji:feature/update_spack-stack-dev_from_develop?expand=1

Testing

CI
For testing with spack-stack, see Update spack submodule branch spack-stack-dev from spack develop spack-stack#1877

Due to performance and modeling reasons, the specs that are reused in concretization are represented in the ASP problem with their pure build dependencies trimmed. On `develop` this means that any preference of the form: ```yaml packages: c: prefer: - llvm ``` is "disregarded" for purely reused specs, since the solver is not aware of the compiler nodes used for them. This commit fixes the semantics of virtual preferences so that they are respected also for reused specs. It does so without representing their compilers as nodes - so the modeling of the DAG is not steered towards different optimal solutions by the missing compilers. Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

* Handle URL paths Signed-off-by: Ryan Krattiger <ryan.krattiger@kitware.com>

The current condition under which Spack starts a POSIX jobserver is whether all `gmake`s and `ninja`s in the DAG support `--jobserver-auth=FIFO:PATH`. That's does not take into account the (currently) unmodelled implicit dependency of `gcc` on `gmake`. GCC's `lto-wrapper` executable generates a temporary Makefile and invokes make either from the `MAKE` environment variable, or by looking in `PATH`. I don't think this is very trivial to fix, other than making GCC inject `gmake` as a build dependency, but that leads to circular dependencies since `gmake` is used to build GCC. Easier to revert for now and move this feature to the experimental installer PR where we can probably get away with it for now without a fix. This reverts commit 9c1ed1a. Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>

…ases (spack#50477) * Append to install-time test log for multiple test phases Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * log.py: fix tmp dir arg Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * log.py: fix typo (:) Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * use existing tests/pkg Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * remove new test file/pkg Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * test/cmd/install.py: skip log append check for win32 Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * test/cmd/install.py: skip install 'tests' for win32 Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * test/cmd/install.py: fix install test-skip logic (win32) Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * test/cmd/install.py: skip test_package_output for windows Signed-off-by: Alex Richert <alexander.richert@noaa.gov> * Revert "test/cmd/install.py: skip test_package_output for windows" This reverts commit f14beed. Signed-off-by: Alex Richert <alexander.richert@noaa.gov> --------- Signed-off-by: Alex Richert <alexander.richert@noaa.gov>

Third try to enable concretization caching in `develop`. Past attempts were reverted due to cache management issues and race conditions. See spack#48198 for the introduction of the concretization cache, manifest, etc. That PR only added testing for the cache itself, not the manifest or cleanup procedures. This PR: - [x] Adds tests for cleanup - [x] Refactors the cache to store base32 hashes like the rest of Spack's hashes - [x] gzips cache entries for a smaller memory footprint - [x] Removes redundant stat calls - [x] Less aggressive/error prone cleanup phase - [x] Adds an entry limit to the cache, removes the size limit - [x] Rewrite pruning to use LRU instead of FIFO - [x] Canonicalizes and strips `clingo` input before hashing, for greater hit rate This PR simplifies the locking around the concretization cache. Entries are stored by the base32 hash of their inputs, and there is a single lock file (as we do in many places in spack) for all such entries. We use byte range locks to manage all the entries. Cleanup has changed to be less expensive overall. There is no more size limit; there is only an entry limit, and specs in the cache are compressed. Specs are ~35k, and the initial limit is 1000 cache entries, or ~35 MB. This limit is customizable in configuration. To check if the cache needs cleanup, we need only list files in the cache directory. To clean up, we stat the files and clean up the least recently used concretizations. We remove half the cache at a time so that we do not pay for cleanup on every concretization when we're close to the cache limit. The concretization cache is stored in the `misc_cache` by default, so `spack clean -m` is all that is needed to get rid of it. The location of the concretization cache can be specified independently (with a `url` config option) but we will document and expand on that use later as we integrate it in CI. --------- Signed-off-by: John Parent <john.parent@kitware.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>

…ack#51118) This commit fundamentally changes how externals are handled in the solver and introduces new methods for defining dependencies between them, addressing the design described in issue spack#49697. Core Solver Change: ------------------- Previously, externals were treated as a form of "configuration" and handled as a pervasive special case throughout the solver rules. With this change, externals are now modeled as fully concrete, installed specs *before* the solve begins. This allows the solver to treat them just like any other installed package, reusing them as available. This greatly simplifies the solver logic and removes numerous special cases. This new model enables more robust dependency specification, which is provided in two new formats: Inline Spec Syntax: ------------------- Users can now specify dependencies directly in the main `spec:` string, leveraging the familiar Spack syntax. packages: mpileaks: externals: - spec: "mpileaks@2.3 %mpich %callpath" prefix: /user/path Spack resolves these dependencies against other defined external packages. The system will error if a dependency is ambiguous (matches more than one external) or cannot be satisfied. Dependency types are inferred from the package recipe if not explicitly provided. YAML `dependencies:` Configuration: ---------------------------------- For more complex or unambiguous configurations, a new `dependencies:` block is supported. This provides more explicit control. packages: mpileaks: externals: - spec: "mpileaks@2.3" prefix: /user/path dependencies: - id: callpath_id deptypes: [link] - spec: mpich deptypes: [build, link] Dependencies in this block can be specified by `spec:` (which must be unambiguous) or by a unique `id:` that refers to another external (and must also be unambiguous). This format also allows for explicit setting of `deptypes` and `virtuals`. Backward Compatibility (Python): -------------------------------- For now, this change preserves the existing behavior of "stitching" external Python extensions (e..g, `py-foo`) to a single, guessed Python implementation if no explicit dependency is provided. This is maintained to avoid breaking existing configurations, but users can now use the new syntax to explicitly state the Python dependency. Fixes spack#49697 --------- Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

This is a hack while we figure out why it's doing this Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>

`spack info` has always shown dependencies grouped by type, but not conditional dependency information. This generifies the code used to display variants and uses it for dependencies as well. As with variants, dependencies are shown grouped by conditions, with unconditional dependencies first. - [x] generify when-grouping code to use a `Formatter` class - [x] Rewrite variant code to use generic formatter. - [x] Rewrite dependency code to use generic formatter. - [x] add full type annotations to `cmd/info.py` Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>