Add Slurm support to rrun with PMIx-based coordination by pentschev · Pull Request #775 · rapidsai/rapidsmpf

pentschev · 2026-01-11T21:39:46Z

This PR adds Slurm support for rrun, enabling RapidsMPF to run without MPI. This is achieved by adding SlurmBackend class that wraps PMIx for process coordination, implementing bootstrap operations (put/get/barrier/sync) using PMIx primitives.

The new execution mode delivers a passthrough approach with multiple tasks per node, one task per GPU. This is similar to the way MPI applications launch in Slurm, but unlike mpirun which should not be part of the application execution, rrun must act as launcher to the application. If rrun is omitted, Slurm will automatically fallback to MPI (if available).

Usage example:

  srun \
      --mpi=pmix \
      --nodes=2 \
      --ntasks-per-node=4 \
      --cpus-per-task=36 \
      --gpus-per-task=1 \
      --gres=gpu:4 \
      rrun ./benchmarks/bench_shuffle -C ucxx

copy-pr-bot · 2026-01-11T21:39:49Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

pentschev · 2026-01-11T21:39:53Z

/ok to test

pentschev · 2026-01-11T21:47:06Z

/ok to test

pentschev · 2026-01-11T21:57:03Z

/ok to test

pentschev · 2026-01-11T22:05:52Z

/ok to test

pentschev · 2026-01-13T22:12:23Z

/ok to test

pentschev · 2026-01-13T22:12:43Z

/ok to test

pentschev · 2026-01-14T09:54:16Z

/ok to test

nirandaperera

Did a partial review

nirandaperera · 2026-02-05T03:50:10Z

cpp/include/rapidsmpf/bootstrap/utils.hpp

+std::optional<std::string> getenv_optional(std::string_view name);
+
+/**
+ * @brief Parse integer from environment variable.
+ *
+ * Retrieves an environment variable and parses it as an integer.
+ *
+ * @param name Name of the environment variable to retrieve.
+ * @return Parsed integer value, or std::nullopt if not set.
+ * @throws std::runtime_error if the variable is set but cannot be parsed as an integer.
+ */
+std::optional<int> getenv_int(std::string_view name);


Question: cant we use rapidsmpf Options here?

No, we can't use most of the rest of RapidsMPF due to direct or indirect CUDA dependencies, and we can't have any CUDA dependencies in the bootstrapping. In the cpp files we have comments stating that:

// NOTE: Do not use RAPIDSMPF_EXPECTS or RAPIDSMPF_FAIL in this file. // Using these macros introduces a CUDA dependency via rapidsmpf/error.hpp. // Prefer throwing standard exceptions instead.

Again, all the rrun stuff can't include cuda headers. So we could refactor things, but it is fiddly.

nirandaperera · 2026-02-05T03:52:47Z

cpp/src/bootstrap/bootstrap.cpp

 /**
 * @brief Detect backend from environment variables.
 */
 Backend detect_backend() {


Is this order the same as what's mentioned here https://github.com/rapidsai/rapidsmpf/pull/775/changes#diff-6b0f67c1b74a322dd090ffd62247887e46f1e57e97029778c34db6e341f0e5aeR25-R27 ? 🤔

Good catch! This has changed many times during development, fixed it now.

nirandaperera · 2026-02-05T04:02:41Z

cpp/src/bootstrap/bootstrap.cpp

@@ -108,6 +102,41 @@ Context init(Backend backend) {
            }
            break;


Can move these to some detail/ util methods outside the switch case scope?

Then we can switch like,

Context init(Backend backend) { if (backend == Backend::AUTO) { backend = detect_backend(); } // Get rank and nranks based on backend switch (backend) { case Backend::FILE: return file_backend_init(); ... ...

Good idea, done.

nirandaperera · 2026-02-05T04:03:34Z

cpp/src/bootstrap/bootstrap.cpp

Use RAPIDSMPF_EXPECTS?

Can't without refactoring to make rapidsmpf/error.hpp not depend on cuda_runtime_api.h.

nirandaperera · 2026-02-05T04:04:42Z

cpp/src/bootstrap/bootstrap.cpp

+                throw std::runtime_error(
+                    "Could not determine rank for Slurm backend. "
+                    "Ensure you're running with 'srun --mpi=pmix'."
+                );
+            }
+
+            try {
+                ctx.nranks = get_nranks();
+            } catch (const std::runtime_error& e) {
+                throw std::runtime_error(
+                    "Could not determine nranks for Slurm backend. "
+                    "Ensure you're running with 'srun --mpi=pmix'."
+                );
+            }
+
+            if (!(ctx.rank >= 0 && ctx.rank < ctx.nranks)) {
+                throw std::runtime_error(
+                    "Invalid rank: " + std::to_string(ctx.rank) + " must be in range [0, "
+                    + std::to_string(ctx.nranks) + ")"
+                );


RAPIDSMPF_FAIL?

No, see again my previous comment, and also at the top of this file https://github.com/rapidsai/rapidsmpf/pull/775/files#diff-d80df374c64a4d18659a70bf48420afd18a77b6cd650861db904aff4cb867765R19-R21.

nirandaperera · 2026-02-05T04:07:39Z

cpp/src/bootstrap/bootstrap.cpp

+#ifdef RAPIDSMPF_HAVE_SLURM
+    case Backend::SLURM:
+        {
+            detail::SlurmBackend backend{ctx};


Should we create SlurmBackend every operation here? Or should we attach it to the ctx with a Backend interface class and shared_ptr?

I agree, this wasn't ideal and I had plans to address it in a future PR. Since you pointed out I'll address it here.

Done now in 82441ef .

nirandaperera · 2026-02-05T04:14:55Z

cpp/src/bootstrap/slurm_backend.cpp

+void check_pmix_status(pmix_status_t status, std::string const& operation) {
+    if (status != PMIX_SUCCESS) {
+        throw std::runtime_error(operation + " failed: " + pmix_error_string(status));
+    }
+}


Maybe, let's use a helper function like RAPIDSMPF_MPI

This is the helper function, I'm not sure exactly what you're proposing, is it just the name not following the same RAPIDSMPF_... pattern or something else? If it's just the naming pattern I can adjust it, but I tried to prevent using the same pattern given the CUDA restrictions we have to avoid creating confusion with the "regular" RapidsMPF code.

nirandaperera · 2026-02-05T04:53:54Z

cpp/src/bootstrap/ucxx.cpp

+    std::string result;
+    result.reserve(input.size() * 2);
+    for (char ch : input) {
+        auto c = static_cast<unsigned char>(ch);
+        result.push_back(hex_chars[c >> 4]);
+        result.push_back(hex_chars[c & 0x0F]);
+    }


should we use stringstream?

We could use a stringstream if there's a strong preference for that, however, the existing implementation should have considerably better performance, given it's not overly complicated I'd prefer to leave the current one.

pentschev · 2026-02-03T10:48:06Z

cpp/tools/rrun.cpp

+/**
+ * @brief Apply topology-based bindings for a specific GPU.
+ *
+ * This function sets CPU affinity, NUMA memory binding, and network device
+ * environment variables based on the topology information for the given GPU.
+ *
+ * @param cfg Configuration containing topology information.
+ * @param gpu_id GPU ID to apply bindings for.
+ * @param verbose Print warnings on failure.
+ */
+void apply_topology_bindings(Config const& cfg, int gpu_id, bool verbose) {
+    if (!cfg.topology.has_value() || gpu_id < 0) {
+        return;
+    }
+
+    auto it = cfg.gpu_topology_map.find(gpu_id);
+    if (it == cfg.gpu_topology_map.end()) {
+        if (verbose) {
+            std::cerr << "[rrun] Warning: No topology information for GPU " << gpu_id
+                      << std::endl;
+        }
+        return;
+    }
+
+    auto const& gpu_info = *it->second;
+
+    if (cfg.bind_cpu && !gpu_info.cpu_affinity_list.empty()) {
+        if (!set_cpu_affinity(gpu_info.cpu_affinity_list)) {
+            if (verbose) {
+                std::cerr << "[rrun] Warning: Failed to set CPU affinity for GPU "
+                          << gpu_id << std::endl;
+            }
+        }
+    }
+
+    if (cfg.bind_memory && !gpu_info.memory_binding.empty()) {
+        if (!set_numa_memory_binding(gpu_info.memory_binding)) {
+#if RAPIDSMPF_HAVE_NUMA
+            if (verbose) {
+                std::cerr << "[rrun] Warning: Failed to set NUMA memory binding for GPU "
+                          << gpu_id << std::endl;
+            }
+#endif
+        }
+    }
+
+    if (cfg.bind_network && !gpu_info.network_devices.empty()) {
+        std::string ucx_net_devices;
+        for (size_t i = 0; i < gpu_info.network_devices.size(); ++i) {
+            if (i > 0) {
+                ucx_net_devices += ",";
+            }
+            ucx_net_devices += gpu_info.network_devices[i];
+        }
+        setenv("UCX_NET_DEVICES", ucx_net_devices.c_str(), 1);
+    }
+}


This was just moved to a new function for better organization.

pentschev · 2026-02-03T10:53:37Z

cpp/tools/rrun.cpp

+        } else if (arg == "--") {
+            // Everything after -- is the application and its arguments
+            if (i + 1 < argc) {
+                cfg.app_binary = argv[i + 1];
+                for (int j = i + 2; j < argc; ++j) {
+                    cfg.app_args.push_back(argv[j]);
+                }
+            }
+            break;


Optional -- separator for disambiguation of rrun arguments and application+arguments. Not specifying one is still supported via the else condition below.

pentschev · 2026-02-05T08:13:10Z

cpp/include/rapidsmpf/bootstrap/utils.hpp

+std::optional<std::string> getenv_optional(std::string_view name);
+
+/**
+ * @brief Parse integer from environment variable.
+ *
+ * Retrieves an environment variable and parses it as an integer.
+ *
+ * @param name Name of the environment variable to retrieve.
+ * @return Parsed integer value, or std::nullopt if not set.
+ * @throws std::runtime_error if the variable is set but cannot be parsed as an integer.
+ */
+std::optional<int> getenv_int(std::string_view name);


No, we can't use most of the rest of RapidsMPF due to direct or indirect CUDA dependencies, and we can't have any CUDA dependencies in the bootstrapping. In the cpp files we have comments stating that:

// NOTE: Do not use RAPIDSMPF_EXPECTS or RAPIDSMPF_FAIL in this file. // Using these macros introduces a CUDA dependency via rapidsmpf/error.hpp. // Prefer throwing standard exceptions instead.

pentschev · 2026-02-05T08:19:54Z

cpp/src/bootstrap/bootstrap.cpp

 /**
 * @brief Detect backend from environment variables.
 */
 Backend detect_backend() {


Good catch! This has changed many times during development, fixed it now.

pentschev · 2026-02-05T08:23:23Z

cpp/src/bootstrap/bootstrap.cpp

@@ -108,6 +102,41 @@ Context init(Backend backend) {
            }
            break;


Good idea, done.

pentschev · 2026-02-05T08:23:30Z

cpp/src/bootstrap/bootstrap.cpp

+                throw std::runtime_error(
+                    "Could not determine rank for Slurm backend. "
+                    "Ensure you're running with 'srun --mpi=pmix'."
+                );
+            }
+
+            try {
+                ctx.nranks = get_nranks();
+            } catch (const std::runtime_error& e) {
+                throw std::runtime_error(
+                    "Could not determine nranks for Slurm backend. "
+                    "Ensure you're running with 'srun --mpi=pmix'."
+                );
+            }
+
+            if (!(ctx.rank >= 0 && ctx.rank < ctx.nranks)) {
+                throw std::runtime_error(
+                    "Invalid rank: " + std::to_string(ctx.rank) + " must be in range [0, "
+                    + std::to_string(ctx.nranks) + ")"
+                );


No, see again my previous comment, and also at the top of this file https://github.com/rapidsai/rapidsmpf/pull/775/files#diff-d80df374c64a4d18659a70bf48420afd18a77b6cd650861db904aff4cb867765R19-R21.

pentschev · 2026-02-05T08:24:21Z

cpp/src/bootstrap/slurm_backend.cpp

+void check_pmix_status(pmix_status_t status, std::string const& operation) {
+    if (status != PMIX_SUCCESS) {
+        throw std::runtime_error(operation + " failed: " + pmix_error_string(status));
+    }
+}


This is the helper function, I'm not sure exactly what you're proposing, is it just the name not following the same RAPIDSMPF_... pattern or something else? If it's just the naming pattern I can adjust it, but I tried to prevent using the same pattern given the CUDA restrictions we have to avoid creating confusion with the "regular" RapidsMPF code.

pentschev · 2026-02-05T08:31:42Z

cpp/src/bootstrap/ucxx.cpp

+    std::string result;
+    result.reserve(input.size() * 2);
+    for (char ch : input) {
+        auto c = static_cast<unsigned char>(ch);
+        result.push_back(hex_chars[c >> 4]);
+        result.push_back(hex_chars[c & 0x0F]);
+    }


We could use a stringstream if there's a strong preference for that, however, the existing implementation should have considerably better performance, given it's not overly complicated I'd prefer to leave the current one.

pentschev · 2026-02-05T09:24:27Z

cpp/src/bootstrap/bootstrap.cpp

+#ifdef RAPIDSMPF_HAVE_SLURM
+    case Backend::SLURM:
+        {
+            detail::SlurmBackend backend{ctx};


I agree, this wasn't ideal and I had plans to address it in a future PR. Since you pointed out I'll address it here.

wence-

Partial

wence- · 2026-02-05T09:15:08Z

dependencies.yaml

+  build-pmix:
+    common:
+      - output_types: [conda, pyproject, requirements]
+        packages:
+          - libpmix-devel >=5.0,<6.0


pmix isn't available on pypi (I don't think we should try and make it so either). So I think this should only go to the conda output?

Good catch, fixed.

wence- · 2026-02-05T09:16:50Z

cpp/include/rapidsmpf/bootstrap/bootstrap.hpp

+ *
+ * Different backends have different visibility semantics for put() operations:
+ * - Slurm/PMIx: Requires explicit fence (PMIx_Fence) to make data visible across nodes.
+ * - FILE: put() operations are immediately visible via atomic filesystem operations.


Aside (we don't have to do anything here). Many shared filesystems actually don't promise posix-style atomicity for rename/fclose etc...

This is a good point. We should definitely handle that better in the future.

wence- · 2026-02-05T09:17:35Z

cpp/include/rapidsmpf/bootstrap/file_backend.hpp

+     * For FileBackend, this is a no-op since put() operations use atomic
+     * file writes that are immediately visible to all processes via the
+     * shared filesystem.
+     */


Suggested change

*/

This is an implementation detail that could easily become out of date.

It indeed is. However, we explain happens for FileBackend, we do the same for SlurmBackend. Would you prefer this is moved to a comment in the implementation? I think it's important to document it somewhere, but I don't have a preference if here or in the implementation as comment.

I see now you have more questions/suggestions about this. I just moved all the implementation details from docstrings as comments in the implementation, also improved the interface documentation to be generally suitable for all backends in 7d1c478.

wence- · 2026-02-05T09:18:51Z

cpp/include/rapidsmpf/bootstrap/slurm_backend.hpp

+ * # Hybrid mode: one task per node, 4 GPUs per task, two nodes.
+ * srun \
+ *     --mpi=pmix \
+ *     --nodes=2 \
+ *     --ntasks-per-node=1 \
+ *     --cpus-per-task=144 \
+ *     --gpus-per-task=4 \
+ *     --gres=gpu:4 \
+ *     rrun -n 4 ./benchmarks/bench_shuffle -C ucxx


question: Can you explain the reason to want to support both passthrough (makes sense to me, all the information is configured using slurm) and "hybrid", where we do only partial launching with srun?

Yes, the passthrough mode essentially requires the user to determine the ideal topoplogy (number of tasks, GPUs, CPUs, etc.) and in that case rrun is only coordinating the bootstrapping and nothing more. In the hybrid mode the user just needs to specify the number of processes (currently should match the number of GPUs) and rrun will ensure all the topology is properly setup, just like if you were running baremetal, and also (will) allow the user to specify a custom topology setup (i.e., provide a json with the exact setup that is desired, which may be useful for new HW bringup and experimentation, in case, for example, the number of CPUs should be partitioned unevenly). IOW, the hybrid mode is both a convenience and insurance in case Slurm doesn't necessarily does everything properly.

Do you think I should document what I wrote above in the code? My plan is to add a better document (like a README file) with more details about bootstrap in general in a follow-up PR, so not sure if we want all that information in the code here as well.

One more thing about this, I've been thinking of alternative ways to support launching for both single-node and multi-node non-Slurm setups and we'll need some sort of KVS for that, so one thing I have in mind is to use PMIx itself for that. I know we would need some infrastructure to leverage PMIx (since Slurm is not present in those cases where PMIx is already there), but the hybrid implementation would likely be reused for that, although I have nothing more concrete yet.

IOW, the hybrid mode is both a convenience and insurance in case Slurm doesn't necessarily does everything properly.

Is this a concern today?

I think it would be easier to review this PR if we brought the two modes in in two pieces rather than all at once.

Fair enough, let me break this down into two PRs. I'll have the current PR implementing most of the infrastructure and the passthrough mode, and move the hybrid mode into a new PR.

I've removed it now in 5b9da5c (about 500 LOC shorter now) and started a new draft #844 to be reviewed/merged after this goes in.

wence- · 2026-02-05T09:19:42Z

cpp/include/rapidsmpf/bootstrap/slurm_backend.hpp

+     * The key-value pair is committed immediately and made visible to other
+     * ranks via a fence operation.


Suggested change

* The key-value pair is committed immediately and made visible to other

* ranks via a fence operation.

* The key-value pair is committed immediately and made visible to other

* ranks after a collective `sync()`.

None of the methods mention the word "fence".

Done in 7d1c478

wence- · 2026-02-05T09:22:46Z

cpp/include/rapidsmpf/bootstrap/slurm_backend.hpp

+     * All ranks must call this before any rank proceeds. The fence also
+     * ensures all committed key-value pairs are visible to all ranks.
+     *


I think we should remove the committing key-value pairs part. In the abstract backend sync makes things visible, barrier is just a barrier.

Done in 7d1c478

wence- · 2026-02-05T09:23:21Z

cpp/include/rapidsmpf/bootstrap/slurm_backend.hpp

+     * @brief Ensure all previous put() operations are globally visible.
+     *
+     * For Slurm/PMIx backend, this executes PMIx_Fence to make all committed
+     * key-value pairs visible across all nodes. This is required because
+     * PMIx_Put + PMIx_Commit only makes data locally visible; PMIx_Fence
+     * performs the global synchronization and data exchange.
+     *
+     * @throws std::runtime_error if PMIx_Fence fails.


Again, I am not sure the docstring should talk about the implementation.

Done in 7d1c478

wence- · 2026-02-05T09:30:25Z

cpp/include/rapidsmpf/bootstrap/utils.hpp

+std::optional<std::string> getenv_optional(std::string_view name);
+
+/**
+ * @brief Parse integer from environment variable.
+ *
+ * Retrieves an environment variable and parses it as an integer.
+ *
+ * @param name Name of the environment variable to retrieve.
+ * @return Parsed integer value, or std::nullopt if not set.
+ * @throws std::runtime_error if the variable is set but cannot be parsed as an integer.
+ */
+std::optional<int> getenv_int(std::string_view name);


Again, all the rrun stuff can't include cuda headers. So we could refactor things, but it is fiddly.

wence- · 2026-02-05T09:36:30Z

cpp/tools/rrun.cpp

+ *
+ * @throws std::runtime_error on timeout or launch failure.
+ */
+std::string launch_rank0_and_get_address(


question: It seems like a lot of the complexity in launching here is around hybrid mode. But I am still not entirely clear why we need it, can you explain?

Please see my response to your previous question https://github.com/rapidsai/rapidsmpf/pull/775/files#r2767917764.

wence- · 2026-02-05T09:37:54Z

cpp/tools/rrun.cpp

+    // Barrier to ensure data exchange
+    backend.barrier();


My understanding of the API contract is that this should be backend.sync(). As it happens barrier and sync for the pmix backend both do the same thing. But semantically, we're documented as requiring sync here.

Indeed, this precedes the time sync() was introduced so it's outdated. Fixed now in acc85fc.

gforsyth

The updates to the conda recipes look good to me, but I'd like @KyleFromNVIDIA to take a look at the CMake changes

gforsyth

Oh, ok, I'm not part of the cmake codeowners here (correctly) -- approving packaging changes.

pentschev · 2026-02-05T15:27:23Z

The updates to the conda recipes look good to me, but I'd like @KyleFromNVIDIA to take a look at the CMake changes

Oh, ok, I'm not part of the cmake codeowners here (correctly) -- approving packaging changes.

Thanks Gil, appreciate it. Would indeed be nice to have Kyle review CMake as well, thanks for tagging him.

pentschev added 2 commits January 11, 2026 13:37

Add Slurm support for rrun

aa9821b

Add libpmix-devel dependency

9642eae

pentschev self-assigned this Jan 11, 2026

pentschev added feature request New feature or request non-breaking Introduces a non-breaking change labels Jan 11, 2026

Add build-pmix dependency matrix

f80ffcf

Fix libpmix-devel conda build dependency

d39476c

Pin libpmix <6.0

90c3565

pentschev added 3 commits January 13, 2026 14:09

Support srun (with debug)

92686f4

Remove debugging

02fd79d

Add Slurm mode to rrun

0a569f8

Merge remote-tracking branch 'upstream/main' into rrun-slurm

c3bb33d

Fix run on Slurm check without breaking run with mpirun

66da920

pentschev added 9 commits January 14, 2026 03:30

Support multiple ranks with single Slurm task

df1ed25

Use file backend in Slurm hybrid mode

fff8d86

Use rrun parent process as PMIx coordinator

e87e535

Fix subprocess coordination

af4c390

Move rank launching into new functions

5417c51

Use different execute functions for each launch condition

ceda7a8

Make coordinate_root_address_via_pmix cleaner with std::optional

0876fd6

Add [rrun] prefix to cout/cerr messages

72d2399

Generalize into setup_launch_and_cleanup

20d5ae3

Update rrun usage instructions

0763300

pentschev changed the title ~~Slurm support for rrun~~ Add Slurm support to rrun with PMIx-based coordination Feb 4, 2026

pentschev marked this pull request as ready for review February 4, 2026 22:37

pentschev requested review from a team as code owners February 4, 2026 22:37

pentschev requested a review from gforsyth February 4, 2026 22:37

pentschev added 2 commits February 4, 2026 15:00

Move rrun PMIx complexity to SlurmBackend

76d2f29

Fix build for the case PMIx is unavailable

530adab

nirandaperera reviewed Feb 5, 2026

View reviewed changes

pentschev added 2 commits February 5, 2026 00:18

Fix detection order documentation

a779cda

Use specific initializer functions

3841f65

pentschev commented Feb 5, 2026

View reviewed changes

wence- reviewed Feb 5, 2026

View reviewed changes

Add backend interface, use a pointer to the interface in Context

82441ef

pentschev requested a review from a team as a code owner February 5, 2026 09:41

pentschev added 2 commits February 5, 2026 01:43

Remove build-pmix from wheels

9d42fd7

Generalized Backend docstrings, moved implementation details to cpp

7d1c478

pentschev requested review from nirandaperera and wence- February 5, 2026 10:16

pentschev added 6 commits February 5, 2026 02:17

Merge remote-tracking branch 'upstream/main' into rrun-slurm

4f8ed8a

Use sync instead of barrier

acc85fc

Merge remote-tracking branch 'upstream/main' into rrun-slurm

bf15cc1

Merge remote-tracking branch 'upstream/main' into rrun-slurm

3ee7a21

Fix BackendType in Python

2891959

Remove Slurm hybrid mode

5b9da5c

pentschev mentioned this pull request Feb 5, 2026

Add hybrid Slurm support to rrun with PMIx-based coordination #844

Draft

gforsyth reviewed Feb 5, 2026

View reviewed changes

gforsyth approved these changes Feb 5, 2026

View reviewed changes

		* The key-value pair is committed immediately and made visible to other
		* ranks via a fence operation.

Conversation

pentschev commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Jan 11, 2026

Uh oh!

pentschev commented Jan 11, 2026

Uh oh!

pentschev commented Jan 11, 2026

Uh oh!

pentschev commented Jan 11, 2026

Uh oh!

pentschev commented Jan 11, 2026

Uh oh!

pentschev commented Jan 13, 2026

Uh oh!

pentschev commented Jan 13, 2026

Uh oh!

pentschev commented Jan 14, 2026

Uh oh!

nirandaperera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wence- left a comment

Choose a reason for hiding this comment

Uh oh!

pentschev commented Jan 11, 2026 •

edited

Loading