Consider splitting parallel catchup v2 out of supercluster

[bboston7]

With the addition of new options exclusive to parallel catchup v2, the mission is increasingly diverging from the rest of supercluster. Moreover, parallel catchup v2 is driven primarily through helm templates, with F# used mostly to fill in template values, unlike the other missions where the F# code is driving the mission.

It might be easier to maintain both supercluster and parallel catchup if we split parallel catchup into its own repo. This would help to avoid cluttering supercluster with many parallel catchup specific options, and also could help in keeping the parallel catchup implementation cleaner.

I'm opening this ticket to serve as a discussion point for this idea. The scope is to decide what to do (if anything).

[jayz22]

Agree the parallel catchup has different design from the rest of the mission. Not sure if splitting it out of the repo reduces the maintenance.

The two sides mostly share the same set of options, which primarily come from core requirements (e.g. adding a new stellar-core.cfg option) which has to be updated in both sides regardless the approach.

The recently added new, diverging options (labels and taint) come from the need (which is still at experiment state iiuc @jacekn) to schedule parallel catchup separately from other missions for scalability. Those options theoretically would not be needed once we figure out a stable approach going forward.

The two sides have a bit of code sharing, such as using the same F# k8s library for log dumping, though its not that big and could be rewritten.
Parallel catchup also uses F# for control flow, i.e. checking status and reporting it to Jenkins. We could rewrite that into a new thin client using e.g. Python. Or even having the client as a pod running inside k8s (in which case we need a way for Jenkins CI workers to interact directly with k8s).

The only use case so far for parallel catchup has been CI, splitting it out will require setting up the Jenkins pipeline for both repos (or at least a separate command for the new repo), which would require more maintenance.

[jacekn]

Yeah I agree with @jayz22 that split might not reduce maintenance. Two separate projects would also mean that full supercluster run would require us to build and execute two separate binaries. It's doable but it would add some complexity.
The parallel catchup V2 mission isn't that complex at k8s level: it's just ConfigMaps, StatefulSet and a Deployment for the most part. So perhaps it could possible refactor it so that the objects are crated from F# directly rather than via helm?

It would also be nice to make supercluster faster and run more things in parallel. For example we could try running some larger missions in parallel rather that in sequence. We could also look into allowing multiple, independent, supercluster runs to execute at the same time. For those things to happen I think we'd need to add even more options, some of which can be per-mission.

Above makes me think that it might be a good idea to explore using config file instead of cli options. Config file could allow us to have nice flexibility for all missions. For example taints and label selectors could have default values set at the top level that we could override per-mission if needed. We could perhaps have different "profiles" for supercluster runs that would be defined in a config file rather than at Jenkins job level.

[bboston7]

Thanks for the input! I appreciate now some ways in which splitting PCv2 out might increase complexity, and I'm on board with not making any changes there. More details in responses below.

Those options theoretically would not be needed once we figure out a stable approach going forward.

Ah, I didn't realize these options might go away at some point. I was/am concerned about the parallel catchup v2 (PCv2) option set and functionality diverging to the point where there is little overlap between the PCv2 mission and the remainder of supercluster. I agree that we aren't there yet. It's nice to hear we may never get there.

Two separate projects would also mean that full supercluster run would require us to build and execute two separate binaries.

I was actually thinking PCv2 wouldn't even be a part of the supercluster pipeline anymore. Instead, it would have its own pipeline that perhaps runs a little less frequently so that we can have more frequent runs of the other missions. The long runtime for PCv2 means we sometimes go long stretches without a complete CI run (often due to repeated cancellations to test other things), which has downstream impacts like infrequent updates to our acceptance tagged builds (which means changes don't get automatically deployed on pubnet watchers), and the "piling up" of untested commits that make it difficult to debug future pipeline failures.

At the time of writing this, the last complete full supercluster run was 12 days ago. Given the events of the last week, that's maybe not the most useful data point, but before that it was 9 days between successful runs which is still not great.

So perhaps it could possible refactor it so that the objects are crated from F# directly rather than via helm?

I think this is actually getting to the root of why I opened this issue in the first place. I actually prefer the Helm configuration for this mission, partly because there is a wealth of Kubernetes documentation online that assumes the usage of Helm charts. I think it would actually make PCv2 more annoying to modify if it used F# objects the same way as the rest of supercluster.

In my mind, there's a fundamental difference in the way PCv2 works versus the way the rest of supercluster works. Most supercluster missions set up synthetic networks to test the interaction between stellar-core nodes. They need a way to programmatically change this environment (e.g. changing the load profile, running a network survey, or upgrading the network) during the test itself, and the F# code orchestrates those dynamic changes.

However, in PCv2 the environment is static and there is no interaction between pods (correct me if I'm wrong here). This feels fundamentally different, and as a result much of the F# supercluster infrastructure feels irrelevant to the mission. Maybe that's OK, but it's ultimately why I wanted to check in on what our future plans for PCv2 should be.

It would also be nice to make supercluster faster and run more things in parallel. For example we could try running some larger missions in parallel rather that in sequence. We could also look into allowing multiple, independent, supercluster runs to execute at the same time. For those things to happen I think we'd need to add even more options, some of which can be per-mission.

This would be fantastic, especially in addition to speeding up PCv2 (which I think is still on the roadmap?).

Above makes me think that it might be a good idea to explore using config file instead of cli options. Config file could allow us to have nice flexibility for all missions. For example taints and label selectors could have default values set at the top level that we could override per-mission if needed.

This is a great idea that would really help with the usability of supercluster.

[jacekn]

At the time of writing this, the last complete full supercluster run was 12 days ago. Given the events of the last week, that's maybe not the most useful data point, but before that it was 9 days between successful runs which is still not great.

We should be able to improve this significantly. Parallel catchup is much faster now and I believe we can make it take <5h or even less.
I'm also testing changes that would allow us to start parallel catchup at the same time as large tests. Currently we wait for large tests to complete before starting parallel catchup.

Above makes me think that it might be a good idea to explore using config file instead of cli options. Config file could allow us to have nice flexibility for all missions. For example taints and label selectors could have default values set at the top level that we could override per-mission if needed.

This is a great idea that would really help with the usability of supercluster.

What could also be nice in the future is for that config to contain list of missions to run. That would allow us to create different configs, or "profiles" and control missions that way, rather than rely on Jenkins executing individual missions. It would be a nice stepping stone towards removing dependency on Jenkins.