Clean command improvements

[jacekn]

What problem does your feature solve?

The clean command currently removes all resources in the namespace. When two independent runs are started the first one that reaches the clean stage will destroy all resources, including those deployed bu the other run.

I would like for us to think about how to improve supercluster behaviour to make it easier to launch multiple, independent, runs at the same time.

What would you like to see?

I can think of two ways we can solve this:

1. Update the clean command to only remove relevant resources. Perhaps use a nonce across all resources to track what needs to be deleted.
2. Add capability for supercluster to create k8s namespaces dynamically at runtime and deploy all resources there. This would allow us to delete the namespace at the end of the run without impacting any other runs. As a side effect cleanup would be easier - we could just delete the namespace and all resources would be cleaned up automatically. Small downside of this approach is that supercluster would need more broad access to k8s that it currently does.

[jayz22]

Another alternative is to use helm charts (similar to parallel catchup v2) to manage resources. This way a single command helm uninstall can clean up resource within that release (and nothing else). Most missions should be able to fit into that mold.