Add DDA rollback functionality

[khewonc]

What does this PR do?

Adds rollback functionality for DDAs with fleet experiments. Includes:

• stop: trigger rollback (for eventual stopExperiment)
• timeout: rollback after 15min of a running experiment
• abort: user makes a manual change (ignores manual change if done at the same time as the timeout b/c of complexity)

Motivation

https://datadoghq.atlassian.net/browse/CONTP-1404

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

TBA. Test commands for now:

Stop signal

# update dda spec to create new revision
kubectl patch dd <name> --type=merge -p '{"spec":{"global":{"tags":["env:test"]}}}'
# new experiment mock (generation should match dda generation after patch)
kubectl patch dd <name> --type=merge --subresource=status -p "{\"status\":{\"experiment\":{\"phase\":\"running\",\"generation\":2,\"id\":\"exp-1\"}}}"
# stop experiment mock
kubectl patch dd <name> --type=merge --subresource status --patch 'status: {experiment: {phase: stopped}}'

Timeout

# update dda spec to create new revision
kubectl patch dd <name> --type=merge -p '{"spec":{"global":{"tags":["env:test"]}}}'
# new experiment mock (generation should match dda generation after patch)
kubectl patch dd <name> --type=merge --subresource=status -p "{\"status\":{\"experiment\":{\"phase\":\"running\",\"generation\":2,\"id\":\"exp-1\"}}}"
# wait 15 min (starting from when the new revision was created)

Abort

# update dda spec to create new revision
kubectl patch dd <name> --type=merge -p '{"spec":{"global":{"tags":["env:test"]}}}'
# new experiment mock (generation should match dda generation after patch)
kubectl patch dd <name> --type=merge --subresource=status -p "{\"status\":{\"experiment\":{\"phase\":\"running\",\"generation\":2,\"id\":\"exp-1\"}}}"
# update dda spec again
kubectl patch dd <name> --type=merge -p '{"spec":{"global":{"tags":["env:foo"]}}}'

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 75.72254% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.15%. Comparing base (a1206ff) to head (6106687).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/controller/datadogagent_controller.go	41.02%	17 Missing and 6 partials ⚠️
internal/controller/datadogagent/experiment.go	84.78%	7 Missing and 7 partials ⚠️
...controller/datadogagent/controller_reconcile_v2.go	16.66%	2 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2838      +/-   ##
==========================================
+ Coverage   38.94%   40.15%   +1.21%     
==========================================
  Files         313      315       +2     
  Lines       27139    27956     +817     
==========================================
+ Hits        10570    11227     +657     
- Misses      15780    15919     +139     
- Partials      789      810      +21     

Flag	Coverage Δ
unittests	40.15% <75.72%> (+1.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/datadogagent/controller.go	94.73% <ø> (+1.87%)	⬆️
...ler/datadogagent/controller_reconcile_v2_common.go	33.90% <100.00%> (+0.28%)	⬆️
...er/datadogagent/controller_reconcile_v2_helpers.go	65.00% <100.00%> (+0.17%)	⬆️
internal/controller/datadogagent/revision.go	81.51% <100.00%> (+3.53%)	⬆️
...controller/datadogagent/controller_reconcile_v2.go	61.00% <16.66%> (-1.06%)	⬇️
internal/controller/datadogagent/experiment.go	84.78% <84.78%> (ø)
internal/controller/datadogagent_controller.go	59.54% <41.02%> (-7.13%)	⬇️

... and 14 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1206ff...6106687. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d358c9c2bd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[zhuminyi]

nit: It is better to call abortExperiment first to detects user edit. If there is a user edit, phase will be changed to aborted and user's edit will be preserved (this is a narrow window of race condtion).

[khewonc]

I initially had it called first as an early return, but calling abortExperiment first right now causes the operator logs to look like it's aborting when it's actually in timeout. I ended up deciding to reorder over complicating the function

[arbll]

Is this safe to do in two steps ? What if the second steps fails after the first step succeeded ?

[arbll]

Like won't this prevent rollbacks after apply ?

[khewonc]

It should be fine to do in two steps. The actual rollback is handled in manageExperiment so it won't prevent experiment rollbacks. We don't allow user-initiated manual rollbacks so no issues there. There is one bug though in that after a rollback, if the user tries to apply the same change again, the operator will immediately roll back so it looks like there was no change. I'll add a fix for that