Configurable Workspace AutoApply

[bx-julian]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Currently, the Workspace is hardcoded to be set to AutoApply: true (see here and here).

I would like to configure this boolean via a root level field on the spec, autoApply:

apiVersion: app.terraform.io/v1alpha1
kind: Workspace
metadata:
  name: hello
spec:
  autoApply: false
  ...

It appears this change may be trivial, as instance is passed to CheckWorkspace here and so instance.Spec.AutoApply could be passed to CreateWorkspace here.

However I am unsure if the controller already handles patching an existing workspace, it looks like here some extra checks are done where based on the spec, the TF cloud workspace resource is further modified, suggesting this feature is not as trivial as at first glance. To support configurable AutoApply it looks like the controller will need to account for spec.autoApply changing and keeping it in sync with the TF cloud resource. Should this functionality be added to the CheckWorkspace function?

Is there a design decision why this feature is not currently supported, perhaps for the functionality of the reconciliation loop? If a workspace is set to autoApply false, will the controller continue to work and just be stuck in a pending status for longer?

[redeux]

Hi Jualian. Thanks for the issue. Would you mind elaborating on the use case for autoApply: false?

[bx-julian]

We are using Terraform to manage our GKE clusters and node pools. For certain environments we want to require a human approval of the plan before running the apply. One specific example is so that we can double check gkeVersion matches from manifest to real infra so there are no unexpected node pool upgrades - example if GKE forced an auto upgrade but the manifest is outdated so an apply would cause a version rollback. Theoretically this could be done with something like OPA gatekeeper ensuring Workspace specs are good, but even then we still desire to have manual control in some environments, especially for controlling deletions and for cost reasons. As well we are using GitOps so any developer can PR Workflow resources, it would be difficult to implement uniform authorization/RBAC at the Git/CI level and K8s level; with manual approvals an ops admin can allow another developer's GitOps manifests to be applied.

Currently we are using danisla/terraform-operator which allows separating plan from apply by using separate TerraformPlan and TerraformApply CRDs - this works but is not ideal for a GitOps workflow since we need to change kinds instead of removing a resource. This is why we are specifically looking for a Terraform K8s implementation which can schedule a destroy via a finalizer - to support a GitOps workflow.

Rancher's terraform-controller implements approvals by requiring an annotation to be set before proceeding to the apply.

See Terraform Automation With Argo on Kubernetes: Part1 for a solution built with Argo Events and Argo Workflows (using Argo Events as a "lightweight controller" which triggers Argo Workflows in response to CRD create/delete events) which supports manual approval of plans. Argo Workflows itself can implement a "suspend" pauses the Workflow until an actor "resume"s it.

---

To summarize, our primary requirements for a Terraform K8s implementation to suit our GitOps workflow are:

• Finalizer that triggers resource destroy, so we can remove files from git to destroy infra, without having to edit files in git
• Ability to toggle manual approvals since we would like manual control for some environments and resources
• State is maintained outside K8s (Terraform Cloud, object storage, etc), so that cluster with Terraform CRDs can be destroyed + recreated safely (so that Git is the "backup" and etcd does not need to be backed up)

[mcoletti]

Are there any plans or ETA to support being able to override the default for the auto apply in the workspace CRD?

[amchau]

We will be interested in such a feature as well. Currently if we set the auto-apply to off in the UI, the workspace resource on k8s has a state of planning and wondering if the status does change in k8s once the approval is done in the UI to something like applied or errorred?

[ops-hummus]

Any plans on making this request? Would love to have it

[jochen-schreiber]

I prepared a PR for this: #148

[cmars]

I would like to see this move forward.

In our deployments k8s changes are driven by Helm and ArgoCD. They only occasionally affect the Terraform Workspaces managed by this operator, but when they do it is incredibly important that we have the ability to review the consequences those changes will have for our infrastructure before applying them.

Without this, it is far too easy to make unintended changes in Terraform through the layers of Helm and k8s config above.

We use Terraform Cloud; the ideal outcome I'd like to see:

• Terraform Operator creates a plan in a Terraform Cloud workspace
• CD pipeline blocks on a human review and approval / "terraform apply" of said plan in Terraform Cloud