-
Notifications
You must be signed in to change notification settings - Fork 10
Improve portability of dags deployer app #40
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Refactor gcloud composer environments run wrapper commands with something like kubectl exec.
Benefits
- generalize it's usefulness to "roll your own" k8s deployments or airflow.
- Pave the way to eliminate glcoud dependency of dagsdeployer app. (Would also require a refactor to use go storage client for file copying / hashing etc.) This would allow us to use a MUCH smaller container for the dagsdeployer app (e.g. distroless). However, this may be a lot of work for not much advantage.
Refactor description:
- migrate most of the airflow logic out of composer_ops.go to a airflow_ops package
- Add a k8s wrapper for airflow cli commands to replace the gcloud wrapper
- Use the composer go client to retrieve k8s cluster details
Other considerations
This should include adding a k8s airflow deployment to the integration test infrastructure.
References:
k8s go client for kubectl exec from go code
composer go client (to retrieve k8s cluster details)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request