Yandex Cloud Managed Airflow Cluster

Features

• Create a managed Airflow cluster
• Easy to use in other resources via outputs

Airflow cluster definition

First, you need to create a VPC network with three subnets, bucket for DAG-files, and service account with role "managed-airflow.integrationProvider".

Airflow module requires the following input variables:

• Folder ID.
• VPC network subnet IDs.
• Service Account ID.
• Bucket Name.
• Admin Password.

Example

See examples section

How to configure Terraform for Yandex Cloud

• Install YC CLI
• Add environment variables for terraform auth in Yandex.Cloud

export YC_TOKEN=$(yc iam create-token)
export YC_CLOUD_ID=$(yc config get cloud-id)
export YC_FOLDER_ID=$(yc config get folder-id)

Requirements

Name	Version
terraform	>= 1.3.0
random	> 3.3
yandex	> 0.89

Providers

Name	Version
local	n/a
random	> 3.3
yandex	> 0.89

Modules

No modules.

Inputs

Name	Description	Type	Default	Required
folder_id	Folder id where Airflow cluster resides.	string	no	yes
description	Airflow cluster description.	string	"Managed Airflow cluster: terraform"	no
name	Airflow cluster name.	string	"airflow-cluster"	no
service_account_id	Yandex Cloud SA ID for Airflow	string	no	yes
security_group_ids	List of sg for Airflow cluster	list(string)	[ ]	no
bucket_name	Bucket name for DAG-file storage	string	no	yes
resource_preset_id	Preset for entities	string	"c1-m4"	no
entity	Airflow entities as webserver, scheduler, worker	object()	{ webserver = { count = 1 } scheduler = { count = 1 } worker = { min_count = 1 max_count = 2 } }	no
airflow_config	Optional Map of Map of String for Airflow configuration	map(map(string))	{ }	no
pip_packages	Optional List of pip packages	list(string)	[ ]	no
pip_packages	Optional List of deb packages	list(string)	[ ]	no
logging	A boolean variable for enable logging	bool	false	no
lockbox_secrets_backend	A boolean variable for enable lockbox secrets backend	bool	false	no
admin_password	The password for the Airflow with validation	string	no	yes

Outputs

Name	Description
airflow_cluster_name	Airflow cluster name
airflow_cluster_id	Airflow cluster id
airflow_admin_password	Airflow admin password (sensitive)
airflow_cluster_service_account_id	Service account ID for the Airflow cluster
airflow_cluster_logging_enabled	Whether logging is enabled for the Airflow cluster
airflow_cluster_min_worker_count	Minimum number of worker nodes in the Airflow cluster
airflow_cluster_max_worker_count	Maximum number of worker nodes in the Airflow cluster
airflow_cluster_s3_bucket	S3 bucket used for Airflow code sync