Kubernetes Variables for AWS MWAA support with Airflow#1234
Closed
valayDave wants to merge 4 commits intovalay/airflow-gcp-finalfrom
Closed
Kubernetes Variables for AWS MWAA support with Airflow#1234valayDave wants to merge 4 commits intovalay/airflow-gcp-finalfrom
valayDave wants to merge 4 commits intovalay/airflow-gcp-finalfrom
Conversation
savingoyal
reviewed
Jan 19, 2023
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
4499490 to
2a55cb6
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
5c97b11 to
172c228
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
2a55cb6 to
ecc43a0
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
172c228 to
fbe23d8
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
ecc43a0 to
8c9881f
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
fbe23d8 to
aff5a37
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
8c9881f to
322969f
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
aff5a37 to
0821a0a
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
322969f to
68d46d9
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
0821a0a to
85f219c
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
68d46d9 to
110ec3b
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
832191e to
98f2db1
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
110ec3b to
cbc9b8e
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
98f2db1 to
5adba1f
Compare
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
cbc9b8e to
b0f7b72
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
5adba1f to
3862a14
Compare
Contributor
romain-intel
left a comment
romain-intel
left a comment
There was a problem hiding this comment.
Did not review. No concern except the proliferation of config variables...
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
b0f7b72 to
f1b429f
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
3862a14 to
272d86e
Compare
This was referenced Jan 30, 2023
valayDave
commented
Jan 30, 2023
Collaborator
Author
valayDave
left a comment
valayDave
left a comment
There was a problem hiding this comment.
Lopped some small changes relating to DOCStrings + SQL Sensor Support for Airflow version > 2.4.0
Comment on lines
+21
to
+60
| """ | ||
| The `@airflow_external_task_sensor` decorator attaches a Airflow [ExternalTaskSensor](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/sensors/external_task/index.html#airflow.sensors.external_task.ExternalTaskSensor) before the start step of the flow. | ||
| This decorator only works when a flow is scheduled on Airflow and is compiled using `airflow create`. More than one `@airflow_external_task_sensor` can be added as a flow decorators. Adding more than one decorator will ensure that `start` step starts only after all sensors finish. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| timeout : int | ||
| Time, in seconds before the task times out and fails. (Default: 3600) | ||
| poke_interval : int | ||
| Time in seconds that the job should wait in between each try. (Default: 60) | ||
| mode : string | ||
| How the sensor operates. Options are: { poke | reschedule }. (Default: "poke") | ||
| exponential_backoff : bool | ||
| allow progressive longer waits between pokes by using exponential backoff algorithm. (Default: True) | ||
| pool : string | ||
| the slot pool this task should run in, | ||
| slot pools are a way to limit concurrency for certain tasks. (Default:None) | ||
| soft_fail : bool | ||
| Set to true to mark the task as SKIPPED on failure. (Default: False) | ||
| name : string | ||
| Name of the sensor on Airflow | ||
| description : string | ||
| Description of sensor in the Airflow UI | ||
| external_dag_id : string | ||
| The dag_id that contains the task you want to wait for. | ||
| external_task_ids : List[string] | ||
| The list of task_ids that you want to wait for. | ||
| If None (default value) the sensor waits for the DAG. (Default: None) | ||
| allowed_states : List[string] | ||
| Iterable of allowed states, (Default: ['success']) | ||
| failed_states : List[string] | ||
| Iterable of failed or dis-allowed states. (Default: None) | ||
| execution_delta : datetime.timedelta | ||
| time difference with the previous execution to look at, | ||
| the default is the same logical date as the current task or DAG. (Default: None) | ||
| check_existence: bool | ||
| Set to True to check if the external task exists or check if | ||
| the DAG to wait for exists. (Default: True) | ||
| """ | ||
|
|
Collaborator
Author
There was a problem hiding this comment.
@savingoyal : Added Docstrings for all 3 decorators. Please review when you get the chance.
Comment on lines
+427
to
+437
| try: | ||
| from airflow.sensors.sql import SqlSensor | ||
| except ImportError as e: | ||
| try: | ||
| from airflow.providers.common.sql.sensors.sql import SqlSensor | ||
| except ImportError as e: | ||
| raise AirflowSensorNotFound( | ||
| "This DAG requires a `SqlSensor`. " | ||
| "Install the Airflow SQL provider using : " | ||
| "`pip install apache-airflow-providers-common-sql`" | ||
| ) |
Collaborator
Author
There was a problem hiding this comment.
Had to make this change so that SQL sensors can be supported. Airflow Recently removed SQL sensors from Airflow v2.4.0 . This is why we have this additional code path added here.
- Fixed SQL Sensor import.
valayDave
force-pushed
the
valay/airflow-gcp-final
branch
from
f1b429f to
a1edca6
Compare
valayDave
force-pushed
the
valay/airflow-mwaa-support
branch
from
0f00a44 to
cb27496
Compare
This was referenced Jan 31, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on top of #1226
Introduces new config variables :
METAFLOW_AIRFLOW_KUBERNETES_KUBECONFIG_CONTEXT: Context to use in the kubeconfig file.METAFLOW_AIRFLOW_KUBERNETES_KUBECONFIG_FILE: Path to the kubeconfig file to provide the Airflow KubernetesOperatorSetting Either of these or
METAFLOW_AIRFLOW_KUBERNETES_CONN_IDwill setin_cluster=Truein the AirflowKubernetesOperatorChanges done to support AWS MWAA