diff --git a/docs/how-tos/airflow/generate-dags-from-yml.md b/docs/how-tos/airflow/generate-dags-from-yml.md index ab6c6b4..b82e56b 100644 --- a/docs/how-tos/airflow/generate-dags-from-yml.md +++ b/docs/how-tos/airflow/generate-dags-from-yml.md @@ -34,7 +34,7 @@ generate: ``` :::tip -If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs.md#configure-transformdbt-covesconfigyml-file). For [Fivetran](./run-fivetran-sync-jobs.md#configure-transformdbt-covesconfigyml-file) contact us to complete the setup. +If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs#generate-dags-from-yml-with-dbt-coves). For [Fivetran](./run-fivetran-sync-jobs#configure-your-transform-dbt-coves-config-yml-file) contact us to complete the setup. ::: ## Create the yml file for your Airflow DAG diff --git a/docs/how-tos/airflow/initial-setup.md b/docs/how-tos/airflow/initial-setup.md index b3f4551..9f64ce4 100644 --- a/docs/how-tos/airflow/initial-setup.md +++ b/docs/how-tos/airflow/initial-setup.md @@ -39,7 +39,7 @@ Each requires specific information to be provided during configuration. Our reco - **Git branch name** The branch airflow will monitor for changes. If you have more than 1 environment (Development and Production), we suggest `airflow_development` for the development environment and `main` for the production environment. Note: You would need to create an `airflow_development` branch in your repo. If only have 1 environment, then the `Git branch name` should be `main`. :::tip -We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](how-tos/datacoves/how_to_projects.md). +We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](../datacoves/how_to_projects/). ::: **S3 Sync** diff --git a/docs/how-tos/airflow/run-fivetran-sync-jobs.md b/docs/how-tos/airflow/run-fivetran-sync-jobs.md index d928020..6f02b65 100644 --- a/docs/how-tos/airflow/run-fivetran-sync-jobs.md +++ b/docs/how-tos/airflow/run-fivetran-sync-jobs.md @@ -27,7 +27,8 @@ Create a new connection using the following details: :::tip Once your Fivetran API key and secret have been generated, for security reasons, the secret cannot be viewed again through the Fivetran interface. If you lose or forget your API secret, you will need to generate a new API key and secret pair so be sure to store them somewhere secure for reference later. See Fivetran Documentation on how to generate your Fivetran `API Key` and `API Secret`. ::: -### Configure your transform/.dbt-coves/config.yml file + +### Configure your transform dbt-coves config yml file By default, dbt-coves cannot query the necessary information for Fivetran connections. You will need to configure these in your yml DAG manually, or contact us to configure Datacoves with the necessary information. Below are the configurations in for dbt-coves airflow-dags. You will need to configure these if using dbt-coves to generate DAGs from YML @@ -120,12 +121,12 @@ You will need to define two operators: `fivetran_provider.operators.fivetran.Fiv ![Fivetran Connection ID](assets/fivetran_connector_id.png) - **do_xcom_push**: Indicate that the output of the task should be sent to XCom, making it available for other tasks to use. - - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection). + - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection). - **example_task_sensor**: Name your Sensor task accordingly and define arguments below. - **operator**: `fivetran_provider.sensors.fivetran.FivetranSensor` - **connector_id**: Find in Fivetran UI. - **poke_interval**: The poke interval is the time in seconds that the sensor waits before rechecking if the connector is done loading data. Defaults to 60. - - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection). + - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection). - **dependencies**: A list of tasks this task depends on. ### YAML version diff --git a/docs/reference/admin-menu/projects.md b/docs/reference/admin-menu/projects.md index b71558a..0c67277 100644 --- a/docs/reference/admin-menu/projects.md +++ b/docs/reference/admin-menu/projects.md @@ -11,7 +11,7 @@ A Project is the highest grouping in Datacoves. It is what contains environments The Datacoves landing page (Launch Pad) follows this hierarchy: :::tip -See our How To - [Projects](how-tos/datacoves/how_to_projects.md) +See our How To - [Projects](/docs/how-tos/datacoves/how_to_projects/) ::: ![Project Environment Difference](./assets/launchpad_environments_projects.png) diff --git a/static/llms.txt b/static/llms.txt index 383cae6..d7c1e6c 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -1196,7 +1196,7 @@ Each requires specific information to be provided during configuration. Our reco - **Git branch name** The branch airflow will monitor for changes. If you have more than 1 environment (Development and Production), we suggest `airflow_development` for the development environment and `main` for the production environment. Note: You would need to create an `airflow_development` branch in your repo. If only have 1 environment, then the `Git branch name` should be `main`. :::tip -We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](how-tos/datacoves/how_to_projects.md). +We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](../datacoves/how_to_projects/). ::: **S3 Sync** @@ -1963,7 +1963,7 @@ generate: ``` :::tip -If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs.md#configure-transformdbt-covesconfigyml-file). For [Fivetran](./run-fivetran-sync-jobs.md#configure-transformdbt-covesconfigyml-file) contact us to complete the setup. +If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs#generate-dags-from-yml-with-dbt-coves). For [Fivetran](./run-fivetran-sync-jobs#configure-your-transform-dbt-coves-config-yml-file) contact us to complete the setup. ::: ## Create the yml file for your Airflow DAG @@ -2828,7 +2828,8 @@ Create a new connection using the following details: :::tip Once your Fivetran API key and secret have been generated, for security reasons, the secret cannot be viewed again through the Fivetran interface. If you lose or forget your API secret, you will need to generate a new API key and secret pair so be sure to store them somewhere secure for reference later. See Fivetran Documentation on how to generate your Fivetran `API Key` and `API Secret`. ::: -### Configure your transform/.dbt-coves/config.yml file + +### Configure your transform dbt-coves config yml file By default, dbt-coves cannot query the necessary information for Fivetran connections. You will need to configure these in your yml DAG manually, or contact us to configure Datacoves with the necessary information. Below are the configurations in for dbt-coves airflow-dags. You will need to configure these if using dbt-coves to generate DAGs from YML @@ -2921,12 +2922,12 @@ You will need to define two operators: `fivetran_provider.operators.fivetran.Fiv ![Fivetran Connection ID](assets/fivetran_connector_id.png) - **do_xcom_push**: Indicate that the output of the task should be sent to XCom, making it available for other tasks to use. - - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection). + - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection). - **example_task_sensor**: Name your Sensor task accordingly and define arguments below. - **operator**: `fivetran_provider.sensors.fivetran.FivetranSensor` - **connector_id**: Find in Fivetran UI. - **poke_interval**: The poke interval is the time in seconds that the sensor waits before rechecking if the connector is done loading data. Defaults to 60. - - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection). + - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection). - **dependencies**: A list of tasks this task depends on. ### YAML version @@ -7624,7 +7625,7 @@ A Project is the highest grouping in Datacoves. It is what contains environments The Datacoves landing page (Launch Pad) follows this hierarchy: :::tip -See our How To - [Projects](how-tos/datacoves/how_to_projects.md) +See our How To - [Projects](/docs/how-tos/datacoves/how_to_projects/) ::: ![Project Environment Difference](./assets/launchpad_environments_projects.png)