Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/generate-dags-from-yml.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ generate:
```

:::tip
If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs.md#configure-transformdbt-covesconfigyml-file). For [Fivetran](./run-fivetran-sync-jobs.md#configure-transformdbt-covesconfigyml-file) contact us to complete the setup.
If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs#generate-dags-from-yml-with-dbt-coves). For [Fivetran](./run-fivetran-sync-jobs#configure-your-transform-dbt-coves-config-yml-file) contact us to complete the setup.
:::
## Create the yml file for your Airflow DAG

Expand Down
2 changes: 1 addition & 1 deletion docs/how-tos/airflow/initial-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Each requires specific information to be provided during configuration. Our reco
- **Git branch name** The branch airflow will monitor for changes. If you have more than 1 environment (Development and Production), we suggest `airflow_development` for the development environment and `main` for the production environment. Note: You would need to create an `airflow_development` branch in your repo. If only have 1 environment, then the `Git branch name` should be `main`.

:::tip
We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](how-tos/datacoves/how_to_projects.md).
We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](../datacoves/how_to_projects/).
:::

**S3 Sync**
Expand Down
7 changes: 4 additions & 3 deletions docs/how-tos/airflow/run-fivetran-sync-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ Create a new connection using the following details:
:::tip
Once your Fivetran API key and secret have been generated, for security reasons, the secret cannot be viewed again through the Fivetran interface. If you lose or forget your API secret, you will need to generate a new API key and secret pair so be sure to store them somewhere secure for reference later. See <a href="https://fivetran.com/docs/rest-api/getting-started" target="_blank" rel="noopener">Fivetran Documentation</a> on how to generate your Fivetran `API Key` and `API Secret`.
:::
### Configure your transform/.dbt-coves/config.yml file

### Configure your transform dbt-coves config yml file

By default, dbt-coves cannot query the necessary information for Fivetran connections. You will need to configure these in your yml DAG manually, or contact us to configure Datacoves with the necessary information.
Below are the configurations in for dbt-coves airflow-dags. You will need to configure these if using dbt-coves to generate DAGs from YML
Expand Down Expand Up @@ -120,12 +121,12 @@ You will need to define two operators: `fivetran_provider.operators.fivetran.Fiv
![Fivetran Connection ID](assets/fivetran_connector_id.png)

- **do_xcom_push**: Indicate that the output of the task should be sent to XCom, making it available for other tasks to use.
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection).
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection).
- **example_task_sensor**: Name your Sensor task accordingly and define arguments below.
- **operator**: `fivetran_provider.sensors.fivetran.FivetranSensor`
- **connector_id**: Find in Fivetran UI.
- **poke_interval**: The poke interval is the time in seconds that the sensor waits before rechecking if the connector is done loading data. Defaults to 60.
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection).
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection).
- **dependencies**: A list of tasks this task depends on.

### YAML version
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/admin-menu/projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ A Project is the highest grouping in Datacoves. It is what contains environments
The Datacoves landing page (Launch Pad) follows this hierarchy:

:::tip
See our How To - [Projects](how-tos/datacoves/how_to_projects.md)
See our How To - [Projects](/docs/how-tos/datacoves/how_to_projects/)
:::

![Project Environment Difference](./assets/launchpad_environments_projects.png)
Expand Down
13 changes: 7 additions & 6 deletions static/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1196,7 +1196,7 @@ Each requires specific information to be provided during configuration. Our reco
- **Git branch name** The branch airflow will monitor for changes. If you have more than 1 environment (Development and Production), we suggest `airflow_development` for the development environment and `main` for the production environment. Note: You would need to create an `airflow_development` branch in your repo. If only have 1 environment, then the `Git branch name` should be `main`.

:::tip
We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](how-tos/datacoves/how_to_projects.md).
We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](../datacoves/how_to_projects/).
:::

**S3 Sync**
Expand Down Expand Up @@ -1963,7 +1963,7 @@ generate:
```

:::tip
If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs.md#configure-transformdbt-covesconfigyml-file). For [Fivetran](./run-fivetran-sync-jobs.md#configure-transformdbt-covesconfigyml-file) contact us to complete the setup.
If using an Extract and Load tool in your DAG you can dynamically generate your sources; however, additional configuration will be needed inside the config.yml file. See [Airbyte](./run-airbyte-sync-jobs#generate-dags-from-yml-with-dbt-coves). For [Fivetran](./run-fivetran-sync-jobs#configure-your-transform-dbt-coves-config-yml-file) contact us to complete the setup.
:::
## Create the yml file for your Airflow DAG

Expand Down Expand Up @@ -2828,7 +2828,8 @@ Create a new connection using the following details:
:::tip
Once your Fivetran API key and secret have been generated, for security reasons, the secret cannot be viewed again through the Fivetran interface. If you lose or forget your API secret, you will need to generate a new API key and secret pair so be sure to store them somewhere secure for reference later. See <a href="https://fivetran.com/docs/rest-api/getting-started" target="_blank" rel="noopener">Fivetran Documentation</a> on how to generate your Fivetran `API Key` and `API Secret`.
:::
### Configure your transform/.dbt-coves/config.yml file

### Configure your transform dbt-coves config yml file

By default, dbt-coves cannot query the necessary information for Fivetran connections. You will need to configure these in your yml DAG manually, or contact us to configure Datacoves with the necessary information.
Below are the configurations in for dbt-coves airflow-dags. You will need to configure these if using dbt-coves to generate DAGs from YML
Expand Down Expand Up @@ -2921,12 +2922,12 @@ You will need to define two operators: `fivetran_provider.operators.fivetran.Fiv
![Fivetran Connection ID](assets/fivetran_connector_id.png)

- **do_xcom_push**: Indicate that the output of the task should be sent to XCom, making it available for other tasks to use.
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection).
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection).
- **example_task_sensor**: Name your Sensor task accordingly and define arguments below.
- **operator**: `fivetran_provider.sensors.fivetran.FivetranSensor`
- **connector_id**: Find in Fivetran UI.
- **poke_interval**: The poke interval is the time in seconds that the sensor waits before rechecking if the connector is done loading data. Defaults to 60.
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#id=fivetran-connection).
- **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection).
- **dependencies**: A list of tasks this task depends on.

### YAML version
Expand Down Expand Up @@ -7624,7 +7625,7 @@ A Project is the highest grouping in Datacoves. It is what contains environments
The Datacoves landing page (Launch Pad) follows this hierarchy:

:::tip
See our How To - [Projects](how-tos/datacoves/how_to_projects.md)
See our How To - [Projects](/docs/how-tos/datacoves/how_to_projects/)
:::

![Project Environment Difference](./assets/launchpad_environments_projects.png)
Expand Down