From 0f6a905f5bd6a05730448e9004b0a7bef132a33d Mon Sep 17 00:00:00 2001 From: Amy Chan Date: Wed, 17 Sep 2025 11:58:57 -0700 Subject: [PATCH] improved navigation changes adding dynamic pages and re-ordering --- docs/best-practices/vscode/README.md | 5 + .../vscode/tips.md | 0 .../Admin/{README.mdx => README.txt} | 2 +- docs/getting-started/Admin/_category_.yaml | 8 + .../Admin/configure-airflow.md | 2 +- .../configure-repository-using-dbt-coves.md | 2 +- .../Admin/configure-repository.md | 4 +- .../Admin/creating-airflow-dags.md | 10 +- docs/getting-started/Admin/user-management.md | 2 +- .../developer/{README.md => README.txt} | 0 .../getting-started/developer/_category_.yaml | 7 + docs/how-tos/_category_.yaml | 2 +- docs/how-tos/airflow/DAGs/_category_.yaml | 6 + .../{ => DAGs}/assets/admin-connections.png | Bin .../assets/airbyte-connection-details.png | Bin .../assets/airbyte-replication-frequency.png | Bin .../assets/airflow_adf_connection.png | Bin .../assets/airflow_databricks_connection.png | Bin .../{ => DAGs}/assets/databricks_compute.png | Bin .../{ => DAGs}/assets/databricks_copyurl.png | Bin .../assets/extract_load_airflow_dag.png | Bin .../assets/fivetran-connection-details.png | Bin .../assets/fivetran_connector_id.png | Bin .../assets/generate_airflow_dag.gif | Bin .../{ => DAGs}/assets/see-dag-docs.gif | Bin .../{ => DAGs}/create-dag-level-docs.md | 0 .../{ => DAGs}/dynamically-set-schedule.md | 0 .../airflow/{ => DAGs}/external-python-dag.md | 0 .../{ => DAGs}/generate-dags-from-yml.md | 0 .../{ => DAGs}/get-current-branch-name.md | 0 .../airflow/{ => DAGs}/retry-dbt-tasks.md | 0 .../airflow/{ => DAGs}/run-adf-pipeline.md | 4 +- .../{ => DAGs}/run-airbyte-sync-jobs.md | 2 +- .../{ => DAGs}/run-databricks-notebook.md | 8 +- docs/how-tos/airflow/{ => DAGs}/run-dbt.md | 0 .../{ => DAGs}/run-fivetran-sync-jobs.md | 6 +- .../airflow/{ => DAGs}/s3-to-snowflake.md | 14 +- docs/how-tos/airflow/{ => DAGs}/test-dags.md | 0 .../use-variables-and-connections.md | 8 +- docs/how-tos/airflow/README.md | 19 +- docs/how-tos/airflow/_category_.yaml | 6 + docs/how-tos/airflow/_category_bk.yaml | 6 - docs/how-tos/airflow/send-emails.md | 2 +- .../datacoves/{README.md => README.txt} | 0 docs/how-tos/datacoves/_category_.yaml | 6 + .../how_to_configure_aws_secrets_manager.md | 2 +- .../how-tos/datahub/{README.md => README.txt} | 0 docs/how-tos/datahub/_category_.yaml | 7 + docs/how-tos/dataops/README.md | 2 +- docs/how-tos/dbt/{README.md => README.txt} | 0 docs/how-tos/dbt/_category_.yaml | 7 + docs/how-tos/git/README.md | 2 +- docs/how-tos/metrics-and-logs/README.md | 4 +- docs/how-tos/my_airflow/README.mdx | 2 +- .../snowflake/{README.md => README.txt} | 0 docs/how-tos/snowflake/_category_.yaml | 7 + .../superset/{README.md => README.txt} | 0 docs/how-tos/superset/_category_.yaml | 7 + docs/how-tos/vscode/{README.md => README.txt} | 0 docs/how-tos/vscode/_category_.yaml | 6 + docs/how-tos/vscode/datacoves-copilot/llm.mdx | 78 - .../vscode/datacoves-copilot/v1-llm-config.md | 4 +- .../vscode/datacoves-copilot/v1-llm.mdx | 8 +- docs/how-tos/vscode/datacoves-copilot/v1.md | 3 +- .../vscode/datacoves-copilot/v2-llm-config.md | 4 +- docs/how-tos/vscode/datacoves-copilot/v2.md | 3 +- docs/reference/admin-menu/README.md | 2 +- .../airflow/{README.md => README.txt} | 2 +- docs/reference/airflow/_category_.yaml | 6 + .../reference/airflow/datacoves-decorators.md | 2 +- .../datacoves/{README.md => README.txt} | 0 docs/reference/datacoves/_category_.yaml | 6 + .../assets/csv_edit_button.jpg | Bin .../assets/csv_edit_view.jpg | Bin .../assets/csv_rainbow_view.jpg | Bin .../{vscode => datacoves}/csv-in-datacoves.md | 0 .../metrics-and-logs/README.md | 0 .../assets/dashboards-list.png | Bin .../assets/dashboards-nav.png | Bin .../assets/metrics-and-logs-nav.png | Bin .../metrics-and-logs/grafana.md | 0 docs/reference/vscode/README.md | 4 +- docusaurus.config.js | 65 +- src/pages/sla.md | 86 + static/llms.txt | 3933 ++++++++--------- 85 files changed, 2128 insertions(+), 2255 deletions(-) create mode 100644 docs/best-practices/vscode/README.md rename docs/{reference => best-practices}/vscode/tips.md (100%) rename docs/getting-started/Admin/{README.mdx => README.txt} (97%) create mode 100644 docs/getting-started/Admin/_category_.yaml rename docs/getting-started/developer/{README.md => README.txt} (100%) create mode 100644 docs/getting-started/developer/_category_.yaml create mode 100644 docs/how-tos/airflow/DAGs/_category_.yaml rename docs/how-tos/airflow/{ => DAGs}/assets/admin-connections.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/airbyte-connection-details.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/airbyte-replication-frequency.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/airflow_adf_connection.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/airflow_databricks_connection.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/databricks_compute.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/databricks_copyurl.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/extract_load_airflow_dag.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/fivetran-connection-details.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/fivetran_connector_id.png (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/generate_airflow_dag.gif (100%) rename docs/how-tos/airflow/{ => DAGs}/assets/see-dag-docs.gif (100%) rename docs/how-tos/airflow/{ => DAGs}/create-dag-level-docs.md (100%) rename docs/how-tos/airflow/{ => DAGs}/dynamically-set-schedule.md (100%) rename docs/how-tos/airflow/{ => DAGs}/external-python-dag.md (100%) rename docs/how-tos/airflow/{ => DAGs}/generate-dags-from-yml.md (100%) rename docs/how-tos/airflow/{ => DAGs}/get-current-branch-name.md (100%) rename docs/how-tos/airflow/{ => DAGs}/retry-dbt-tasks.md (100%) rename docs/how-tos/airflow/{ => DAGs}/run-adf-pipeline.md (97%) rename docs/how-tos/airflow/{ => DAGs}/run-airbyte-sync-jobs.md (98%) rename docs/how-tos/airflow/{ => DAGs}/run-databricks-notebook.md (96%) rename docs/how-tos/airflow/{ => DAGs}/run-dbt.md (100%) rename docs/how-tos/airflow/{ => DAGs}/run-fivetran-sync-jobs.md (96%) rename docs/how-tos/airflow/{ => DAGs}/s3-to-snowflake.md (91%) rename docs/how-tos/airflow/{ => DAGs}/test-dags.md (100%) rename docs/how-tos/airflow/{ => DAGs}/use-variables-and-connections.md (90%) create mode 100644 docs/how-tos/airflow/_category_.yaml delete mode 100644 docs/how-tos/airflow/_category_bk.yaml rename docs/how-tos/datacoves/{README.md => README.txt} (100%) create mode 100644 docs/how-tos/datacoves/_category_.yaml rename docs/how-tos/datahub/{README.md => README.txt} (100%) create mode 100644 docs/how-tos/datahub/_category_.yaml rename docs/how-tos/dbt/{README.md => README.txt} (100%) create mode 100644 docs/how-tos/dbt/_category_.yaml rename docs/how-tos/snowflake/{README.md => README.txt} (100%) create mode 100644 docs/how-tos/snowflake/_category_.yaml rename docs/how-tos/superset/{README.md => README.txt} (100%) create mode 100644 docs/how-tos/superset/_category_.yaml rename docs/how-tos/vscode/{README.md => README.txt} (100%) create mode 100644 docs/how-tos/vscode/_category_.yaml delete mode 100644 docs/how-tos/vscode/datacoves-copilot/llm.mdx rename docs/reference/airflow/{README.md => README.txt} (66%) create mode 100644 docs/reference/airflow/_category_.yaml rename docs/reference/datacoves/{README.md => README.txt} (100%) create mode 100644 docs/reference/datacoves/_category_.yaml rename docs/reference/{vscode => datacoves}/assets/csv_edit_button.jpg (100%) rename docs/reference/{vscode => datacoves}/assets/csv_edit_view.jpg (100%) rename docs/reference/{vscode => datacoves}/assets/csv_rainbow_view.jpg (100%) rename docs/reference/{vscode => datacoves}/csv-in-datacoves.md (100%) rename docs/reference/{ => datacoves}/metrics-and-logs/README.md (100%) rename docs/reference/{ => datacoves}/metrics-and-logs/assets/dashboards-list.png (100%) rename docs/reference/{ => datacoves}/metrics-and-logs/assets/dashboards-nav.png (100%) rename docs/reference/{ => datacoves}/metrics-and-logs/assets/metrics-and-logs-nav.png (100%) rename docs/reference/{ => datacoves}/metrics-and-logs/grafana.md (100%) create mode 100644 src/pages/sla.md diff --git a/docs/best-practices/vscode/README.md b/docs/best-practices/vscode/README.md new file mode 100644 index 00000000..aec87184 --- /dev/null +++ b/docs/best-practices/vscode/README.md @@ -0,0 +1,5 @@ +--- +title: VS Code +sidebar_position: 5 +--- +# VS Cod Best Practices and Tips diff --git a/docs/reference/vscode/tips.md b/docs/best-practices/vscode/tips.md similarity index 100% rename from docs/reference/vscode/tips.md rename to docs/best-practices/vscode/tips.md diff --git a/docs/getting-started/Admin/README.mdx b/docs/getting-started/Admin/README.txt similarity index 97% rename from docs/getting-started/Admin/README.mdx rename to docs/getting-started/Admin/README.txt index 3f5d968c..9b76d4f3 100644 --- a/docs/getting-started/Admin/README.mdx +++ b/docs/getting-started/Admin/README.txt @@ -1,6 +1,6 @@ --- title: Administrator -sidebar_position: 1 +sidebar_position: 3 --- # Datacoves Getting Started Guides - Admin diff --git a/docs/getting-started/Admin/_category_.yaml b/docs/getting-started/Admin/_category_.yaml new file mode 100644 index 00000000..d4769986 --- /dev/null +++ b/docs/getting-started/Admin/_category_.yaml @@ -0,0 +1,8 @@ +label: Administrator +position: 3 +link: + type: generated-index + title: Administrator + description: Welcome Administrator! You have the important role of configuring the Datacoves platform to fit your needs. No worries, you are not alone. We are here to help you every step of the way so you and your team can start delivering valuable insights in no time! Please follow the documentation below to get your dbt project up and running. + + diff --git a/docs/getting-started/Admin/configure-airflow.md b/docs/getting-started/Admin/configure-airflow.md index 454f290c..189e1870 100644 --- a/docs/getting-started/Admin/configure-airflow.md +++ b/docs/getting-started/Admin/configure-airflow.md @@ -1,5 +1,5 @@ --- -title: Configure Airflow +title: Airflow - Configure sidebar_position: 2 --- # Configuring Airflow diff --git a/docs/getting-started/Admin/configure-repository-using-dbt-coves.md b/docs/getting-started/Admin/configure-repository-using-dbt-coves.md index f40b0a87..f0d8eed9 100644 --- a/docs/getting-started/Admin/configure-repository-using-dbt-coves.md +++ b/docs/getting-started/Admin/configure-repository-using-dbt-coves.md @@ -1,6 +1,6 @@ --- title: Configure Git Repository Using dbt-coves -sidebar_position: 4 +sidebar_position: 12 --- # Initial Datacoves Repository Setup diff --git a/docs/getting-started/Admin/configure-repository.md b/docs/getting-started/Admin/configure-repository.md index be1fb20f..80adb494 100644 --- a/docs/getting-started/Admin/configure-repository.md +++ b/docs/getting-started/Admin/configure-repository.md @@ -1,6 +1,6 @@ --- title: Configure Git Repository -sidebar_position: 3 +sidebar_position: 8 --- # Update Repository for Airflow @@ -12,7 +12,7 @@ Now that you have configured your Airflow settings you must ensure that your rep **Step 3:** **This step is optional** if you would like to make use of the [dbt-coves](https://github.com/datacoves/dbt-coves?tab=readme-ov-file#airflow-dags-generation-arguments) `dbt-coves generate airflow-dags` command. Create the `dags_yml_definitions` folder inside of your newly created `orchestrate` folder. This will leave you with two folders inside `orchestrate`- `orchestrate/dags` and `orchestrate/dags_yml_definitions`. -**Step 4:** **This step is optional** if you would like to make use of the dbt-coves' extension `dbt-coves generate airflow-dags` command. You must create a config file for dbt-coves. Please follow the [generate DAGs from yml](how-tos/airflow/generate-dags-from-yml.md) docs. +**Step 4:** **This step is optional** if you would like to make use of the dbt-coves' extension `dbt-coves generate airflow-dags` command. You must create a config file for dbt-coves. Please follow the [generate DAGs from yml](how-tos/airflow/DAGs/generate-dags-from-yml.md) docs. ## Create a profiles.yml diff --git a/docs/getting-started/Admin/creating-airflow-dags.md b/docs/getting-started/Admin/creating-airflow-dags.md index 66395abb..6d608361 100644 --- a/docs/getting-started/Admin/creating-airflow-dags.md +++ b/docs/getting-started/Admin/creating-airflow-dags.md @@ -1,6 +1,6 @@ --- -title: Creating Airflow Dags -sidebar_position: 7 +title: Airflow - Creating Dags +sidebar_position: 3 --- # Creating Airflow Dags @@ -19,12 +19,12 @@ During the Airflow configuration step you added the `orchestrate` folder and the ## DAG 101 in Datacoves 1. If you are eager to see Airflow and dbt in action within Datacoves, here is the simplest way to run dbt with Airflow. - [Run dbt](/docs/how-tos/airflow/run-dbt) + [Run dbt](/docs/how-tos/airflow/DAGs/run-dbt) 2. You have 2 options when it comes to writing DAGs in Datacoves. You can write them out using Python and place them in the `orchestrate/dags` directory, or you can generate your DAGs with `dbt-coves` from a YML definition. - [Generate DAGs from yml definitions](/docs/how-tos/airflow/generate-dags-from-yml) this is simpler for users not accustomed to using Python + [Generate DAGs from yml definitions](/docs/how-tos/airflow/DAGs/generate-dags-from-yml) this is simpler for users not accustomed to using Python 3. You may also wish to use external libraries in your DAGs such as Pandas. In order to do that effectively, you can create custom Python scripts in a separate directory such as `orchestrate/python_scripts` and use the `DatacovesBashOperator` to handle all the behind the scenes work as well as run your custom script.**You will need to contact us beforehand to pre-configure any python libraries you need.** - [External Python DAG](/docs/how-tos/airflow/external-python-dag) + [External Python DAG](/docs/how-tos/airflow/DAGs/external-python-dag) diff --git a/docs/getting-started/Admin/user-management.md b/docs/getting-started/Admin/user-management.md index 7e32b52d..274de2c7 100644 --- a/docs/getting-started/Admin/user-management.md +++ b/docs/getting-started/Admin/user-management.md @@ -1,6 +1,6 @@ --- title: User Management -sidebar_position: 8 +sidebar_position: 20 --- # User Management diff --git a/docs/getting-started/developer/README.md b/docs/getting-started/developer/README.txt similarity index 100% rename from docs/getting-started/developer/README.md rename to docs/getting-started/developer/README.txt diff --git a/docs/getting-started/developer/_category_.yaml b/docs/getting-started/developer/_category_.yaml new file mode 100644 index 00000000..96e3e598 --- /dev/null +++ b/docs/getting-started/developer/_category_.yaml @@ -0,0 +1,7 @@ +label: Developer +position: 5 +link: + type: generated-index + title: Developer + description: Welcome Developer! Please use these getting started guides to accelerate your onboarding. The guides cover everything from configuring your user settings to fundamental tips and shortcuts. + diff --git a/docs/how-tos/_category_.yaml b/docs/how-tos/_category_.yaml index ac37a156..db3239c2 100644 --- a/docs/how-tos/_category_.yaml +++ b/docs/how-tos/_category_.yaml @@ -1,5 +1,5 @@ label: How-tos -position: 2 +position: 1 link: type: generated-index title: How-to Guides diff --git a/docs/how-tos/airflow/DAGs/_category_.yaml b/docs/how-tos/airflow/DAGs/_category_.yaml new file mode 100644 index 00000000..2e89053e --- /dev/null +++ b/docs/how-tos/airflow/DAGs/_category_.yaml @@ -0,0 +1,6 @@ +label: DAGs +position: 11 +link: + type: generated-index + title: DAGs + description: Here you will find information on DAGs in Datacoves. diff --git a/docs/how-tos/airflow/assets/admin-connections.png b/docs/how-tos/airflow/DAGs/assets/admin-connections.png similarity index 100% rename from docs/how-tos/airflow/assets/admin-connections.png rename to docs/how-tos/airflow/DAGs/assets/admin-connections.png diff --git a/docs/how-tos/airflow/assets/airbyte-connection-details.png b/docs/how-tos/airflow/DAGs/assets/airbyte-connection-details.png similarity index 100% rename from docs/how-tos/airflow/assets/airbyte-connection-details.png rename to docs/how-tos/airflow/DAGs/assets/airbyte-connection-details.png diff --git a/docs/how-tos/airflow/assets/airbyte-replication-frequency.png b/docs/how-tos/airflow/DAGs/assets/airbyte-replication-frequency.png similarity index 100% rename from docs/how-tos/airflow/assets/airbyte-replication-frequency.png rename to docs/how-tos/airflow/DAGs/assets/airbyte-replication-frequency.png diff --git a/docs/how-tos/airflow/assets/airflow_adf_connection.png b/docs/how-tos/airflow/DAGs/assets/airflow_adf_connection.png similarity index 100% rename from docs/how-tos/airflow/assets/airflow_adf_connection.png rename to docs/how-tos/airflow/DAGs/assets/airflow_adf_connection.png diff --git a/docs/how-tos/airflow/assets/airflow_databricks_connection.png b/docs/how-tos/airflow/DAGs/assets/airflow_databricks_connection.png similarity index 100% rename from docs/how-tos/airflow/assets/airflow_databricks_connection.png rename to docs/how-tos/airflow/DAGs/assets/airflow_databricks_connection.png diff --git a/docs/how-tos/airflow/assets/databricks_compute.png b/docs/how-tos/airflow/DAGs/assets/databricks_compute.png similarity index 100% rename from docs/how-tos/airflow/assets/databricks_compute.png rename to docs/how-tos/airflow/DAGs/assets/databricks_compute.png diff --git a/docs/how-tos/airflow/assets/databricks_copyurl.png b/docs/how-tos/airflow/DAGs/assets/databricks_copyurl.png similarity index 100% rename from docs/how-tos/airflow/assets/databricks_copyurl.png rename to docs/how-tos/airflow/DAGs/assets/databricks_copyurl.png diff --git a/docs/how-tos/airflow/assets/extract_load_airflow_dag.png b/docs/how-tos/airflow/DAGs/assets/extract_load_airflow_dag.png similarity index 100% rename from docs/how-tos/airflow/assets/extract_load_airflow_dag.png rename to docs/how-tos/airflow/DAGs/assets/extract_load_airflow_dag.png diff --git a/docs/how-tos/airflow/assets/fivetran-connection-details.png b/docs/how-tos/airflow/DAGs/assets/fivetran-connection-details.png similarity index 100% rename from docs/how-tos/airflow/assets/fivetran-connection-details.png rename to docs/how-tos/airflow/DAGs/assets/fivetran-connection-details.png diff --git a/docs/how-tos/airflow/assets/fivetran_connector_id.png b/docs/how-tos/airflow/DAGs/assets/fivetran_connector_id.png similarity index 100% rename from docs/how-tos/airflow/assets/fivetran_connector_id.png rename to docs/how-tos/airflow/DAGs/assets/fivetran_connector_id.png diff --git a/docs/how-tos/airflow/assets/generate_airflow_dag.gif b/docs/how-tos/airflow/DAGs/assets/generate_airflow_dag.gif similarity index 100% rename from docs/how-tos/airflow/assets/generate_airflow_dag.gif rename to docs/how-tos/airflow/DAGs/assets/generate_airflow_dag.gif diff --git a/docs/how-tos/airflow/assets/see-dag-docs.gif b/docs/how-tos/airflow/DAGs/assets/see-dag-docs.gif similarity index 100% rename from docs/how-tos/airflow/assets/see-dag-docs.gif rename to docs/how-tos/airflow/DAGs/assets/see-dag-docs.gif diff --git a/docs/how-tos/airflow/create-dag-level-docs.md b/docs/how-tos/airflow/DAGs/create-dag-level-docs.md similarity index 100% rename from docs/how-tos/airflow/create-dag-level-docs.md rename to docs/how-tos/airflow/DAGs/create-dag-level-docs.md diff --git a/docs/how-tos/airflow/dynamically-set-schedule.md b/docs/how-tos/airflow/DAGs/dynamically-set-schedule.md similarity index 100% rename from docs/how-tos/airflow/dynamically-set-schedule.md rename to docs/how-tos/airflow/DAGs/dynamically-set-schedule.md diff --git a/docs/how-tos/airflow/external-python-dag.md b/docs/how-tos/airflow/DAGs/external-python-dag.md similarity index 100% rename from docs/how-tos/airflow/external-python-dag.md rename to docs/how-tos/airflow/DAGs/external-python-dag.md diff --git a/docs/how-tos/airflow/generate-dags-from-yml.md b/docs/how-tos/airflow/DAGs/generate-dags-from-yml.md similarity index 100% rename from docs/how-tos/airflow/generate-dags-from-yml.md rename to docs/how-tos/airflow/DAGs/generate-dags-from-yml.md diff --git a/docs/how-tos/airflow/get-current-branch-name.md b/docs/how-tos/airflow/DAGs/get-current-branch-name.md similarity index 100% rename from docs/how-tos/airflow/get-current-branch-name.md rename to docs/how-tos/airflow/DAGs/get-current-branch-name.md diff --git a/docs/how-tos/airflow/retry-dbt-tasks.md b/docs/how-tos/airflow/DAGs/retry-dbt-tasks.md similarity index 100% rename from docs/how-tos/airflow/retry-dbt-tasks.md rename to docs/how-tos/airflow/DAGs/retry-dbt-tasks.md diff --git a/docs/how-tos/airflow/run-adf-pipeline.md b/docs/how-tos/airflow/DAGs/run-adf-pipeline.md similarity index 97% rename from docs/how-tos/airflow/run-adf-pipeline.md rename to docs/how-tos/airflow/DAGs/run-adf-pipeline.md index 84c74c31..1aa1fc00 100644 --- a/docs/how-tos/airflow/run-adf-pipeline.md +++ b/docs/how-tos/airflow/DAGs/run-adf-pipeline.md @@ -34,7 +34,7 @@ You can use Airflow in Datacoves to trigger a Microsoft Azure Data Factory pipel **Step 1:** In Datacoves, a user with the `securityadmin` role must go to the `Airflow Admin -> Connection` menu. - ![Airflow Connection](assets/admin-connections.png) + ![Airflow Connection](./assets/admin-connections.png) **Step 2:** Create a new connection using the following details. @@ -58,7 +58,7 @@ You can use Airflow in Datacoves to trigger a Microsoft Azure Data Factory pipel Replace the values in the screenshot below with the actual values found above. ::: -![adf connection](assets/airflow_adf_connection.png) +![adf connection](./assets/airflow_adf_connection.png) ## Example DAG diff --git a/docs/how-tos/airflow/run-airbyte-sync-jobs.md b/docs/how-tos/airflow/DAGs/run-airbyte-sync-jobs.md similarity index 98% rename from docs/how-tos/airflow/run-airbyte-sync-jobs.md rename to docs/how-tos/airflow/DAGs/run-airbyte-sync-jobs.md index 0f372046..414cf341 100644 --- a/docs/how-tos/airflow/run-airbyte-sync-jobs.md +++ b/docs/how-tos/airflow/DAGs/run-airbyte-sync-jobs.md @@ -15,7 +15,7 @@ Support for Fivetran Tasks coming soon. More Information in [run Fivetran sync j ### Ensure your Airflow environment is properly configured -Follow this guide on [How to set up Airflow](./initial-setup)'s environment. +Follow this guide on [How to set up Airflow](../initial-setup)'s environment. ### Airbyte connection diff --git a/docs/how-tos/airflow/run-databricks-notebook.md b/docs/how-tos/airflow/DAGs/run-databricks-notebook.md similarity index 96% rename from docs/how-tos/airflow/run-databricks-notebook.md rename to docs/how-tos/airflow/DAGs/run-databricks-notebook.md index d192a88f..c08f125c 100644 --- a/docs/how-tos/airflow/run-databricks-notebook.md +++ b/docs/how-tos/airflow/DAGs/run-databricks-notebook.md @@ -19,7 +19,7 @@ You can use Airflow in Datacoves to trigger a Databricks notebook. This guide wi **Step 2:** Navigate to compute. -![databricks compute](assets/databricks_compute.png) +![databricks compute](./assets/databricks_compute.png) **Step 3:** Click on your desired cluster. @@ -37,7 +37,7 @@ If you do not have admin privileges, work with an admin to get the token. Follow **Step 2:** To the right of the notebook name, there will be three dots. Click on this and select the option to copy the full path to your clipboard. -![copy url](assets/databricks_copyurl.png) +![copy url](./assets/databricks_copyurl.png) ## Handling Databricks Variables in Airflow @@ -56,7 +56,7 @@ It is possible to hardcode these two variables in your DAG if you don’t see th **Step 1:** A user with Airflow admin privileges must go to the `Airflow Admin -> Connection` menu. -![admin connection](assets/admin-connections.png) +![admin connection](./assets/admin-connections.png) **Step 2:** Create a new connection using the following details: @@ -65,7 +65,7 @@ It is possible to hardcode these two variables in your DAG if you don’t see th - **Host:** Your Databricks host. E.g. `https://.databricks.com` - **Password:** Enter your `Databricks Token` -![Databricks Connection](assets/airflow_databricks_connection.png) +![Databricks Connection](./assets/airflow_databricks_connection.png) **Step 3:** Click `Save` diff --git a/docs/how-tos/airflow/run-dbt.md b/docs/how-tos/airflow/DAGs/run-dbt.md similarity index 100% rename from docs/how-tos/airflow/run-dbt.md rename to docs/how-tos/airflow/DAGs/run-dbt.md diff --git a/docs/how-tos/airflow/run-fivetran-sync-jobs.md b/docs/how-tos/airflow/DAGs/run-fivetran-sync-jobs.md similarity index 96% rename from docs/how-tos/airflow/run-fivetran-sync-jobs.md rename to docs/how-tos/airflow/DAGs/run-fivetran-sync-jobs.md index 6f02b65e..585e0d60 100644 --- a/docs/how-tos/airflow/run-fivetran-sync-jobs.md +++ b/docs/how-tos/airflow/DAGs/run-fivetran-sync-jobs.md @@ -4,7 +4,7 @@ sidebar_position: 31 --- # Run Fivetran sync jobs -In Addition to triggering Airbyte loads jobs [run Airbyte sync jobs](/docs/how-tos/airflow/run-airbyte-sync-jobs) you can also trigger Fivetran jobs from your Airflow DAG. +In Addition to triggering Airbyte loads jobs [run Airbyte sync jobs](/docs/how-tos/airflow/DAGs/run-airbyte-sync-jobs) you can also trigger Fivetran jobs from your Airflow DAG. ## Before you start @@ -108,7 +108,7 @@ dag = daily_loan_run() ### Fields reference - **extract_and_load_fivetran**: The name of the task group. This can be named whatever you like and will show up in airflow. -![Extract and Load DAG](assets/extract_load_airflow_dag.png) +![Extract and Load DAG](./assets/extract_load_airflow_dag.png) - **tooltip**: The tooltip argument allows you to provide explanatory text or helpful hints about specific elements in the Airflow UI - **tasks**: Define all of your tasks within the task group. @@ -118,7 +118,7 @@ You will need to define two operators: `fivetran_provider.operators.fivetran.Fiv - **operator**: `fivetran_provider.operators.fivetran.FivetranOperator` - **connector_id**: Find in Fivetran UI. Select your desired source. Click into `Setup` and locate the `Fivetran Connector ID` - ![Fivetran Connection ID](assets/fivetran_connector_id.png) + ![Fivetran Connection ID](./assets/fivetran_connector_id.png) - **do_xcom_push**: Indicate that the output of the task should be sent to XCom, making it available for other tasks to use. - **fivetran_conn_id**: This is the `connection_id` that was configured above in the Fivetran UI as seen [above](#fivetran-connection). diff --git a/docs/how-tos/airflow/s3-to-snowflake.md b/docs/how-tos/airflow/DAGs/s3-to-snowflake.md similarity index 91% rename from docs/how-tos/airflow/s3-to-snowflake.md rename to docs/how-tos/airflow/DAGs/s3-to-snowflake.md index 7816685c..eb3a4308 100644 --- a/docs/how-tos/airflow/s3-to-snowflake.md +++ b/docs/how-tos/airflow/DAGs/s3-to-snowflake.md @@ -34,7 +34,7 @@ CREATE OR REPLACE STAGE TEST_STAGE This example will use SnowSQL and our data looks like this: -![Sample data](assets/s3_sample_data.jpg) +![Sample data](../assets/s3_sample_data.jpg) The SnowSQL command to upload the file to the Snowflake stage is below: ``` @@ -61,11 +61,11 @@ CREATE OR REPLACE TABLE TEST_TABLE The TEST_TABLE schema now looks like this: -![test_table](assets/s3_test_table_schema.jpg) +![test_table](../assets/s3_test_table_schema.jpg) However, the table does not have any data loaded into it. -![Empty test_table](assets/s3_test_table_empty.jpg) +![Empty test_table](../assets/s3_test_table_empty.jpg) **Step 6:** To load the data from the file we used as a template we use the following COPY INTO SQL. @@ -78,11 +78,11 @@ COPY INTO TEST_TABLE And we can now see the data in the table: -![test_table copied](assets/s3_test_table_copied.jpg) +![test_table copied](../assets/s3_test_table_copied.jpg) **Step 7:** Now we’re going to load another file into TEST_TABLE that has an additional column. -![Test data additional column](assets/s3_test_table_additional_col.jpg) +![Test data additional column](../assets/s3_test_table_additional_col.jpg) Again, we will use the SnowSQL PUT command seen below: @@ -100,7 +100,7 @@ COPY INTO TEST_TABLE And now the table has an additional column called COUNTRY_CODE: -![test_table additional column](assets/s3_test_table_additional_call_snowflake.jpg) +![test_table additional column](../assets/s3_test_table_additional_call_snowflake.jpg) ## Loading JSON data into a variant column @@ -136,5 +136,5 @@ COPY INTO VARIANT_TABLE Our variant table now looks like this: -![json variant table](assets/json_variant_table.jpg) +![json variant table](../assets/json_variant_table.jpg) diff --git a/docs/how-tos/airflow/test-dags.md b/docs/how-tos/airflow/DAGs/test-dags.md similarity index 100% rename from docs/how-tos/airflow/test-dags.md rename to docs/how-tos/airflow/DAGs/test-dags.md diff --git a/docs/how-tos/airflow/use-variables-and-connections.md b/docs/how-tos/airflow/DAGs/use-variables-and-connections.md similarity index 90% rename from docs/how-tos/airflow/use-variables-and-connections.md rename to docs/how-tos/airflow/DAGs/use-variables-and-connections.md index 7aa91950..ec3a38a0 100644 --- a/docs/how-tos/airflow/use-variables-and-connections.md +++ b/docs/how-tos/airflow/DAGs/use-variables-and-connections.md @@ -9,12 +9,12 @@ dbt-coves generate airflow-dags does not support reading variables/connections, ::: The best way to store and retrieve information within Airflow is to use `Variables` and `Connections`, both available on the `Admin` upper dropdown. -![select More](./assets/variables_connections_ui.png) +![select More](../assets/variables_connections_ui.png) The main difference between them is that [Variables](https://airflow.apache.org/docs/apache-airflow/2.3.1/howto/variable.html) is a generic multi-purpose store, while [Connections](https://airflow.apache.org/docs/apache-airflow/2.3.1/howto/connection.html) are aimed at third-party providers. :::tip -Rather than using connections or variables stored in Airflow’s database, we recommend using a Secrets Manager. These secrets are encrypted and can be stored either in [Datacoves Secrets manager](./use-datacoves-secrets-manager.mdx) or a third-party secrets manager like [AWS Secrets Manager](./use-aws-secrets-manager.mdx) +Rather than using connections or variables stored in Airflow’s database, we recommend using a Secrets Manager. These secrets are encrypted and can be stored either in [Datacoves Secrets manager](../use-datacoves-secrets-manager.mdx) or a third-party secrets manager like [AWS Secrets Manager](../use-aws-secrets-manager.mdx) ::: ## Usage @@ -24,7 +24,7 @@ Rather than using connections or variables stored in Airflow’s database, we re After creating a variable in Airflow's UI, using it is as simple as importing the `Variable` model in your DAG and `getting` it's name. If a variable contains `SECRET` on it's name, value will be hidden: -![select More](./assets/variable_creation.png) +![select More](../assets/variable_creation.png) ```python from pendulum import datetime @@ -56,7 +56,7 @@ Consuming connections data is also straightforward, though you need to take it's In the following example, a connection of `type Airbyte` is created, and it's `host` is echoed in a DAG. -![select More](./assets/connection_creation.png) +![select More](../assets/connection_creation.png) ```python from pendulum import datetime diff --git a/docs/how-tos/airflow/README.md b/docs/how-tos/airflow/README.md index 4963ec99..d9237ad2 100644 --- a/docs/how-tos/airflow/README.md +++ b/docs/how-tos/airflow/README.md @@ -1,20 +1,9 @@ --- -title: Airflow -sidebar_position: 15 +title: Airflow - What to know +sidebar_position: 1 +id: airflow-index --- -# Airflow in Datacoves - -These how to guides are dedicated to Airflow within Datacoves. Here you will find information on how to **Enable and configure Airflow** - -* Create Airflow jobs(DAGs) -* Run dbt with in an Airflow DAG -* Sending notifications -* Customizing Airflow worker environments (docker images) -* Requesting Airflow worker resources - -And more! - -## What to know +# Airflow - What to know - `Ruff` is installed to show unused imports and unused variables as well as python linting. - [Datacoves Decorators](/reference/airflow/datacoves-decorators.md) simplify working with dbt, syncing databases, and running commands in Airflow. - [My Airflow](/docs/how-tos/my_airflow/) can help speed up your DAG writing experience. diff --git a/docs/how-tos/airflow/_category_.yaml b/docs/how-tos/airflow/_category_.yaml new file mode 100644 index 00000000..e58c543e --- /dev/null +++ b/docs/how-tos/airflow/_category_.yaml @@ -0,0 +1,6 @@ +label: Airflow +position: 1 +link: + type: generated-index + title: Airflow in Datacoves + description: These how-to guides are dedicated to Airflow within Datacoves. Here you will find information on how to enable and configure Airflow. \ No newline at end of file diff --git a/docs/how-tos/airflow/_category_bk.yaml b/docs/how-tos/airflow/_category_bk.yaml deleted file mode 100644 index 3a48eaf1..00000000 --- a/docs/how-tos/airflow/_category_bk.yaml +++ /dev/null @@ -1,6 +0,0 @@ -label: Airflow -position: 1 -link: - type: generated-index - title: Airflow - description: Guides and how-tos for using Airflow within Datacoves. diff --git a/docs/how-tos/airflow/send-emails.md b/docs/how-tos/airflow/send-emails.md index 646f2086..5a01d95e 100644 --- a/docs/how-tos/airflow/send-emails.md +++ b/docs/how-tos/airflow/send-emails.md @@ -45,7 +45,7 @@ In Datacoves 3.3 and up, the `SMTP` will be automatically added to your environm Viola!🎉 The Airflow service will be restarted shortly and will now include the SMTP configuration required to send emails. -:::note **Getting Started Guide:** If you are making your way through our [getting started guide](../../getting-started/Admin/), please continue on to [developing DAGs](getting-started/Admin/creating-airflow-dags.md). +:::note **Getting Started Guide:** If you are making your way through our [getting started guide](/docs/category/administrator), please continue on to [developing DAGs](getting-started/Admin/creating-airflow-dags.md). ::: ## Set up a custom SMTP (Optional) diff --git a/docs/how-tos/datacoves/README.md b/docs/how-tos/datacoves/README.txt similarity index 100% rename from docs/how-tos/datacoves/README.md rename to docs/how-tos/datacoves/README.txt diff --git a/docs/how-tos/datacoves/_category_.yaml b/docs/how-tos/datacoves/_category_.yaml new file mode 100644 index 00000000..f4c589e2 --- /dev/null +++ b/docs/how-tos/datacoves/_category_.yaml @@ -0,0 +1,6 @@ +label: Datacoves +position: 2 +link: + type: generated-index + title: Datacoves + description: These how to guides are dedicated to **Datacoves admin configurations**. \ No newline at end of file diff --git a/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md b/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md index 69415485..26184345 100644 --- a/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md +++ b/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md @@ -86,7 +86,7 @@ Please, follow the [AWS Secrets Manager documentation](https://docs.aws.amazon.c ![Secrets Backend](../assets/aws_secrets_connection.jpg) -To learn how to read a variable from the AWS Secrets Manager check out our [How To](../README.md) +To learn how to read a variable from the AWS Secrets Manager check out our [How To](/docs/category/how-tos/) :::tip For security purposes, once this has been saved you will not be able to view the values. To modify the Secrets backend you will need to set the Secrets backend to `None` and save the changes. Then start the setup again. diff --git a/docs/how-tos/datahub/README.md b/docs/how-tos/datahub/README.txt similarity index 100% rename from docs/how-tos/datahub/README.md rename to docs/how-tos/datahub/README.txt diff --git a/docs/how-tos/datahub/_category_.yaml b/docs/how-tos/datahub/_category_.yaml new file mode 100644 index 00000000..0f3a00e6 --- /dev/null +++ b/docs/how-tos/datahub/_category_.yaml @@ -0,0 +1,7 @@ + +label: Datahub +position: 3 +link: + type: generated-index + title: Datahub + description: These how to guides are dedicated to DataHub in Datacoves. \ No newline at end of file diff --git a/docs/how-tos/dataops/README.md b/docs/how-tos/dataops/README.md index cbb72044..32a6723a 100644 --- a/docs/how-tos/dataops/README.md +++ b/docs/how-tos/dataops/README.md @@ -1,6 +1,6 @@ --- title: DataOps -sidebar_position: 62 +sidebar_position: 4 --- # DataOps in Datacoves diff --git a/docs/how-tos/dbt/README.md b/docs/how-tos/dbt/README.txt similarity index 100% rename from docs/how-tos/dbt/README.md rename to docs/how-tos/dbt/README.txt diff --git a/docs/how-tos/dbt/_category_.yaml b/docs/how-tos/dbt/_category_.yaml new file mode 100644 index 00000000..416e5d7e --- /dev/null +++ b/docs/how-tos/dbt/_category_.yaml @@ -0,0 +1,7 @@ + +label: dbt +position: 5 +link: + type: generated-index + title: dbt + description: These how to guides cover how to resolve common dbt issues. \ No newline at end of file diff --git a/docs/how-tos/git/README.md b/docs/how-tos/git/README.md index 55e13a8d..13ddfaa3 100644 --- a/docs/how-tos/git/README.md +++ b/docs/how-tos/git/README.md @@ -1,6 +1,6 @@ --- title: Git -sidebar_position: 70 +sidebar_position: 6 --- # How to use Git diff --git a/docs/how-tos/metrics-and-logs/README.md b/docs/how-tos/metrics-and-logs/README.md index 58044d85..4837dd87 100644 --- a/docs/how-tos/metrics-and-logs/README.md +++ b/docs/how-tos/metrics-and-logs/README.md @@ -1,10 +1,10 @@ --- title: Metrics & Logs -sidebar_position: 71 +sidebar_position: 7 --- # Metrics and Logs How Tos -Datacoves provides [Grafana](/reference/metrics-and-logs/grafana.md) to monitor Airflow, Docker image builds, and more! +Datacoves provides [Grafana](/reference/datacoves/metrics-and-logs/grafana.md) to monitor Airflow, Docker image builds, and more! A user must have a Datacoves role with Grafana access. These include, `Datacoves Admin`, `Project Admin`, or `Environment Admin`. diff --git a/docs/how-tos/my_airflow/README.mdx b/docs/how-tos/my_airflow/README.mdx index 93b96a66..3c9692ef 100644 --- a/docs/how-tos/my_airflow/README.mdx +++ b/docs/how-tos/my_airflow/README.mdx @@ -1,6 +1,6 @@ --- title: My Airflow -sidebar_position: 72 +sidebar_position: 8 --- # My Airflow 101 diff --git a/docs/how-tos/snowflake/README.md b/docs/how-tos/snowflake/README.txt similarity index 100% rename from docs/how-tos/snowflake/README.md rename to docs/how-tos/snowflake/README.txt diff --git a/docs/how-tos/snowflake/_category_.yaml b/docs/how-tos/snowflake/_category_.yaml new file mode 100644 index 00000000..f6e495b3 --- /dev/null +++ b/docs/how-tos/snowflake/_category_.yaml @@ -0,0 +1,7 @@ + +label: Snowflake +position: 9 +link: + type: generated-index + title: snowflake + description: These how to guides are dedicated to Snowflake. Learn how to set up different aspects of Snowflake including a proper design of roles and permissions. \ No newline at end of file diff --git a/docs/how-tos/superset/README.md b/docs/how-tos/superset/README.txt similarity index 100% rename from docs/how-tos/superset/README.md rename to docs/how-tos/superset/README.txt diff --git a/docs/how-tos/superset/_category_.yaml b/docs/how-tos/superset/_category_.yaml new file mode 100644 index 00000000..9315c7ba --- /dev/null +++ b/docs/how-tos/superset/_category_.yaml @@ -0,0 +1,7 @@ + +label: Superset +position: 10 +link: + type: generated-index + title: Superset + description: These how to guides are dedicated to Superset in Datacoves. \ No newline at end of file diff --git a/docs/how-tos/vscode/README.md b/docs/how-tos/vscode/README.txt similarity index 100% rename from docs/how-tos/vscode/README.md rename to docs/how-tos/vscode/README.txt diff --git a/docs/how-tos/vscode/_category_.yaml b/docs/how-tos/vscode/_category_.yaml new file mode 100644 index 00000000..e961ac58 --- /dev/null +++ b/docs/how-tos/vscode/_category_.yaml @@ -0,0 +1,6 @@ +label: vscode +position: 50 +link: + type: generated-index + title: VS Code in Datacoves + description: These how to guides are dedicated to our Hosted VS Code. This powerful IDE gives you all the power you need along with the customization options. These steps are essential to our Transform Tab. \ No newline at end of file diff --git a/docs/how-tos/vscode/datacoves-copilot/llm.mdx b/docs/how-tos/vscode/datacoves-copilot/llm.mdx deleted file mode 100644 index 5637077c..00000000 --- a/docs/how-tos/vscode/datacoves-copilot/llm.mdx +++ /dev/null @@ -1,78 +0,0 @@ ---- -sidebar_position: 1 ---- -# Working with an LLM in Datacoves Copilot - -Once you have [configured your LLM](./v2-llm-config.md), working with the AI chat is a breeze! - -## Pre-configured presets - -While there are many use cases for AI in the development workflow there are a couple of tasks that come up regularly. To improve the user experience, we have created some presets that provide the LLM with specific instructions to get the best result. Use these or configure your own custom presets. - -Current out of the box presents include: - -- **Document dbt model:** Will provide descriptions for yml file you have open and provided as context. -- **Explain dbt model:** Will explain a dbt model file you have open and provided as context. -- **Explain dbt macro:** Will explain a dbt macro file you have open and provided as context. -- **Add descriptions to CSV:** Will provide descriptions for CSV files you have open and provided as context. - -## How to use the Chat/Presets - -### Extension UI/UX - -To use your LLM head to the Datacoves Power User extension and open up `Copilot`. - -LLM Menu - -### Provide files for context -The most important thing to remember is providing context to the LLM. In order to do this simply open up the file you wish to work with. - -![llm file open](assets/llm_file_open.png) - -### Send specific lines of code for context -Highlight the lines of code you wish to send. - -![context lines](assets/llm_line_context.png) - -### Don't provide context -In the case that you don't want to send an open file for context you can select the icon below. - -![llm context](assets/llm_context.gif) - -## Using LLM code -Datacoves Copilot makes copying AI generated code simple with 3 options. - -![Code Options](assets/llm_code_options.png) - -### Replace -Replace the entire code file with the LLM code. - -![llm Replace](assets/llm_replace.gif) - -### Insert -Insert LLM code into your file. - -![llm Insert](assets/llm_insert.gif) - -### Copy -Traditional copy and paste. - -![llm Copy](assets/llm_copy.gif) - -### New Chat -To start a new chat click on the `New Chat` button. - -![New Chat](assets/llm_new_chat.png) - -## Prefer a Video? - -
-