Studies with Airflow

created a docker-compose file with services: Airflow, PostgreSQL, MinIO.
Using Admin -> Connections -> Add Connection (Here I can use to create a connection between Airflow and other services [PostgreSQL, MinIO (AWS Service), Databricks])

Folders:

dags/airflow_connect_db

Dag will create a table in Postgres and insert data, all the sql files will be stored in: sql/ directory.

dags/tutorial_load_data/

1. test_minio_dag

Dag will test the connection from Airflow to MinIO, as the button test was failing in the Airflow API.

2. ny_covid_data

Dag will read json file from a url, upload the file in MinIO (but, it will check if the file is already in the bucket) and execute some validation with the data.

dags/first_sensor/

1. dag_s3
   
   Dag will check if a file arrived inside the bucket.