Skip to content

ken-10/poketcgdata-workflows

Repository files navigation

PokeTCGData Workflows

Automated system to ingest daily Pokemon TCG pricing data from Pokemon TCG IO API.
The data is displayed in a live dashboard that can be found here. (Coming soon)

Note

The way this project is deployed/architected may not make sense due to budgeting and to better learn the applications being used.

Architecture

Architecture Diagram

Tools

Apache Airflow dbt Terraform Docker

Database/Cloud Hosting

Azure SQL AWS

Setup

This does require a Azure SQL Database or Microsoft SQL Server instance to store data. There is usually a free tier for Azure SQL Database.

If you want to run this 100% locally, skip the steps related to terraform and AWS CLI. Also, set up a Microsoft SQL Server 2022 Linux container with Docker locally on your own as the database.

Caution

Although there are free tiers for Azure SQL and AWS EC2 instances, please be aware that costs may incur with these services.
I am not responsible for any costs that occur.

Prerequisites

  • To run on local/development, install the following on your local machine:
  • To deploy on AWS via Terraform, install the following on your local machine:
    • Install **Terraform CLI **
    • Create an AWS account
    • Install and configure AWS CLI

How to start Airflow locally

There is a powershell script setup.sh that contains scripted functions to bring containers up, down, and restart them (locally).

👆 To bring up the environment, run the following command:

sh setup.sh start

👇 To bring down the environment, run the following command:

sh setup.sh down

🔄 To restart the environment, run the following command:

sh setup.sh restart

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors