Python webscraper that uses Selenium, Redis, and Postgres to queue cards to scrape and store those prices from the website TCG player.
This project is meant to be run using docker and the docker-compose.yml file but it is possibel to build and run locally which may require changing the connections inside of db.py and cardQueue.py.
Prereqs: Docker Installed
Step 1: Build the Docker contawiners
docker compose build
Step 2: Run Docker containers
docker compose up -d
Step 3: Verify Docker contains are running
docker ps
You should see that there are 3 worker containers along with a Redis and Postgres container.
This project contains a very simple CLI to add cards to the queue and see the size of the queue. To use the CLI run
docker compose exec main python main.py [CLI ARGS]
The possible arguments are
- -set "Set Name" Adds entire set to queue
- -format "Format Name" Adds entire format to queue
- -info Prints size of the queue to terminal
Additionally workers can be scaled in docker by running
docker compose up -d --scale worker=[NUMBER OF WORKERS]