Description

This projects querries images from the Pexels website through there REST API. The images are then sent a the Kafka broker to then be downloaded to a data volume. All of this project is orchestrated by docker composed to make its deployment fast and easy.

Set Up

1. Get an API key from Pexels and create a Config.py with the following values : ' API_KEY = [INSERT API KEY HERE] '

2. Launche following command with the docker deamon running : 'docker compose up'

Docker Diagrame

[Pexels REST API]->[DataPusher(Python)]->[Kafka]->[DataPuller(Python)]->[Data(volume)]

Environment Variable

Datapusher

- 'DATAPUSHER_KAFKA_TOPIC': Tells DataPusher what topic to use when pusing data to kafka
- 'DATAPUSHER_KAFKA_SERVER': Tells DataPusher the IP and port number to access Kafka

Datapuller

- 'DATAPULLER_KAFKA_TOPIC': Tells DataPuller what topic to use when pulling data from kafka
- 'DATAPULLER_KAFKA_SERVER': Tells DataPuller the IP and port number to access Kafka
- 'DATAPULLER_KAFKA_GROUP': Tells DataPuller what group to join when consuming Kafka data

Documentation

All documentation can be found in docs/index.html