Shapeshift

A scalable, distributed system for extracting and transforming LibreTexts content into various export formats.

Development

Shapeshift services are split into two containers, shapeshift-api and shapeshift-processor, which use MySQL for job data storage.

Ephermeral MySQL

If you want to start an ephermeral MySQL container for development, you can do so with:

# Start
docker compose -f docker-compose-mysql.dev.yml up -d
# Stop
docker compose -f docker-compose-mysql.dev.yml down

Connecting to Locally Installed Resources

If you have an existing local MySQL or LocalStack installation running on your machine, you can connect to them by enabling host networking mode. Add this config to each service in docker-compose.dev.yaml:

network_mode: "host"

This will allow your Shapeshift containers to connect to the MySQL/LocalStack running on your device using localhost as the host name.

The Main Stack

After MySQL is set up, start/stop the stack with:

# Start
docker compose -f docker-compose.dev.yaml up -d
# Stop
docker compose -f docker-compose.dev.yaml down

Run Development Build

You can use the run-dev-build.sh script to build and start the API and Processor containers with your local changes. The first build may take a few moments; subsequent builds will be faster.

./run-dev-build.sh

Use Control-C to stop the containers.

AWS Emulation

LocalStack can be used to emulate AWS services (like SQS and Secrets Manager) locally as a Docker container, allowing you to test the entire job workflow end-to-end. Create a free account and install the CLI to get started. Set the AWS_REGION, LOCALSTACK_HOST, and LOCALSTACK_PORT environment variables appropriately. Test data is stored on your machine unless the Cloud Pods feature is used.

Running a Test Request

curl --request POST \
  --url http://localhost:80/api/v1/job \
  --header 'content-type: application/json' \
  --data '{"url":"https://dev.libretexts.org/Sandboxes/eaturner_at_ucdavis.edu/Test_Book","highPriority":false}'

License

MIT