Skip to content

Conversation

dvdotsenko
Copy link

  • separated destination DB from container that connects and loads data
  • separated Ruby runner container from DB image. Now all images are separete and pure as they come from Docker Hub
  • reduced whole download, convert, upload process into 2 make commands

Goals:

  1. Intentionally separate destination DB instance from all the Ruby-based conversion (and runtime) code. Intentionally separated psql runner from destination DB instance (now psql runs in a separate container and quits). This is to allow destination DB to be any instance of Postgres. Just feed credentials to this script and send data to destination Postgres instance.
  2. Use pure vendor images for everything: Ruby, Postgres. Makes things run faster (no image build step) and allows easy swap of versions of Ruby and Postgres runtimes.
  3. I tend to use this a lot now to bootstrap a demo server. Needed full automation and as few commands as possible. Also needed reentrant run (where if AdventureWorks.zip is already downloaded, script automatically just goes to next step. This allowed air-gapped use on internet-limited machines.

- separated destination DB from container that connects and loads data
- separated Ruby runner container from DB image. Now all images are separete and pure as they come from Docker Hub
- reduced whole process to 2 `make` commands
Names are too short to be intuitive. They don't join or transform anything - just copy of table from another schema. They pollute schema definitions in data lineage tools
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant