|
| 1 | +(postgresql-tutorial)= |
| 2 | +# Load data from PostgreSQL into CrateDB |
| 3 | + |
| 4 | +The tutorial will walk you through starting [PostgreSQL] and CrateDB, |
| 5 | +inserting a record into PostgreSQL, loading data into a CrateDB table, |
| 6 | +and validating that the data has been stored successfully. |
| 7 | +The data transfer is supported by the |
| 8 | +{ref}`CrateDB Toolkit Ingestr I/O <ctk:ingestr>` data pipeline elements. |
| 9 | + |
| 10 | +## Prerequisites |
| 11 | + |
| 12 | +Docker is used for running all components. This approach works consistently |
| 13 | +across Linux, macOS, and Windows. Alternatively, you can use Podman. |
| 14 | + |
| 15 | +Create a shared network. |
| 16 | +```shell |
| 17 | +docker network create cratedb-demo |
| 18 | +``` |
| 19 | + |
| 20 | +Start CrateDB. |
| 21 | +```shell |
| 22 | +docker run --rm --name=cratedb --network=cratedb-demo \ |
| 23 | + --publish=4200:4200 --publish=5432:5432 --env=CRATE_HEAP_SIZE=2g \ |
| 24 | + docker.io/crate -Cdiscovery.type=single-node |
| 25 | +``` |
| 26 | + |
| 27 | +Start PostgreSQL. |
| 28 | +```shell |
| 29 | +docker run --rm --name=postgresql --network=cratedb-demo \ |
| 30 | + --publish=6432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \ |
| 31 | + docker.io/postgres postgres -c log_statement=all |
| 32 | +``` |
| 33 | +:::{note} |
| 34 | +Because CrateDB is configured to listen on port `5432` with its PostgreSQL |
| 35 | +interface, let's use a different port for PostgreSQL itself. |
| 36 | +::: |
| 37 | + |
| 38 | +Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client |
| 39 | +programs. |
| 40 | + |
| 41 | +::::{tab-set} |
| 42 | + |
| 43 | +:::{tab-item} Linux and macOS |
| 44 | +To make the settings persistent, add them to your shell profile (`~/.profile`). |
| 45 | +```shell |
| 46 | +alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" |
| 47 | +alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk" |
| 48 | +alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql" |
| 49 | +``` |
| 50 | +::: |
| 51 | +:::{tab-item} Windows PowerShell |
| 52 | +To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). |
| 53 | +```powershell |
| 54 | +function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } |
| 55 | +function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args } |
| 56 | +function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args } |
| 57 | +``` |
| 58 | +::: |
| 59 | +:::{tab-item} Windows Command |
| 60 | +```shell |
| 61 | +doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* |
| 62 | +doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $* |
| 63 | +doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $* |
| 64 | +``` |
| 65 | +::: |
| 66 | + |
| 67 | +:::: |
| 68 | + |
| 69 | +## Usage |
| 70 | + |
| 71 | +Write a few sample records to PostgreSQL. |
| 72 | +```shell |
| 73 | +psql "postgresql://postgres:postgres@postgresql:5432/" <<SQL |
| 74 | +CREATE DATABASE test; |
| 75 | +\connect test; |
| 76 | +CREATE TABLE IF NOT EXISTS demo (id BIGINT, data JSONB); |
| 77 | +INSERT INTO demo (id, data) VALUES (1, '{"temperature": 42.84, "humidity": 83.1}'); |
| 78 | +INSERT INTO demo (id, data) VALUES (2, '{"temperature": 84.84, "humidity": 56.99}'); |
| 79 | +SQL |
| 80 | +``` |
| 81 | + |
| 82 | +Invoke the data transfer pipeline. |
| 83 | +```shell |
| 84 | +ctk-ingest load table \ |
| 85 | + "postgresql://postgres:postgres@postgresql:5432/test?table=public.demo" \ |
| 86 | + --cluster-url="crate://crate:crate@cratedb:4200/doc/postgresql_demo" |
| 87 | +``` |
| 88 | + |
| 89 | +Inspect data stored in CrateDB. |
| 90 | +```shell |
| 91 | +crash --hosts cratedb -c "SELECT * FROM doc.postgresql_demo" |
| 92 | +``` |
| 93 | + |
| 94 | + |
| 95 | +[PostgreSQL]: https://www.postgresql.org/ |
0 commit comments