Skip to content

Commit d0f552e

Browse files
committed
Integrate/PostgreSQL: Add section with starter tutorial
The goal is to present concise walkthroughs without many bells and whistles, which get to the point of getting you started quickly. Use the canonical template to present another data nozzle based on the CTK Ingestr I/O subsystem.
1 parent f59600a commit d0f552e

File tree

3 files changed

+135
-0
lines changed

3 files changed

+135
-0
lines changed

docs/integrate/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ n8n/index
5151
nifi/index
5252
node-red/index
5353
plotly/index
54+
postgresql/index
5455
Power BI <powerbi/index>
5556
prometheus/index
5657
pyviz/index

docs/integrate/postgresql/index.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
(postgresql)=
2+
# PostgreSQL
3+
4+
```{div} .float-right
5+
[![postgresql-logo](https://www.postgresql.org/media/img/about/press/elephant.png){height=60px loading=lazy}][PostgreSQL]
6+
```
7+
```{div} .clearfix
8+
```
9+
10+
:::{rubric} About
11+
:::
12+
13+
[PostgreSQL] is the world's most advanced open source relational database.
14+
15+
:::{rubric} Synopsis
16+
:::
17+
18+
```shell
19+
uvx 'cratedb-toolkit[io-ingestr]' load table \
20+
"postgresql://<username>:<password>@host:port/dbname?table=demo" \
21+
--cluster-url="crate://crate:crate@localhost:4200/testdrive/postgresql_demo"
22+
```
23+
24+
:::{rubric} Learn
25+
:::
26+
27+
::::{grid}
28+
29+
:::{grid-item-card} Tutorial: Use CrateDB Toolkit
30+
:link: postgresql-tutorial
31+
:link-type: ref
32+
Load data from PostgreSQL into CrateDB using CrateDB Toolkit.
33+
:::
34+
35+
::::
36+
37+
38+
:::{toctree}
39+
:maxdepth: 1
40+
:hidden:
41+
Tutorial <tutorial>
42+
:::
43+
44+
45+
[PostgreSQL]: https://www.postgresql.org/
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
(postgresql-tutorial)=
2+
# Load data from PostgreSQL into CrateDB
3+
4+
The tutorial will walk you through starting [PostgreSQL] and CrateDB,
5+
inserting a record into PostgreSQL, loading data into a CrateDB table,
6+
and validating that the data has been stored successfully.
7+
The data transfer is supported by the
8+
{ref}`CrateDB Toolkit Ingestr I/O <ctk:ingestr>` data pipeline elements.
9+
10+
## Prerequisites
11+
12+
Docker is used for running all components. This approach works consistently
13+
across Linux, macOS, and Windows. Alternatively, you can use Podman.
14+
15+
Create a shared network.
16+
```shell
17+
docker network create cratedb-demo
18+
```
19+
20+
Start CrateDB.
21+
```shell
22+
docker run --rm --name=cratedb --network=cratedb-demo \
23+
--publish=4200:4200 --publish=5432:5432 \
24+
--env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node
25+
```
26+
27+
Start PostgreSQL.
28+
```shell
29+
docker run --rm --name=postgresql --network=cratedb-demo \
30+
--publish=5433:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \
31+
docker.io/postgres postgres -c log_statement=all
32+
```
33+
34+
Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client
35+
programs.
36+
37+
::::{tab-set}
38+
39+
:::{tab-item} Linux and macOS
40+
To make the settings persistent, add them to your shell profile (`~/.profile`).
41+
```shell
42+
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
43+
alias ctk="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit ctk"
44+
alias psql="docker run --rm --network=cratedb-demo docker.io/postgres psql"
45+
```
46+
:::
47+
:::{tab-item} Windows PowerShell
48+
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
49+
```powershell
50+
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
51+
function ctk { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit ctk @args }
52+
function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args }
53+
```
54+
:::
55+
:::{tab-item} Windows Command
56+
```shell
57+
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
58+
doskey ctk=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit ctk $*
59+
doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $*
60+
```
61+
:::
62+
63+
::::
64+
65+
## Usage
66+
67+
Write a few sample records to PostgreSQL.
68+
```shell
69+
psql "postgresql://postgres:postgres@postgresql:5432/" <<SQL
70+
CREATE TABLE IF NOT EXISTS demo (id BIGINT, data JSONB);
71+
INSERT INTO demo (id, data) VALUES (1, '{"temperature": 42.84, "humidity": 83.1}');
72+
INSERT INTO demo (id, data) VALUES (2, '{"temperature": 84.84, "humidity": 56.99}');
73+
SQL
74+
```
75+
76+
Invoke the data transfer pipeline.
77+
```shell
78+
ctk load table \
79+
"postgresql://postgres:postgres@postgresql/public?table=demo" \
80+
--cluster-url="crate://crate:crate@cratedb:4200/testdrive/postgresql_demo"
81+
```
82+
83+
Inspect data stored in CrateDB.
84+
```shell
85+
crash --hosts cratedb -c "SELECT * FROM testdrive.postgresql_demo"
86+
```
87+
88+
89+
[PostgreSQL]: https://www.postgresql.org/

0 commit comments

Comments
 (0)