Skip to content

Commit a2d054f

Browse files
committed
Integrate/PostgreSQL: Add section with starter tutorial
The goal is to present concise walkthroughs without many bells and whistles, which get to the point of getting you started quickly. Use the canonical template to present another data nozzle based on the CTK Ingestr I/O subsystem.
1 parent bb54283 commit a2d054f

File tree

4 files changed

+146
-0
lines changed

4 files changed

+146
-0
lines changed

docs/ingest/etl/index.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,10 @@ Use visual data flow and integration frameworks and platforms.
121121
MySQL and MariaDB are well-known free and open-source relational database management
122122
systems (RDBMS), available as standalone and managed variants.
123123

124+
- {ref}`postgresql`
125+
126+
PostgreSQL is the world's most advanced open source relational database.
127+
124128
- {ref}`sql-server`
125129

126130
Microsoft SQL Server Integration Services (SSIS) is a component of the Microsoft SQL
@@ -237,6 +241,7 @@ Load data from datasets and open table formats.
237241
- {ref}`n8n`
238242
- {ref}`nifi`
239243
- {ref}`node-red`
244+
- {ref}`postgresql`
240245
- {ref}`risingwave`
241246
- {ref}`sql-server`
242247
- {ref}`streamsets`

docs/integrate/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ n8n/index
5252
nifi/index
5353
node-red/index
5454
plotly/index
55+
postgresql/index
5556
Power BI <powerbi/index>
5657
prometheus/index
5758
pyviz/index

docs/integrate/postgresql/index.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
(postgresql)=
2+
# PostgreSQL
3+
4+
```{div} .float-right
5+
[![postgresql-logo](https://www.postgresql.org/media/img/about/press/elephant.png){height=60px loading=lazy}][PostgreSQL]
6+
```
7+
```{div} .clearfix
8+
```
9+
10+
:::{rubric} About
11+
:::
12+
13+
[PostgreSQL] is the world's most advanced open source relational database.
14+
15+
:::{rubric} Synopsis
16+
:::
17+
18+
```shell
19+
uvx 'cratedb-toolkit[io-ingestr]' load table \
20+
"postgresql://postgres:postgres@localhost:5432/test?table=public.demo" \
21+
--cluster-url="crate://crate:crate@localhost:4200/doc/postgresql_demo"
22+
```
23+
24+
:::{rubric} Learn
25+
:::
26+
27+
::::{grid}
28+
29+
:::{grid-item-card} Tutorial: Use CrateDB Toolkit
30+
:link: postgresql-tutorial
31+
:link-type: ref
32+
Load data from PostgreSQL into CrateDB using CrateDB Toolkit.
33+
:::
34+
35+
::::
36+
37+
38+
:::{toctree}
39+
:maxdepth: 1
40+
:hidden:
41+
Tutorial <tutorial>
42+
:::
43+
44+
45+
[PostgreSQL]: https://www.postgresql.org/
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
(postgresql-tutorial)=
2+
# Load data from PostgreSQL into CrateDB
3+
4+
The tutorial will walk you through starting [PostgreSQL] and CrateDB,
5+
inserting a record into PostgreSQL, loading data into a CrateDB table,
6+
and validating that the data has been stored successfully.
7+
The data transfer is supported by the
8+
{ref}`CrateDB Toolkit Ingestr I/O <ctk:ingestr>` data pipeline elements.
9+
10+
## Prerequisites
11+
12+
Docker is used for running all components. This approach works consistently
13+
across Linux, macOS, and Windows. Alternatively, you can use Podman.
14+
15+
Create a shared network.
16+
```shell
17+
docker network create cratedb-demo
18+
```
19+
20+
Start CrateDB.
21+
```shell
22+
docker run --rm --name=cratedb --network=cratedb-demo \
23+
--publish=4200:4200 --publish=5432:5432 --env=CRATE_HEAP_SIZE=2g \
24+
docker.io/crate -Cdiscovery.type=single-node
25+
```
26+
27+
Start PostgreSQL.
28+
```shell
29+
docker run --rm --name=postgresql --network=cratedb-demo \
30+
--publish=6432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \
31+
docker.io/postgres postgres -c log_statement=all
32+
```
33+
:::{note}
34+
Because CrateDB is configured to listen on port `5432` with its PostgreSQL
35+
interface, let's use a different port for PostgreSQL itself.
36+
:::
37+
38+
Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client
39+
programs.
40+
41+
::::{tab-set}
42+
43+
:::{tab-item} Linux and macOS
44+
To make the settings persistent, add them to your shell profile (`~/.profile`).
45+
```shell
46+
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
47+
alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk"
48+
alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql"
49+
```
50+
:::
51+
:::{tab-item} Windows PowerShell
52+
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
53+
```powershell
54+
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
55+
function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args }
56+
function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args }
57+
```
58+
:::
59+
:::{tab-item} Windows Command
60+
```shell
61+
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
62+
doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $*
63+
doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $*
64+
```
65+
:::
66+
67+
::::
68+
69+
## Usage
70+
71+
Write a few sample records to PostgreSQL.
72+
```shell
73+
psql "postgresql://postgres:postgres@postgresql:5432/" <<SQL
74+
CREATE DATABASE test;
75+
\connect test;
76+
CREATE TABLE IF NOT EXISTS demo (id BIGINT, data JSONB);
77+
INSERT INTO demo (id, data) VALUES (1, '{"temperature": 42.84, "humidity": 83.1}');
78+
INSERT INTO demo (id, data) VALUES (2, '{"temperature": 84.84, "humidity": 56.99}');
79+
SQL
80+
```
81+
82+
Invoke the data transfer pipeline.
83+
```shell
84+
ctk-ingest load table \
85+
"postgresql://postgres:postgres@postgresql:5432/test?table=public.demo" \
86+
--cluster-url="crate://crate:crate@cratedb:4200/doc/postgresql_demo"
87+
```
88+
89+
Inspect data stored in CrateDB.
90+
```shell
91+
crash --hosts cratedb -c "SELECT * FROM doc.postgresql_demo"
92+
```
93+
94+
95+
[PostgreSQL]: https://www.postgresql.org/

0 commit comments

Comments
 (0)