Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/ingest/etl/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,10 @@ Use visual data flow and integration frameworks and platforms.
MySQL and MariaDB are well-known free and open-source relational database management
systems (RDBMS), available as standalone and managed variants.

- {ref}`postgresql`

PostgreSQL is the world's most advanced open source relational database.

- {ref}`sql-server`

Microsoft SQL Server Integration Services (SSIS) is a component of the Microsoft SQL
Expand Down Expand Up @@ -237,6 +241,7 @@ Load data from datasets and open table formats.
- {ref}`n8n`
- {ref}`nifi`
- {ref}`node-red`
- {ref}`postgresql`
- {ref}`risingwave`
- {ref}`sql-server`
- {ref}`streamsets`
Expand Down
1 change: 1 addition & 0 deletions docs/integrate/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ n8n/index
nifi/index
node-red/index
plotly/index
postgresql/index
Power BI <powerbi/index>
prometheus/index
pyviz/index
Expand Down
45 changes: 45 additions & 0 deletions docs/integrate/postgresql/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
(postgresql)=
# PostgreSQL

```{div} .float-right
[![postgresql-logo](https://www.postgresql.org/media/img/about/press/elephant.png){height=60px loading=lazy}][PostgreSQL]
```
```{div} .clearfix
```

:::{rubric} About
:::

[PostgreSQL] is the world's most advanced open source relational database.

:::{rubric} Synopsis
:::

```shell
uvx 'cratedb-toolkit[io-ingestr]' load table \
"postgresql://postgres:postgres@localhost:5432/test?table=public.demo" \
--cluster-url="crate://crate:crate@localhost:4200/doc/postgresql_demo"
```

:::{rubric} Learn
:::

::::{grid}

:::{grid-item-card} Tutorial: Use CrateDB Toolkit
:link: postgresql-tutorial
:link-type: ref
Load data from PostgreSQL into CrateDB using CrateDB Toolkit.
:::

::::


:::{toctree}
:maxdepth: 1
:hidden:
Tutorial <tutorial>
:::


[PostgreSQL]: https://www.postgresql.org/
99 changes: 99 additions & 0 deletions docs/integrate/postgresql/tutorial.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
(postgresql-tutorial)=
# Load data from PostgreSQL into CrateDB

The tutorial will walk you through starting [PostgreSQL] and CrateDB,
inserting a record into PostgreSQL, loading data into a CrateDB table,
and validating that the data has been stored successfully.
The data transfer is supported by the
{ref}`CrateDB Toolkit Ingestr I/O <ctk:ingestr>` data pipeline elements.

## Prerequisites

Docker is used for running all components. This approach works consistently
across Linux, macOS, and Windows. Alternatively, you can use Podman.

Create a shared network.
```shell
docker network create cratedb-demo
```

Start CrateDB.
```shell
docker run --rm --name=cratedb --network=cratedb-demo \
--publish=4200:4200 --publish=5432:5432 \
docker.io/crate -Cdiscovery.type=single-node
```

Start PostgreSQL.
```shell
docker run --rm --name=postgresql --network=cratedb-demo \
--publish=6432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \
docker.io/postgres postgres -c log_statement=all
```
:::{note}
Because CrateDB is configured to listen on port `5432` with its PostgreSQL
interface, let's use a different port for PostgreSQL itself.
:::
:::{note}
Using `POSTGRES_HOST_AUTH_METHOD=trust` disables password checks.
Use it for local demos only.
:::

Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client
programs.

::::{tab-set}

:::{tab-item} Linux and macOS
To make the settings persistent, add them to your shell profile (`~/.profile`).
```shell
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk"
alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql"
```
:::
:::{tab-item} Windows PowerShell
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
```powershell
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args }
function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args }
```
:::
:::{tab-item} Windows Command
```shell
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $*
doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $*
```
:::

::::

## Usage

Write a few sample records to PostgreSQL.
```shell
psql "postgresql://postgres:postgres@postgresql:5432/" <<SQL
CREATE DATABASE test;
\connect test;
CREATE TABLE IF NOT EXISTS demo (id BIGINT, data JSONB);
INSERT INTO demo (id, data) VALUES (1, '{"temperature": 42.84, "humidity": 83.1}');
INSERT INTO demo (id, data) VALUES (2, '{"temperature": 84.84, "humidity": 56.99}');
SQL
```

Invoke the data transfer pipeline.
```shell
ctk-ingest load table \
"postgresql://postgres:postgres@postgresql:5432/test?table=public.demo" \
--cluster-url="crate://crate:crate@cratedb:4200/doc/postgresql_demo"
```

Inspect data stored in CrateDB.
```shell
crash --hosts cratedb -c "SELECT * FROM doc.postgresql_demo"
```


[PostgreSQL]: https://www.postgresql.org/