From ba9d94c81d317ac6d6a2822dc97b08f9bf8e70db Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Wed, 13 Aug 2025 03:44:41 +0200 Subject: [PATCH 1/3] Integrate/AMQP: Add entry point page, micro tutorial, and category item --- docs/ingest/etl/index.md | 8 +++ docs/integrate/amqp/index.md | 53 ++++++++++++++++++ docs/integrate/amqp/tutorial.md | 96 +++++++++++++++++++++++++++++++++ docs/integrate/index.md | 1 + 4 files changed, 158 insertions(+) create mode 100644 docs/integrate/amqp/index.md create mode 100644 docs/integrate/amqp/tutorial.md diff --git a/docs/ingest/etl/index.md b/docs/ingest/etl/index.md index 08fd4486..3af15483 100644 --- a/docs/ingest/etl/index.md +++ b/docs/ingest/etl/index.md @@ -132,6 +132,14 @@ Load data from database systems. ::::{grid-item-card} {material-outlined}`fast_forward;2em` Streams + +- {ref}`amqp` + + The AMQP protocol is an open standard application layer protocol for + message-oriented middleware. The defining features of AMQP are message + orientation, queuing, routing (including point-to-point and + publish-and-subscribe), reliability, and security. + - {ref}`kafka` Apache Kafka is an open-source distributed event streaming platform diff --git a/docs/integrate/amqp/index.md b/docs/integrate/amqp/index.md new file mode 100644 index 00000000..88d88373 --- /dev/null +++ b/docs/integrate/amqp/index.md @@ -0,0 +1,53 @@ +(amqp)= +# AMQP + +```{div} .float-right +[![AMQP logo](https://www.cleo.com/sites/default/files/styles/desktop_664_270_scale/public/2023-12/amqp-logo.png.webp){width=180px loading=lazy}][AMQP] +``` +```{div} .clearfix +``` + +:::{rubric} About +::: + +The [AMQP] protocol is an open standard application layer protocol for +message-oriented middleware. The defining features of AMQP are message +orientation, queuing, routing (including point-to-point and +publish-and-subscribe), reliability, and security. + +:::{rubric} Synopsis +::: + +Use LorryStream to receive JSON data from an AMQP queue, continuously loading +records into CrateDB. +```shell +uvx --from=lorrystream lorry relay \ + "amqp://guest:guest@localhost:5672/%2F?queue=testdrive&content-type=json" \ + "crate://localhost/?table=testdrive" +``` + +:::{rubric} Learn +::: + +[LorryStream] is a lightweight and polyglot stream-processing library, used as a +data backplane, message relay, or pipeline subsystem. + +::::{grid} + +:::{grid-item-card} Tutorial: Use LorryStream +:link: amqp-tutorial +:link-type: ref +How to load data from AMQP into CrateDB using LorryStream. +::: + +:::: + +:::{toctree} +:maxdepth: 1 +:hidden: +Tutorial +::: + + +[LorryStream]: https://lorrystream.readthedocs.io/ +[AMQP]: https://www.amqp.org/ diff --git a/docs/integrate/amqp/tutorial.md b/docs/integrate/amqp/tutorial.md new file mode 100644 index 00000000..9e677260 --- /dev/null +++ b/docs/integrate/amqp/tutorial.md @@ -0,0 +1,96 @@ +(amqp-tutorial)= + +# Load data from an AMQP queue into CrateDB + +The tutorial will walk you through starting the [RabbitMQ] AMQP broker +and CrateDB, publishing JSON data to an AMQP queue, consuming and relaying +it into a CrateDB table continuously, and validating that the data has +been stored successfully. +The data transfer is supported by the [LorryStream AMQP source] data +pipeline element. + +## Prerequisites + +Docker is used for running all components. This approach works consistently +across Linux, macOS, and Windows. + +Alternatively, you can use Podman. You can also use a different AMQP broker such as +Apache Qpid, Apache ActiveMQ, IBM MQ, or Solace. Azure Event Hubs and Azure Service +Bus speak AMQP as well, but with protocol and authentication specifics; adjust +settings accordingly. + +Create a shared network. +```shell +docker network create cratedb-demo +``` + +Start CrateDB. +```shell +docker run --name=cratedb --rm --network=cratedb-demo \ + --publish=4200:4200 --publish=5432:5432 \ + --env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node +``` + +Start RabbitMQ. +```shell +docker run --name=rabbitmq --rm --network=cratedb-demo \ + --publish=5672:5672 docker.io/rabbitmq:3 +``` +> Note: This broker configuration allows anonymous access for demonstration purposes only. +> Do not expose it to untrusted networks. For production, configure authentication/TLS. + +Prepare shortcuts for the CrateDB shell, LorryStream, and the AMQP client +programs. + +::::{tab-set} + +:::{tab-item} Linux and macOS +To make the settings persistent, add them to your shell profile (e.g., `~/.profile` or `~/.zshrc`). +```shell +alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" +alias lorry="docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry" +alias amqpcat="docker run --rm -i --network=cratedb-demo docker.io/cloudamqp/amqpcat amqpcat" +``` +::: +:::{tab-item} Windows PowerShell +To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). +```powershell +function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } +function lorry { docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry @args } +function amqpcat { docker run --rm -i --network=cratedb-demo docker.io/cloudamqp/amqpcat amqpcat @args } +``` +::: +:::{tab-item} Windows Command +```shell +doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* +doskey lorry=docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry $* +doskey amqpcat=docker run --rm -i --network=cratedb-demo docker.io/cloudamqp/amqpcat amqpcat $* +``` +::: + +:::: + +## Usage + +Invoke the data transfer pipeline. +```shell +lorry relay \ + "amqp://guest:guest@rabbitmq:5672/%2F?exchange=default&queue=default&routing-key=testdrive&setup=exchange,queue,bind&content-type=json" \ + "crate://cratedb/?table=testdrive" +``` + +Publish a JSON message to AMQP. +```shell +echo '{"temperature": 42.84, "humidity": 83.1}' | \ + amqpcat --producer --uri='amqp://guest:guest@rabbitmq:5672/%2F' \ + --exchange=default --queue=default --routing-key=testdrive +``` + +Inspect data stored in CrateDB. +```shell +crash --hosts cratedb -c "SELECT * FROM testdrive" +``` + + +[LorryStream AMQP source]: https://lorrystream.readthedocs.io/source/amqp.html +[RabbitMQ]: https://www.rabbitmq.com/ diff --git a/docs/integrate/index.md b/docs/integrate/index.md index 4603241c..47a6e9f4 100644 --- a/docs/integrate/index.md +++ b/docs/integrate/index.md @@ -17,6 +17,7 @@ Please also visit the [Overview of CrateDB integration tutorials]. :maxdepth: 1 airflow/index +amqp/index aws-lambda/index azure-functions/index cluvio/index From db98e988bde3b3f45b417e923220bd6d3441f127 Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Thu, 14 Aug 2025 14:10:43 +0200 Subject: [PATCH 2/3] Chore: Fix link to MongoDB --- docs/integrate/mongodb/tutorial.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/integrate/mongodb/tutorial.md b/docs/integrate/mongodb/tutorial.md index 2c31bc2d..d298ad2c 100644 --- a/docs/integrate/mongodb/tutorial.md +++ b/docs/integrate/mongodb/tutorial.md @@ -1,4 +1,5 @@ (mongodb-tutorial)= +(migrating-mongodb)= # Import data from MongoDB In this quick tutorial, you'll use the [CrateDB Toolkit MongoDB I/O subsystem] From 926e28a5652fa6d3c4f4aac2d9d80f3f5bd1e1c7 Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Thu, 14 Aug 2025 14:44:05 +0200 Subject: [PATCH 3/3] AMQP: Fix link checker on amqp.org -- it went south today ;] --- docs/conf.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/conf.py b/docs/conf.py index 62708699..ea1a2fc1 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -61,6 +61,8 @@ r"https://kubernetes.io/", # Connection to renenyffenegger.ch timed out. r"https://renenyffenegger.ch", + # Failed to establish a new connection: [Errno 111] Connection refused + r"https://www.amqp.org/", ] linkcheck_anchors_ignore_for_url += [