Skip to content

Commit d38aa51

Browse files
amotlamotl
committed
Integrate/MQTT: Add entry point page and micro tutorial
The rig uses Eclipse Mosquitto and LorryStream to invoke the data transfer procedure, all based on using Docker.
1 parent 0953edc commit d38aa51

File tree

4 files changed

+177
-0
lines changed

4 files changed

+177
-0
lines changed

‎docs/ingest/etl/index.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,13 @@ Load data from database systems.
153153
application logs, website clickstreams, and IoT telemetry data, for machine
154154
learning (ML), analytics, and other applications.
155155

156+
- {ref}`mqtt`
157+
158+
MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT).
159+
It is designed as an extremely lightweight publish/subscribe messaging transport
160+
that is ideal for connecting remote devices with a small code footprint and minimal
161+
network bandwidth.
162+
156163
- {ref}`risingwave`
157164

158165
RisingWave is a stream processing and management platform that allows configuring

‎docs/integrate/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ marquez/index
4545
meltano/index
4646
metabase/index
4747
mongodb/index
48+
mqtt/index
4849
mysql/index
4950
n8n/index
5051
nifi/index

‎docs/integrate/mqtt/index.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
(mqtt)=
2+
# MQTT
3+
4+
```{div} .float-right
5+
[![MQTT logo](https://mqtt.org/assets/img/mqtt-logo.svg){width=180px loading=lazy}][MQTT]
6+
```
7+
```{div} .clearfix
8+
```
9+
10+
:::{rubric} About
11+
:::
12+
13+
[MQTT] is an OASIS standard messaging protocol for the Internet of Things (IoT).
14+
15+
It is designed as an extremely lightweight publish/subscribe messaging transport
16+
that is ideal for connecting remote devices with a small code footprint and minimal
17+
network bandwidth.
18+
19+
MQTT today is used in a wide variety of industries, such as automotive, manufacturing,
20+
telecommunications, and oil and gas. It enables efficient, reliable messaging between
21+
devices and backends over constrained networks.
22+
23+
:::{rubric} Synopsis
24+
:::
25+
26+
Use LorryStream to receive JSON data from an MQTT topic, continuously loading
27+
records into CrateDB.
28+
```shell
29+
uvx --from=lorrystream lorry relay \
30+
"mqtt://localhost/testdrive/%23?content-type=json" \
31+
"crate://localhost/?table=testdrive"
32+
```
33+
34+
:::{rubric} Learn
35+
:::
36+
37+
[LorryStream] is a lightweight and polyglot stream-processing library, used as a
38+
data backplane, message relay, or pipeline subsystem.
39+
[Node-RED] is a workflow automation tool that allows you to orchestrate message flows
40+
and transformations via a comfortable web interface.
41+
42+
::::{grid}
43+
44+
:::{grid-item-card} Tutorial: Use LorryStream
45+
:link: mqtt-tutorial
46+
:link-type: ref
47+
How to load data from an MQTT topic into CrateDB using LorryStream.
48+
:::
49+
50+
:::{grid-item-card} Tutorial: Use Node-RED
51+
:link: https://community.cratedb.com/t/ingesting-mqtt-messages-into-cratedb-using-node-red/803
52+
:link-type: url
53+
Ingesting MQTT messages into CrateDB using Node-RED.
54+
:::
55+
56+
::::
57+
58+
:::{toctree}
59+
:maxdepth: 1
60+
:hidden:
61+
Tutorial <tutorial>
62+
:::
63+
64+
65+
[LorryStream]: https://lorrystream.readthedocs.io/
66+
[MQTT]: https://mqtt.org/
67+
[Node-RED]: https://nodered.org/

‎docs/integrate/mqtt/tutorial.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
(mqtt-tutorial)=
2+
# Load data from an MQTT topic into CrateDB
3+
4+
The tutorial will walk you through starting the [Eclipse Mosquitto] broker and CrateDB,
5+
publishing JSON data to an MQTT topic, subscribing to the topic to relay
6+
data into a CrateDB table continuously, and validating that the data has
7+
been stored successfully.
8+
The data transfer is supported by the [LorryStream MQTT source] data
9+
pipeline element.
10+
11+
## Prerequisites
12+
13+
Docker is used for running all components. This approach works consistently
14+
across Linux, macOS, and Windows.
15+
16+
Alternatively, you can use Podman. You can also use a different MQTT broker such as
17+
EMQX, HiveMQ, VerneMQ, or RabbitMQ. Azure IoT Hub speaks MQTT as well, but with
18+
protocol and authentication specifics; adjust settings accordingly.
19+
20+
Create a shared network.
21+
```shell
22+
docker network create cratedb-demo
23+
```
24+
25+
Start CrateDB.
26+
```shell
27+
docker run --name=cratedb --rm -it --network=cratedb-demo \
28+
--publish=4200:4200 --publish=5432:5432 \
29+
--env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node
30+
```
31+
32+
Start Mosquitto.
33+
```shell
34+
docker run --name=mosquitto --rm -it --network=cratedb-demo \
35+
--publish=1883:1883 docker.io/eclipse-mosquitto \
36+
mosquitto -c /mosquitto-no-auth.conf
37+
```
38+
> Note: This broker configuration allows anonymous access for demonstration purposes only.
39+
> Do not expose it to untrusted networks. For production, configure authentication/TLS.
40+
41+
Prepare shortcuts for the CrateDB shell, LorryStream, and the Mosquitto client
42+
programs.
43+
44+
::::{tab-set}
45+
46+
:::{tab-item} Linux and macOS
47+
To make the settings persistent, add them to your shell profile (`~/.profile`).
48+
```shell
49+
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
50+
alias lorry="docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry"
51+
alias mosquitto_pub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_pub"
52+
alias mosquitto_sub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_sub"
53+
```
54+
:::
55+
:::{tab-item} Windows PowerShell
56+
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
57+
```powershell
58+
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
59+
function lorry { docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry @args }
60+
function mosquitto_pub { docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_pub @args }
61+
function mosquitto_sub { docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_sub @args }
62+
```
63+
:::
64+
:::{tab-item} Windows Command
65+
```shell
66+
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
67+
doskey lorry=docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry $*
68+
doskey mosquitto_pub=docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_pub $*
69+
doskey mosquitto_sub=docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_sub $*
70+
```
71+
:::
72+
73+
::::
74+
75+
## Usage
76+
77+
Subscribe to all MQTT topics on the broker to monitor any traffic.
78+
```shell
79+
mosquitto_sub -h mosquitto -t "#" -v
80+
```
81+
82+
Invoke the data transfer pipeline.
83+
```shell
84+
lorry relay \
85+
"mqtt://mosquitto/testdrive/%23?content-type=json" \
86+
"crate://cratedb/?table=testdrive"
87+
```
88+
89+
Publish a JSON message to an MQTT topic.
90+
```shell
91+
echo '{"temperature": 42.84, "humidity": 83.1}' | \
92+
mosquitto_pub -h mosquitto -t testdrive/channel1 -s
93+
```
94+
95+
Inspect data stored in CrateDB.
96+
```shell
97+
crash --hosts cratedb -c "SELECT * FROM testdrive"
98+
```
99+
100+
101+
[Eclipse Mosquitto]: https://mosquitto.org/
102+
[LorryStream MQTT source]: https://lorrystream.readthedocs.io/source/mqtt.html

0 commit comments

Comments
 (0)