diff --git a/README.md b/README.md index b359440..a4524a9 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,8 @@ -[![Build Status](https://travis-ci.org/HSLdevcom/mqtt-pulsar-gateway.svg?branch=master)](https://travis-ci.org/HSLdevcom/mqtt-pulsar-gateway) +## mqtt-pulsar-gateway [![Test and create Docker image](https://github.com/HSLdevcom/mqtt-pulsar-gateway/actions/workflows/test-and-build.yml/badge.svg)](https://github.com/HSLdevcom/mqtt-pulsar-gateway/actions/workflows/test-and-build.yml) -## Description +Application for reading data from MQTT topic and feeding it into Pulsar topic. This application doesn't care about the message payload, it just transfers the bytes. -Application for reading data from MQTT topic and feeding it into Pulsar topic. -This application doesn't care about the payload, it just transfers the bytes. +This project is part of [transitdata pipeline](https://github.com/HSLdevcom/transitdata), but it can also be used in other applications. ## Building @@ -11,36 +10,43 @@ This application doesn't care about the payload, it just transfers the bytes. This project depends on [transitdata-common](https://github.com/HSLdevcom/transitdata-common) project. -Either use released versions from public maven repository or build your own and install to local maven repository: - - ```cd transitdata-common && mvn install``` - ### Locally -- ```mvn compile``` -- ```mvn package``` +- `mvn compile` +- `mvn package` ### Docker image - Run [this script](build-image.sh) to build the Docker image - ## Running -Requirements: -- Pulsar Cluster - - By default uses localhost, override host in PULSAR_HOST if needed. - - Tip: f.ex if running inside Docker in OSX set `PULSAR_HOST=host.docker.internal` to connect to the parent machine - - You can use [this script](https://github.com/HSLdevcom/transitdata/blob/master/bin/pulsar/pulsar-up.sh) to launch it as Docker container -- Connection to an external MQTT server. - - Configure username and password via files - - Set filepath for username via env variable FILEPATH_USERNAME_SECRET, default is `/run/secrets/mqtt_broker_username` - - Set filepath for password via env variable FILEPATH_PASSWORD_SECRET, default is `/run/secrets/mqtt_broker_password` - - Mandatory: Set mqtt-topic via env variable MQTT_TOPIC - - Remember to use a unique MQTT client-id's if you have multiple instances connected to a single broker. - -All other configuration options are configured in the [config file](src/main/resources/environment.conf) -which can also be configured externally via env variable CONFIG_PATH - -Launch Docker container with +### Dependencies -```docker-compose -f compose-config-file.yml up ``` +* Pulsar + * You can use [this script](https://github.com/HSLdevcom/transitdata/blob/master/bin/pulsar/pulsar-up.sh) to launch it as Docker container +* Connection to an MQTT broker + +### Environment variables + +#### MQTT + +* `MQTT_CREDENTIALS_REQUIRED`: whether the broker needs credentials +* `FILEPATH_USERNAME_SECRET`: path to the file containing the username, default is `/run/secrets/mqtt_broker_username` +* `FILEPATH_PASSWORD_SECRET`: path to the file containing the password, default is `/run/secrets/mqtt_broker_password` +* `MQTT_BROKER_HOST`: MQTT broker URL +* `MQTT_TOPIC`: MQTT topic to subscribe +* `MQTT_QOS`: MQTT QoS +* `MQTT_MAX_INFLIGHT`: maximum MQTT messages inflight +* `MQTT_CLEAN_SESSION`: whether to open a clean MQTT session, i.e. if true, unacked messages are not redelivered +* `MQTT_CLIENT_ID`: client ID to use when subscribing to the MQTT topic. Remember to use unique client ID if there is more than one subscription +* `MQTT_ADD_RANDOM_TO_CLIENT_ID`: whether to append random string to the client ID +* `MQTT_KEEP_ALIVE_INTERVAL`: interval in seconds for MQTT keep-alive +* `MQTT_MANUAL_ACK`: whether to ack messages to the MQTT broker only after they have been successfully sent to Pulsar. Setting this to true might cause performance problems with high message rate + +#### Other + +* `IN_FLIGHT_ALERT_THRESHOLD`: send log message if the inflight message count is higher than this +* `MAX_MESSAGES_PER_SECOND`: maximum amount messages per second to be sent to Pulsar. If negative, no limit +* `MSG_MONITORING_INTERVAL`: send log message after this amount of messages has been processed +* `UNHEALTHY_MSG_SEND_INTERVAL_SECS`: consider the service unhealthy if no message has been sent to Pulsar in this period. If negative, no healthcheck for last sent message diff --git a/pom.xml b/pom.xml index 0995adb..23fd97a 100644 --- a/pom.xml +++ b/pom.xml @@ -2,14 +2,14 @@ 4.0.0 fi.hsl.transitlog mqtt-pulsar-gateway - 1.2.0 + 1.2.1 jar UTF-8 11 11 - 1.4.4 + 1.6.1-RC 1.2.5 @@ -63,7 +63,7 @@ org.mockito mockito-core - 4.11.0 + 5.2.0 test @@ -81,7 +81,7 @@ org.apache.maven.plugins maven-assembly-plugin - 3.4.2 + 3.5.0 package @@ -105,7 +105,7 @@ org.apache.maven.plugins maven-compiler-plugin - 3.10.1 + 3.11.0 11 11 @@ -114,7 +114,7 @@ org.apache.maven.plugins maven-surefire-plugin - 2.22.2 + 3.0.0