Sample Kafka & Redis Application

Sample template for building real-time processing applications with Kafka & Redis

Getting Started

Prerequisites

You will need running Kafka & Redis instances. To set them up, see:

• Kafka Quickstart: https://kafka.apache.org/quickstart
• Redis Quickstart: https://redis.io/topics/quickstart

Using the Template

• Main "high-level" dataflow should be scripted into a class implementing TopologyConstructor interface under constructTopology method (see SampleTopologyConstructor)
• Implement your data transformation logic from a variety of transformation interfaces provided by Kafka Streams (e.g. ValueMapper, Transformer, etc), which will be used by the constructTopology method
• For more information, see: https://kafka.apache.org/documentation/streams/

Writing Test Cases

• To pipe in test data through Topology/Tranformations, extend the StreamsIntegrationTestBase class (see example SampleIntegrationTestBase)

Building the Application

• To build the application, cd to project home and execute mvn clean install

Notes

• kafka.security.integration in application.properties - to set to true for integration with Kerberos
• redis.iscluster in application.properties - to use redis cluster mode
• Alternative: Use of @Profile("myprofile") to instantiate conditional beans (e.g. KafkaSecurityConfiguration)
• Alternative: Use of havingValue = in @ConditionalOnProperty annotation to turn on/off features

Built With

• Kafka - Distributed Streaming Platform for Realtime Data Processing
• Redis - In-Memory NoSQL KV Database
• Spring Boot - Opinionated Framework for Spring Based Applications for Dependency Injections
• Java - OpenJDK from AdoptOpenJDK
• Maven - Dependency Management
• Lombok - Framework for Reducing Boilerplate Codes