Kafka Project

A Node.js application integrating Apache Kafka with Express.js for message streaming, featuring a structured producer-consumer setup. This project demonstrates how to run Kafka on Docker, connect it to Node.js, and expose APIs to produce messages.

Table of Contents

• Project Structure
• Features
• Prerequisites
• Setup Instructions
• Running the Application
• Available Scripts
• API Endpoints
• Tech Stack

Project Structure

kafka-project/
├── config/
│   └── kafkaClient.js     # Kafka client configuration
├── routes/
│   └── messageRoutes.js   # Express routes for producing messages
├── .env                   # Environment variables
├── consumer.js            # Kafka consumer logic
├── docker-compose.yml     # Docker setup (Kafka, Zookeeper, Redis)
├── index.js               # Express app entry point
├── producer.js            # Kafka producer logic
├── package.json           # Node.js dependencies
└── README.md              # Project documentation

Features

• Kafka Producer & Consumer using KafkaJS
• Express.js API endpoint to send messages to Kafka topics
• Docker Compose setup for Kafka, Zookeeper, and Redis
• Environment-based configuration with .env file
• Modular and maintainable folder structure

Prerequisites

• Node.js (>= 16.x recommended)
• Docker & Docker Compose

Setup Instructions

1. Clone the Repository

git clone https://github.com/mixro/kafka-project.git
cd kafka-project

2. Install Dependencies

npm install

3. Configure Environment

Create a .env file in the root directory:

PORT=3000
BROKERS_ADDRESS=localhost:9092  # Use kafka:9092 if running Node.js inside Docker

4. Start Kafka via Docker

docker compose up -d

Verify containers are running:

docker ps

5. Create a Kafka Topic (if not auto-created)

docker exec -it kafka kafka-topics \
  --create --topic messaging \
  --bootstrap-server localhost:9092 \
  --partitions 1 --replication-factor 1

Running the Application

Start the Express Server

node index.js

The server will run at:

http://localhost:8400

Produce a Message via API

Send a message to a Kafka topic:

curl -X POST http://localhost:3000/api/messages/produce \
     -H "Content-Type: application/json" \
     -d '{"topic":"messaging","message":"Hello from Express!"}'

Expected response:

{
    "topic": "messaging",
    "message": "second message from producer",
    "status": "Message sent"
}

Run Kafka Consumer

Open another terminal and run:

node consumer.js

Expected output:

✅ Consumer connected
{ key: null, value: 'Hello from Express!', partition: 0, offset: '0' }

Available Scripts

• node index.js – Start the Express API server
• node producer.js – Run a standalone Kafka producer
• node consumer.js – Run the Kafka consumer

API Endpoints

Method	Endpoint	Description
POST	/api/messages/produce	Send a message to a Kafka topic

Tech Stack

• Node.js + Express.js: API server
• KafkaJS: Kafka client for Node.js
• Docker Compose: Orchestration for Kafka, Zookeeper, and Redis