Automated Confluent Platform Deployment with Vagrant & Ansible (Keycloak/LDAP Integrated)🚀

This repository offers an automated solution for deploying the Confluent Platform. It leverages Vagrant to provision a virtual machine and Ansible playbooks to automatically install and configure all Confluent Platform components, including Apache Kafka®.

Optionally, this setup includes configurations for integrating Confluent Platform security with Keycloak and LDAP for robust authentication and authorization.

This automated deployment is designed for local development, testing environments, and proof-of-concept (POC) deployments, providing a quick and reliable way to get a Confluent Platform instance running with essential security features.

---

⚠️ Security Notice

This repository is intended for LOCAL DEVELOPMENT and TESTING ONLY.

All configuration files use placeholder values for passwords and secrets. You MUST configure your own credentials before use. Never use default or example credentials in production environments.

---

Features

• Automated VM provisioning with Vagrant
• Automated Confluent Platform installation and configuration with Ansible
• Optional integration with Keycloak for OIDC/OAuth2 authentication
• Optional integration with LDAP for user/group synchronization
• Ideal for local development, testing, and POCs

---

Requirements

Minimum Requirements:

• Vagrant 2.4.3 or higher
• Virtualization Provider: VMware Fusion/Workstation OR VirtualBox
• Ansible [core 2.18.3] or higher
• Python 3.13+
• Java OpenJDK 21+ (for running Confluent Platform)
• System Resources: 8GB RAM, 4 CPU cores, 100GB disk space (for the VM)

Tested Environment:

macOS 15
Vagrant 2.4.3
VMware Fusion 13.6.2 (or VirtualBox 7.0+)
Java openjdk version "21.0.5" 2024-10-15 LTS
ansible [core 2.18.3]
Python 3.13.2

---

🔧 Initial Configuration

Before starting, you need to configure your credentials. This repository includes example files with placeholder values that you must customize.

1. Configure Ansible Vault Password

Create your vault password file:

cp password.txt.example password.txt
# Edit password.txt and replace 'your-vault-password-here' with a strong password

2. Configure Variables for Ansible

Create your variables file:

cp vars.yaml.example vars.yaml
# Edit vars.yaml and set your LDAP password

3. Configure Keycloak and Database Credentials

Navigate to the keycloakLdap directory and create your environment file:

cd keycloakLdap
cp .env.example .env
# Edit .env and replace all placeholder values with your own credentials

Important: Make sure to set strong, unique passwords for:

• MARIADB_USER and MARIADB_PASSWORD
• MARIADB_ROOT_PASSWORD
• KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD

4. Update LDAP User Passwords

Edit keycloakLdap/ldif/02.ldif and replace the placeholder passwords ({SSHA}changeme) with properly hashed LDAP passwords for the users (alice, charlie, superUser).

5. Configure Keycloak Realm Secrets (Optional)

If you plan to use the Keycloak realm configuration, edit keycloakLdap/keycloak/realm/realm-export.json and replace the following placeholders with your own secrets:

• **REPLACE_WITH_YOUR_CLIENT_SECRET**
• **REPLACE_WITH_YOUR_CONTROL_CENTER_CLIENT_SECRET**
• **REPLACE_WITH_YOUR_HS512_SECRET**
• **REPLACE_WITH_YOUR_HMAC_SECRET**

---

Install Vagrant:

brew tap hashicorp/tap
brew install hashicorp/tap/hashicorp-vagrant

Download and install the Vagrant VMware Utility:

https://developer.hashicorp.com/vagrant/install/vmware

This will install the Vagrant VMware Utility on your system. After installing the Vagrant VMware Utility, the service will not be started until VMware Fusion has been installed. Once the service is started, Vagrant will be able to interact with the service via the Vagrant VMware Desktop plugin. To install the Vagrant VMware Desktop plugin into Vagrant run:

Install the vagrant-vmware-desktop:

vagrant plugin install vagrant-vmware-desktop

Choose Your Virtualization Provider

This repository supports both VMware and VirtualBox as virtualization providers.

Option 1: VMware (Recommended for macOS)

VMware Fusion (macOS) or VMware Workstation (Linux) provides better performance but requires a license.

Start Vagrant with VMware:

VAGRANT_VAGRANTFILE=./vagrant/Vagrantfile vagrant up --provider=vmware_desktop --no-tty

Option 2: VirtualBox (Free and Cross-platform)

VirtualBox is free and works on macOS, Linux, and Windows.

Install VirtualBox:

# macOS
brew install --cask virtualbox

# Linux (Ubuntu/Debian)
sudo apt-get install virtualbox

# Or download from: https://www.virtualbox.org/wiki/Downloads

Start Vagrant with VirtualBox:

VAGRANT_VAGRANTFILE=./vagrant/Vagrantfile vagrant up --provider=virtualbox --no-tty

Note: If you don't specify --provider, Vagrant will automatically use the first available provider on your system.

If you need to ssh to the newly created VM

VAGRANT_VAGRANTFILE=./vagrant/Vagrantfile vagrant ssh default

Install Ansible

brew install ansible

Checking Prerequisites for Installing Confluent Platform with Ansible

Prerequisites for Installing Confluent Platform with Ansible Playbooks

Install the Confluent Platform Ansible collection (latest release)

ansible-galaxy collection install confluent.platform --upgrade

## or if you want to install a specific version

ansible-galaxy collection install confluent.platform==7.8.0 --upgrade

Encrypting Variables with Ansible Vault

After configuring your vars.yaml file with your credentials, encrypt it using Ansible Vault:

ansible-vault encrypt vars.yaml --output=vars.encrypted.yaml --vault-password-file=password.txt

This command:

• Encrypts the vars.yaml file using Ansible Vault
• Outputs the encrypted content to vars.encrypted.yaml
• Uses the password from password.txt to avoid interactive prompts
• Leaves the original vars.yaml file unchanged (which is ignored by git)

Note: Only vars.encrypted.yaml should be committed to the repository. The vars.yaml file containing plain-text secrets is automatically ignored by .gitignore.

Create Kafka Server/Client certificates

./certs/up.sh

Install Confluent Platform with Ansible

You may need to add the following to the ansible.cfg file

[defaults]
hash_behavior = merge

and run ansible

ansible-playbook -i inventories/hosts.simple.yaml confluent.platform.all -e @vars.encrypted.yaml --vault-password-file=password.txt -e "ansible_ssh_private_key_file=.vagrant/machines/default/vmware_desktop/private_key"

or set the variable ANSIBLE_HASH_BEHAVIOUR=merge at runtime as the following

ANSIBLE_HASH_BEHAVIOUR=merge ansible-playbook -i inventories/hosts.simple.yaml confluent.platform.all -e @vars.encrypted.yaml --vault-password-file=password.txt -e "ansible_ssh_private_key_file=.vagrant/machines/default/vmware_desktop/private_key"

Inventories

There are different types of inventory:

• hosts.simple.yaml
• hosts.ccloud.yaml
• hosts.yaml with zookeeper and multiple listeners
• hosts.kraft.yaml with kraft and multiple listeners
• hosts.kraft.sso.yaml with kraft and OIDC/mTLS
• hosts.mTLS.C3sso.yaml with mTLS and Control Center using SSO

Destroy the VM

VAGRANT_VAGRANTFILE=./vagrant/Vagrantfile vagrant destroy

or forcefully destroys the virtual machine managed by Vagrant without prompting for confirmation.

VAGRANT_VAGRANTFILE=./vagrant/Vagrantfile vagrant destroy --force

---

🔒 Security Best Practices

Files That Should NEVER Be Committed

The following files contain sensitive information and are automatically ignored by .gitignore:

• password.txt - Your Ansible Vault password
• vars.yaml - Unencrypted variables with credentials
• keycloakLdap/.env - Database and Keycloak credentials
• Any *.key, *.pem, *.crt files - Private keys and certificates

What IS Safe to Commit

• *.example files - These contain only placeholder values
• vars.encrypted.yaml - Encrypted variables (safe as long as password.txt is not exposed)
• Configuration files with placeholders
• LDIF files with {SSHA}changeme placeholders

Important Reminders

1. Never use default passwords - Always generate strong, unique passwords for each service
2. Keep password.txt secure - This file decrypts your vault. Never share it or commit it to version control
3. Local development only - This setup is NOT hardened for production use
4. Rotate credentials regularly - Especially if you suspect any credential has been compromised
5. Review before committing - Always check what you're about to commit with git status and git diff

Need Help?

If you accidentally committed sensitive data:

1. Do NOT just delete the file - It remains in git history
2. Use git-filter-repo or similar tools to remove it from history
3. Rotate all exposed credentials immediately
4. Consider the exposed data as compromised

---

📝 License

See LICENSE file for details.