diff --git a/docs/modules/kubernetes/pages/shadow-linking.adoc b/docs/modules/kubernetes/pages/shadow-linking.adoc new file mode 120000 index 00000000..715333b9 --- /dev/null +++ b/docs/modules/kubernetes/pages/shadow-linking.adoc @@ -0,0 +1 @@ +../../../../kubernetes/shadow-linking/README.adoc \ No newline at end of file diff --git a/kubernetes/shadow-linking/.gitignore b/kubernetes/shadow-linking/.gitignore new file mode 100644 index 00000000..9fd4f9b7 --- /dev/null +++ b/kubernetes/shadow-linking/.gitignore @@ -0,0 +1 @@ +*.pid diff --git a/kubernetes/shadow-linking/README.adoc b/kubernetes/shadow-linking/README.adoc new file mode 100644 index 00000000..6961f412 --- /dev/null +++ b/kubernetes/shadow-linking/README.adoc @@ -0,0 +1,218 @@ += Explore Shadow Linking for Disaster Recovery +:env-kubernetes: true +:page-categories: Disaster Recovery, Data Replication +:description: Deploy two Redpanda clusters on Kubernetes and configure shadow linking to continuously replicate topics, consumer group offsets, and Schema Registry data. +:page-layout: lab + +Shadow linking is a built-in disaster recovery mechanism that continuously replicates topics, consumer group offsets, and Schema Registry data from a source cluster to a shadow cluster running on Kubernetes. This lab demonstrates how shadow linking works and how you can use it to fail over to a shadow cluster. + +In this lab, you deploy two single-node Redpanda clusters on Kubernetes and configure a shadow link between them. You then explore basic topic shadowing, Schema Registry shadowing, and consumer group failover. + +== Prerequisites + +You must have the following: + +* https://kubernetes.io/docs/tasks/tools/[kubectl^] CLI installed +* https://helm.sh/docs/intro/install/[Helm 3+^] installed +* xref:ROOT:get-started:rpk-install.adoc[rpk] CLI installed +* https://kind.sigs.k8s.io/docs/user/quick-start/#installation[kind^] installed (for local testing) +* https://docs.docker.com/get-docker/[Docker^] installed and running + +== What gets deployed + +The setup script installs the following components: + +* *cert-manager* (namespace: `cert-manager`): Manages TLS certificates +* *Redpanda Operator* (namespace: `rp-operator`): Manages Redpanda cluster lifecycle +* *Source cluster* (namespace: `source`): Primary cluster where you write data +* *Shadow cluster* (namespace: `shadow`): Replication target cluster +* *ShadowLink resource*: Defines the replication relationship between clusters + +The shadow link configuration mirrors all topics using wildcard filtering, replicates consumer group offsets, and synchronizes the Schema Registry, all at 5-second intervals. + +== Run the lab + +. Clone this repository: ++ +[,bash] +---- +git clone https://github.com/redpanda-data/redpanda-labs.git +cd redpanda-labs/kubernetes/shadow-linking +---- + +. Create a local Kubernetes cluster using kind: ++ +[,bash] +---- +kind create cluster --name redpanda-shadow +---- + +. Run the setup script to deploy both clusters and configure shadow linking: ++ +[,bash] +---- +./setup.sh +---- ++ +The script installs cert-manager, deploys the Redpanda Operator, creates the source and shadow clusters, configures the shadow link, sets up port forwarding for local access, and creates rpk profiles for both clusters. + +. Verify both clusters are healthy: ++ +[,bash] +---- +rpk --profile source cluster health +rpk --profile shadow cluster health +---- + +== Explore basic topic shadowing + +This section shows how topics and messages automatically replicate from the source to the shadow cluster. + +. Create a topic on the source cluster: ++ +[,bash] +---- +rpk --profile source topic create basic -p1 -r1 +---- + +. Produce a message to the source cluster: ++ +[,bash] +---- +echo "Hello, world" | rpk --profile source topic produce basic +---- + +. Wait a few seconds for replication, then list topics on the shadow cluster: ++ +[,bash] +---- +rpk --profile shadow topic list +---- ++ +You should see the `basic` topic appear on the shadow cluster. + +. Consume messages from both clusters to verify data replication: ++ +[,bash] +---- +rpk --profile source topic consume basic -n1 +rpk --profile shadow topic consume basic -n1 +---- ++ +The same message appears on both clusters. + +== Explore Schema Registry shadowing + +This section shows how Avro schemas registered on the source cluster replicate to the shadow cluster, allowing shadow-side message deserialization. + +. Register an Avro schema on the source cluster: ++ +[,bash] +---- +rpk --profile source registry schema create syslog-value --schema resources/syslog.avsc --type avro +---- + +. Create a topic for schema-encoded messages: ++ +[,bash] +---- +rpk --profile source topic create syslog -p1 -r1 +---- + +. Produce Avro-encoded syslog records to the source cluster: ++ +[,bash] +---- +cat resources/syslogs.json | rpk --profile source topic produce syslog --schema-id=topic +---- + +. Wait a few seconds, then list schemas on both clusters: ++ +[,bash] +---- +rpk --profile source registry subject list +rpk --profile shadow registry subject list +---- ++ +The `syslog-value` schema appears on both clusters. + +. Consume decoded messages from the shadow cluster using the replicated schema: ++ +[,bash] +---- +rpk --profile shadow topic consume syslog -n1 --use-schema-registry -f'%v\n' | jq +---- ++ +The shadow cluster can deserialize messages using the replicated schema. + +== Explore consumer group failover + +This section shows how consumer group offsets replicate to the shadow cluster, allowing consumers to resume from their exact position after a failover. + +. Create a topic on the source cluster: ++ +[,bash] +---- +rpk --profile source topic create foo -p1 -r1 +---- + +. Produce records and consume them on the source cluster: ++ +[,bash] +---- +seq 0 2 | rpk --profile source topic produce foo +rpk --profile source topic consume foo -n3 -g consumer-group-foo +---- ++ +This creates a consumer group and processes the messages. + +. View consumer group offsets on both clusters: ++ +[,bash] +---- +rpk --profile source group describe consumer-group-foo +rpk --profile shadow group describe consumer-group-foo +---- ++ +The consumer group and its offsets are replicated to the shadow cluster. + +. Trigger a failover to the shadow cluster: ++ +[,bash] +---- +rpk --profile shadow shadow failover shadow-link --all --no-confirm +rpk --profile shadow shadow status shadow-link +---- ++ +This command promotes the shadow cluster to become writable. + +. Produce more records and consume on the shadow cluster: ++ +[,bash] +---- +seq 3 5 | rpk --profile shadow topic produce foo +rpk --profile shadow topic consume foo -n3 -g consumer-group-foo +---- ++ +The consumer resumes from the last committed offset (3) and consumes the new messages. + +== Clean up + +To stop port forwarding, terminate the background processes: + +[,bash] +---- +kill $(cat local-port-forward.pid) +---- + +To delete the kind cluster and all resources: + +[,bash] +---- +kind delete cluster --name redpanda-shadow +---- + +== Suggested reading + +* xref:ROOT:manage:disaster-recovery/shadowing/index.adoc[Shadowing for Disaster Recovery] +* xref:ROOT:deploy:deployment-option/self-hosted/kubernetes/index.adoc[Redpanda on Kubernetes] diff --git a/kubernetes/shadow-linking/config b/kubernetes/shadow-linking/config new file mode 100644 index 00000000..02a2618f --- /dev/null +++ b/kubernetes/shadow-linking/config @@ -0,0 +1,4 @@ +export CERT_MANAGER_NAMESPACE=cert-manager +export OPERATOR_NAMESPACE=rp-operator +export SOURCE_REDPANDA_NAMESPACE=source +export SHADOW_REDPANDA_NAMESPACE=shadow diff --git a/kubernetes/shadow-linking/resources/shadow-cluster.yaml b/kubernetes/shadow-linking/resources/shadow-cluster.yaml new file mode 100644 index 00000000..b7b7e3fd --- /dev/null +++ b/kubernetes/shadow-linking/resources/shadow-cluster.yaml @@ -0,0 +1,43 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: shadow +--- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Redpanda +metadata: + name: redpanda + namespace: shadow +spec: + clusterSpec: + image: + tag: v25.3.4 + external: + enabled: true + service: + enabled: false + addresses: + - localhost + listeners: + kafka: + external: + default: + enabled: true + port: 9094 + advertisedPorts: + - 29094 + statefulset: + replicas: 1 + config: + cluster: + default_topic_replications: 1 + enable_shadow_linking: true + storage: + tiered: + config: + cloud_storage_enabled: false + tls: + enabled: false + auth: + sasl: + enabled: false diff --git a/kubernetes/shadow-linking/resources/shadow-link.yaml b/kubernetes/shadow-linking/resources/shadow-link.yaml new file mode 100644 index 00000000..1aedf808 --- /dev/null +++ b/kubernetes/shadow-linking/resources/shadow-link.yaml @@ -0,0 +1,46 @@ +apiVersion: cluster.redpanda.com/v1alpha2 +kind: ShadowLink +metadata: + name: shadow-link + namespace: shadow +spec: + + # What cluster are we writing to? + shadowCluster: + staticConfiguration: + kafka: + brokers: ["redpanda.shadow.svc.cluster.local:9093"] + admin: + urls: ["http://redpanda.shadow.svc.cluster.local:9644/"] + schemaRegistry: + urls: ["http://redpanda.shadow.svc.cluster.local:8081/"] + + # What cluster are we reading from? + sourceCluster: + staticConfiguration: + kafka: + brokers: ["redpanda.source.svc.cluster.local:9093"] + admin: + urls: ["http://redpanda.source.svc.cluster.local:9644/"] + schemaRegistry: + urls: ["http://redpanda.source.svc.cluster.local:8081/"] + + # Replicate all topics (except for any that are pre-filtered) + topicMetadataSyncOptions: + interval: 5s + autoCreateShadowTopicFilters: + - name: '*' + filterType: include + patternType: literal + + # Replicate all consumer groups + consumerOffsetSyncOptions: + interval: 5s + groupFilters: + - patternType: literal + filterType: include + name: '*' + + # Replicate schema registry (using _schemas topic replication) + schemaRegistrySyncOptions: + schema_registry_shadowing_mode: "topic" diff --git a/kubernetes/shadow-linking/resources/source-cluster.yaml b/kubernetes/shadow-linking/resources/source-cluster.yaml new file mode 100644 index 00000000..31291050 --- /dev/null +++ b/kubernetes/shadow-linking/resources/source-cluster.yaml @@ -0,0 +1,43 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: source +--- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Redpanda +metadata: + name: redpanda + namespace: source +spec: + clusterSpec: + image: + tag: v25.3.4 + external: + enabled: true + service: + enabled: false + addresses: + - localhost + listeners: + kafka: + external: + default: + enabled: true + port: 9094 + advertisedPorts: + - 19094 + statefulset: + replicas: 1 + config: + cluster: + default_topic_replications: 1 + enable_shadow_linking: true + storage: + tiered: + config: + cloud_storage_enabled: false + tls: + enabled: false + auth: + sasl: + enabled: false diff --git a/kubernetes/shadow-linking/resources/syslog.avsc b/kubernetes/shadow-linking/resources/syslog.avsc new file mode 100644 index 00000000..b9c875f2 --- /dev/null +++ b/kubernetes/shadow-linking/resources/syslog.avsc @@ -0,0 +1,24 @@ +{ + "type": "record", + "name": "syslog", + "fields": [ + {"name": "name", "type": "string"}, + {"name": "type", "type": "string"}, + {"name": "message", "type": "string"}, + {"name": "host", "type": "string"}, + {"name": "version", "type": "string"}, + {"name": "tag", "type": "string"}, + {"name": "level", "type": "int"}, + {"name": "facility", "type": "string"}, + {"name": "severity", "type": "int"}, + {"name": "appName", "type": "string"}, + {"name": "remoteAddress", "type": "string"}, + {"name": "rawMessage", "type": "string"}, + {"name": "processId", "type": "string"}, + {"name": "messageId", "type": "string"}, + {"name": "deviceVendor", "type": "string"}, + {"name": "deviceProduct", "type": "string"}, + {"name": "deviceVersion", "type": "string"}, + {"name": "ts", "type": "int"} + ] +} diff --git a/kubernetes/shadow-linking/resources/syslogs.json b/kubernetes/shadow-linking/resources/syslogs.json new file mode 100644 index 00000000..e2da0c82 --- /dev/null +++ b/kubernetes/shadow-linking/resources/syslogs.json @@ -0,0 +1,3 @@ +{"name":"log-001","type":"RFC5424","message":"System startup complete","host":"192.0.2.10","version":"3.25.1","tag":".source.s_src","level":6,"facility":"syslog","severity":6,"appName":"SYSTEM","remoteAddress":"198.51.100.1","rawMessage":"System startup complete","processId":"1001","messageId":"10001","deviceVendor":"example","deviceProduct":"server","deviceVersion":"1.0","ts":1} +{"name":"log-002","type":"RFC5424","message":"User authentication successful","host":"192.0.2.11","version":"3.25.1","tag":".source.s_src","level":6,"facility":"authpriv","severity":6,"appName":"AUTH","remoteAddress":"198.51.100.2","rawMessage":"User authentication successful","processId":"1002","messageId":"10002","deviceVendor":"example","deviceProduct":"server","deviceVersion":"1.0","ts":2} +{"name":"log-003","type":"RFC5424","message":"Configuration updated","host":"192.0.2.12","version":"3.25.1","tag":".source.s_src","level":5,"facility":"syslog","severity":5,"appName":"CONFIG","remoteAddress":"198.51.100.3","rawMessage":"Configuration updated","processId":"1003","messageId":"10003","deviceVendor":"example","deviceProduct":"server","deviceVersion":"1.0","ts":3} diff --git a/kubernetes/shadow-linking/setup.sh b/kubernetes/shadow-linking/setup.sh new file mode 100755 index 00000000..1528f7a2 --- /dev/null +++ b/kubernetes/shadow-linking/setup.sh @@ -0,0 +1,56 @@ +#!/bin/bash +set -e + +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) +pushd $SCRIPT_DIR > /dev/null + +source ./config + +echo "Installing cert-manager..." +helm repo add jetstack https://charts.jetstack.io --force-update > /dev/null 2>&1 +helm upgrade --install cert-manager jetstack/cert-manager \ + --set crds.enabled=true \ + --namespace ${CERT_MANAGER_NAMESPACE} \ + --create-namespace \ + --wait > /dev/null 2>&1 + +echo "Installing Redpanda Operator..." +helm repo add redpanda https://charts.redpanda.com --force-update > /dev/null 2>&1 +helm upgrade --install redpanda-controller redpanda/operator \ + --namespace ${OPERATOR_NAMESPACE} \ + --create-namespace \ + --version v25.3.1 \ + --set crds.enabled=true \ + --wait > /dev/null 2>&1 + +echo "Deploying source cluster (this may take a few minutes)..." +kubectl apply -f resources/source-cluster.yaml > /dev/null +kubectl wait -n ${SOURCE_REDPANDA_NAMESPACE} redpanda/redpanda --for=condition=Ready --timeout=600s 2>/dev/null + +echo "Deploying shadow cluster (this may take a few minutes)..." +kubectl apply -f resources/shadow-cluster.yaml > /dev/null +kubectl wait -n ${SHADOW_REDPANDA_NAMESPACE} redpanda/redpanda --for=condition=Ready --timeout=600s 2>/dev/null + +echo "Creating shadow link..." +kubectl apply -f resources/shadow-link.yaml > /dev/null + +echo "Setting up port forwarding and rpk profiles..." +kubectl port-forward pod/redpanda-0 -n $SOURCE_REDPANDA_NAMESPACE 19094:9094 19644:9644 18081:8081 > /dev/null 2>&1 & +echo $! > local-port-forward.pid + +kubectl port-forward pod/redpanda-0 -n $SHADOW_REDPANDA_NAMESPACE 29094:9094 29644:9644 28081:8081 > /dev/null 2>&1 & +echo $! >> local-port-forward.pid + +sleep 2 + +rpk profile create source -s brokers=localhost:19094 -s admin.hosts=localhost:19644 -s registry.hosts=http://localhost:18081/ > /dev/null 2>&1 || rpk profile use source > /dev/null 2>&1 +rpk profile create shadow -s brokers=localhost:29094 -s admin.hosts=localhost:29644 -s registry.hosts=http://localhost:28081/ > /dev/null 2>&1 || rpk profile use shadow > /dev/null 2>&1 + +popd > /dev/null + +echo "" +echo "Setup complete! Both clusters are ready." +echo "" +echo "Verify cluster health:" +echo " rpk --profile source cluster health" +echo " rpk --profile shadow cluster health"