What is Redpanda?
Redpanda is a streaming data platform. It is Kafka API-compatible, meaning you can use it with existing Kafka clients and tools. However, Redpanda is designed to be faster, more reliable, and easier to use than Kafka.Some key features of Redpanda:
- Compatibility with Kafka API: Redpanda is built to be API-compatible with Apache Kafka, allowing users to seamlessly migrate their Kafka applications to Redpanda.
- Performance: Redpanda is designed for high-performance and low-latency streaming applications.
- Resilience: Redpanda is designed to be highly resilient, providing features like automatic data replication and failover to ensure data availability and durability.
- 10x faster: Redpanda is designed to be up to 10x faster than Kafka. This is because it uses a thread-per-core architecture and a storage design that can push high-performance NVMe drives to their limits.
Here are some of the use cases for Redpanda:
- Event Streaming and Message Brokering: Redpanda is commonly used for building event-driven architectures where applications can produce and consume streams of events.
- Real-Time Analytics: It can be employed for real-time analytics, allowing organizations to process and analyze streaming data in real time.
- Log Aggregation: Redpanda can serve as a centralized platform for log aggregation, collecting and processing logs from various sources in a distributed and scalable manner.
- Monitoring and Metrics: Redpanda can be used to collect and process monitoring and metrics data in real time, providing insights into the performance and health of systems.
- Redpanda website: https://redpanda.com/
- Redpanda documentation: https://redpanda.com/docs/
- Redpanda GitHub repository: https://github.com/redpanda-data/redpanda
Prerequisite:
- Basic knowledge of AWS & EKS
- eksctl installed
Step 1: Create an EKS cluster
- Run the below command to create the EKS cluster in us-east-2region.
Code:
eksctl create cluster --name <cluster-name> --region us-east-2 --nodegroup-name nvme-workers --node-type c5d.2xlarge --nodes 3 --external-dns-access
- The above command will create the VPC, subnets, route table, etc resources in your account.
- If you want to use the default VPC and default subnets then you can create the cluster config file like below and create the cluster using the below command.
Code:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: redpanda-us-west-1
region: us-east-2
version: "1.27"
vpc:
subnets:
public:
us-east-2a: { id: subnet-50cc4a3b }
us-east-2b: { id: subnet-e9976b94 }
us-east-2c: { id: subnet-62e2c62e }
managedNodeGroups:
- name: nvme-workers
desiredCapacity: 3
instanceType: c5d.2xlarge
iam:
withAddonPolicies:
externalDNS: true
- Save the file with <filename.yaml> and run the eksctl create cluster -f <filename.yaml>
- We need to create 3 nodes in our worker group because each redpanda pod will be scheduled on a separate node.
- You can verify your cluster on AWS UI.



Step 2: Create a StorageClass for your local NVMe disks
- When you provisioned the Kubernetes cluster, you selected an instance type that comes with local NVMe disks. However, these disks are not automatically mounted or formatted upon creation. To use these local NVMe disks, you must mount and format them, and you must create the necessary PersistentVolumes (PVs). To automate this process, you can use a Container Storage Interface (CSI) driver.
- Install the LVM CSI driver
Code:
helm repo add metal-stack https://helm.metal-stack.io
helm repo update
helm install csi-driver-lvm metal-stack/csi-driver-lvm \
--namespace csi-driver-lvm \
--create-namespace \
--set lvm.devicePattern='/dev/nvme[1-9]n[0-9]'
- The lvm.devicePattern property specifies the pattern that the CSI driver uses to identify available NVMe volumes on your worker nodes.
Step 3: Create the StorageClass
- Create csi-driver-lvm-striped-xfs.yaml file and add the below code to it.
Code:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-driver-lvm-striped-xfs
provisioner: lvm.csi.metal-stack.io
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
type: "striped"
csi.storage.k8s.io/fstype: xfs
- provisioner: lvm.csi.metal-stack.io is responsible for provisioning the volume.
- reclaimPolicy: It is set to Retain so it will ensure that the underlying volume will not get deleted when the corresponding PVC is deleted.
- volumeBindingMode: The WaitForFirstConsumer mode delays the binding and provisioning of a PersistentVolume until a Pod that uses the PVC is created. This mode is important for ensuring that the PV is created on the same node where the Pod will run because the PV will use the node’s local NVMe volumes.
- allowVolumeExpansion: It will allow the volume to expand after it has been provisioned.
- parameters.type: Combines multiple physical volumes to create a single logical volume. In a striped setup, data is spread across the physical volumes in a way that distributes the I/O load evenly, improving performance by allowing parallel disk I/O operations.
- parameters.csi.storage.k8s.io/fstype: Formats the volumes with the XFS file system. Redpanda Data recommends XFS for its enhanced performance with Redpanda workloads.
- Run the below command to create the storage class.
- After applying this StorageClass, any PVC that references it will attempt to provision storage using the LVM CSI driver and the provided parameters.
Step 4: Configure external access
- We need to configure the EKS cluster to allow external access to the node ports on which the Redpanda deployment will be exposed. You use these node ports in later steps to configure external access to your Redpanda cluster.
- Run the below command to get the ID of the security group that’s associated with the nodes in your EKS cluster
Code:
AWS_SECURITY_GROUP_ID=`aws eks describe-cluster --name <cluster-name> --region <region> | jq -r '.cluster.resourcesVpcConfig.clusterSecurityGroupId'`
- Run the below command to add inbound firewall rules to your EC2 instances so that external traffic can reach the node ports exposed on all Kubernetes worker nodes in the cluster
Code:
aws ec2 authorize-security-group-ingress --region <region> \
--group-id ${AWS_SECURITY_GROUP_ID} \
--ip-permissions '[
{
"IpProtocol": "tcp",
"FromPort": 30081,
"ToPort": 30081,
"IpRanges": [{"CidrIp": "0.0.0.0/0"}]
},
{
"IpProtocol": "tcp",
"FromPort": 30082,
"ToPort": 30082,
"IpRanges": [{"CidrIp": "0.0.0.0/0"}]
},
{
"IpProtocol": "tcp",
"FromPort": 31644,
"ToPort": 31644,
"IpRanges": [{"CidrIp": "0.0.0.0/0"}]
},
{
"IpProtocol": "tcp",
"FromPort": 31092,
"ToPort": 31092,
"IpRanges": [{"CidrIp": "0.0.0.0/0"}]
}
]'
Step 5: Install the cert manager
- Run the below command to install the cert manager.
Code:
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--set installCRDs=true \
--namespace cert-manager \
--create-namespace
- TLS is enabled by default. The Redpanda Helm chart uses a cert manager to manage TLS certificates by default.
Step 6: Install the Redpanda and Redpanda console
- Run the below command to install the redpanda and redpanda console.
Code:
helm repo add redpanda https://charts.redpanda.com \
helm install redpanda redpanda/redpanda \
--namespace <namespace> --create-namespace \
--set "storage.persistentVolume.storageClass=csi-driver-lvm-striped-xfs" \
--wait \
--timeout 1h
- The above command will create the redpanda cluster with SASL disabled. If you want to enable the SASL then you can run the below command.
- storage.persistentVolume.storageClass: Points each PVC associated with the Redpanda brokers to the csi-driver-lvm-striped-xfs StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker.
Code:
helm install redpanda redpanda/redpanda \
--namespace <namespace> --create-namespace \
--set auth.sasl.enabled=true \
--set "auth.sasl.users[0].name=superuser" \
--set "auth.sasl.users[0].password=secretpassword" \
--set external.domain=customredpandadomain.local \
--set "storage.persistentVolume.storageClass=csi-driver-lvm-striped-xfs" \
--wait \
--timeout 1h
- You can replace the username and password as per your need.
- external.domain: The custom domain that each broker advertises to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain.
- auth.sasl.name: Creates a superuser called superuser that can grant permissions to new users in your cluster using access control lists (ACLs).
Step 7: Verify the resources
- Run the below command to verify the resources in your namespace.
Code:
kubectl get all -n <namespace>
dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$ kubectl get all -n <namespace>
NAME READY STATUS RESTARTS AGE
pod/redpanda-0 2/2 Running 0 5h56m
pod/redpanda-1 2/2 Running 0 5h56m
pod/redpanda-2 2/2 Running 0 5h56m
pod/redpanda-configuration-w6l98 0/1 Completed 0 5h55m
pod/redpanda-console-7f48c9f7df-kxzxj 1/1 Running 0 5h56m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redpanda ClusterIP None <none> 9644/TCP,8082/TCP,9093/TCP,33145/TCP,8081/TCP 5h56m
service/redpanda-console ClusterIP 10.100.194.152 <none> 8080/TCP 5h56m
service/redpanda-external NodePort 10.100.36.197 <none> 9645:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP 5h56m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redpanda-console 1/1 1 1 5h56m
NAME DESIRED CURRENT READY AGE
replicaset.apps/redpanda-console-7f48c9f7df 1 1 1 5h56m
NAME READY AGE
statefulset.apps/redpanda 3/3 5h56m
NAME COMPLETIONS DURATION AGE
job.batch/redpanda-configuration 1/1 14s 5h55m
- Run the below command to verify that each Redpanda broker is scheduled on only one Kubernetes node
Code:
kubectl get pod --namespace <namespace> \
-o=custom-columns=NODE:.spec.nodeName,POD_NAME:.metadata.name -l \
app.kubernetes.io/component=redpanda-statefulset
[dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$ kubectl get pod --namespace <namespace> \
> -o=custom-columns=NODE:.spec.nodeName,POD_NAME:.metadata.name -l \
> app.kubernetes.io/component=redpanda-statefulset
NODE POD_NAME
ip-172-31-41-181.us-east-2.compute.internal redpanda-0
ip-172-31-27-63.us-east-2.compute.internal redpanda-1
ip-172-31-10-9.us-east-2.compute.internal redpanda-2
- Run the below command to verify that each Redpanda broker has a bound PVC
Code:
kubectl get persistentvolumeclaim \
--namespace <namespace> \
-o custom-columns=NAME:.metadata.name,STATUS:.status.phase,STORAGECLASS:.spec.storageClassName
[dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$ kubectl get persistentvolumeclaim \
> --namespace <namespace> -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,STORAGECLASS:.spec.storageClassName
NAME STATUS STORAGECLASS
datadir-redpanda-0 Bound csi-driver-lvm-striped-xfs
datadir-redpanda-1 Bound csi-driver-lvm-striped-xfs
datadir-redpanda-2 Bound csi-driver-lvm-striped-xfs
- Run the below command to verify if the Redpanda console is deployed
Code:
kubectl get deployments -n <namespace>
[dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$ kubectl get deployment -n <namespace>
NAME READY UP-TO-DATE AVAILABLE AGE
redpanda-console 1/1 1 1 5h59m
- Run the below command to verify that PVCs are bound
Code:
kubectl get pvc -n <namespace>
[dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$ kubectl get pvc -n <namespace>
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-redpanda-0 Bound pvc-40ab703f-7d44-419e-a0ad-02b07481630b 20Gi RWO csi-driver-lvm-striped-xfs 6h
datadir-redpanda-1 Bound pvc-6828c920-8ee6-44a8-9530-3b9a86560d75 20Gi RWO csi-driver-lvm-striped-xfs 6h
datadir-redpanda-2 Bound pvc-5a733771-1203-427f-ae7a-410fb52a8c4c 20Gi RWO csi-driver-lvm-striped-xfs 6h
- Run the below command to create external access to the Redpanda console
Code:
kubectl --namespace <namespace> port-forward svc/redpanda-console 8080:8080
- You can verify the console on localhost:8080

Console
- Run the below command to create the alias for rpk
- rpk is a command line interface (CLI) toolbox that lets you configure, manage, and tune Redpanda clusters. It also let you manage topics, groups, and access control lists (ACLs). rpk stands for Redpanda Keeper.
Code:
alias rpk="kubectl --namespace <namespace> exec -i -t redpanda-0 -c redpanda -- rpk"
- Run the below command to list the topics
Code:
rpk topic list
NAME PARTITIONS REPLICAS
_schemas 1 3
dhruvin 1 3
dhsoni 1 3
- Run the below command to check the cluster health
Code:
rpk cluster health
CLUSTER HEALTH OVERVIEW
=======================
Healthy: true
Unhealthy reasons: []
Controller ID: 0
All nodes: [0 1 2]
Nodes down: []
Leaderless partitions (0): []
Under-replicated partitions (0): []
- Run the below command to create the topic
Code:
rpk topic create redpanda
TOPIC STATUS
redpanda OK
[dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$
[dhruvinsoni@Dhruvins-MacBook-Pro redpanda (⎈|[email protected]:N/A)]$ rpk topic list
NAME PARTITIONS REPLICAS
_schemas 1 3
dhruvin 1 3
dhsoni 1 3
redpanda 1 3
- Run the below command to produce the message in the topic
Code:
rpk topic produce <topic-name>
Type a message, then press Enter
- Example output:
- Press Ctrl+C to finish producing messages to the topic.
- Run the below command to consume one message from the topic
Code:
rpk topic consume redpanda --num 1
- Explore the topics in redpanda console
Last edited: