![](https://alphabsolutions.com/wp-content/uploads/2023/10/image-17-300x169.png)
This is our recording and observations as we setup a kafka cluster for the first time on a EKS cluster using strimzi. In this blog we will explore in detailed way of how to setup EKS cluster and running apache kafka producer and consumer program on it.
1.Configuring the AWS CLI
Ensure that the AWS CLI is installed in your machine.If it is not installed in your machine then install it by following the link given below
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
After installing AWS CLI ,configure AWS CLI using credential details (Access Key ID,Secret Access Key)
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws.png)
2.Creating EKS cluster
This is the time to create a cluster in EKS(Elastic Kubernetes Service).You can create a cluster using any one of the following options
(1) AWS CLI
(2) AWS Console Management
(3) eksctl
Here we are going to create a cluster using eksctl.For that,first we have to install eksctl. Install eksctl by following the link given below
https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html
After installing eksctl create a cluster using the following command
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws1.png)
This will create a cluster by the name kafka-poc with 4 nodes.
After the cluster is created, the appropriate kubernetes configuration will be added to your kubeconfig file.
5.Install strimzi kafka operator
install strimzi operator using the following command.This command will install strimzi operator with namespace kafka.
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws2.png)
6.Checking strimzi operator is running
We have installed strimzi kafka operator in previous step. We have to check whether it is installed in your cluster using the below command. If the status is running then the strimzi operator has installed in your cluster.
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws3.png)
6.Choosing a storage for your cluster
In kuberenetes two types of storage is there
1.ephemeral
2.persistent
If you are going to just setup kubernetes cluster for testing then you can go ahead with ephemeral storage. Suppose if you are going to setup kubernetes cluster for production then you can go ahead with persistent storage .In persistent storgae,data is stored permanantly.You have to provide a storage capacity to each node using EBS or EFS. In our setup we have used EBS for persitennt storage.
Create your amazon EBS CSI plugin IAM role with eksctl.
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws4.png)
Here you have to replace my-cluster with your cluster name,AmazonEBSCSIDriverPolicy with your driver policy and also replace AmazonEKS_EBS_CSI_DriverRole with your driver role
This command will create iam and attach required AWS managed policy to your cluster.
7.Install Kafka cluster operator
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws5.png)
Run the above command and edit the yaml file as per our requirement.The example yaml file is shown below .
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.3.1
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: loadbalancer
tls: false
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: "3.3"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 1Gi
deleteClaim: false
overrides:
- broker: 0
class: ebs-sc
- broker: 1
class: ebs-sc
- broker: 2
class: ebs-sc
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 1Gi
deleteClaim: false
overrides:
- broker: 0
class: ebs-sc
- broker: 1
class: ebs-sc
- broker: 2
class: ebs-sc
entityOperator:
topicOperator: {}
userOperator: {}
8. Checking zookeeper and Kafka is running
After installing kafka cluster, check zookeeper is running in your machine.
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws6.png)
After installing zookeeper, wait till kafka has to run in the cluster.
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws7.png)
10.Checking all the pods are running
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws8.png)
11.Troubleshoot your problem
If any pod is in pending status check your sc(storage class),PV(persistent Volume),pvc(persistent volume claim).
Kubectl get sc,pv,pvc,svc
(i).Check SC(storage class) is in your cluster
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws9.png)
Here two storage class is stored in your cluster such as ebs-sc, gp-2.gp-2 is the default storage class in your cluster.
(ii).check PV(Persistent Volume) in your cluster
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws10-1024x268.png)
(iii).Check PVC(Persistent Volume Claim) in your cluster
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws11-1024x177.png)
8.To create a topic
The topic should be created for your producer and consumer program.The topic is geeting created using the following command.
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws12.png)
9.To edit topic
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws13.png)
10.To get the hostname and port of kafka
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws14-1024x549.png)
![](https://alphabsolutions.com/wp-content/uploads/2023/10/aws15-1024x549.png)
In this END CERTIFICATE you can see host and port under addresses field.Take that host address and port number and set it in your program as clientId.When you run producer and consumer after changing that address ,producer used to produce the message continously to topic and consumer used to consume message from topic