The basic concept of ApacheKafka and installation

In this blog, we will learn about the workflow of Kafka and its installation on Windows OS.

Before getting into it, will dive into the overall basic idea of Kafka

Kafka is a distributed messaging system that works on a distributed platform that sends data based on the pub-sub(publish-subscribe) model. For example, the producer sends(publish) message to a Topic and the consumer that subscribed to that topic will receive the messages

Producer: The producer sends the data to the Kafka server or broker
Consumer: Consumer receives (or) consumes the data from the Kafka server

The producer sends the data through Topics and each Topic has Partitions.

Kafka Server: It consists of multi nodes each having topics and partitions, where the producer send to a particular topic on multi nodes(replication factor)

Apache Zookeeper: Is a distributed, open-source configuration, synchronization service, like a configuration system which includes all the topic and its partitions, producers, consumer, and consumer groups everything.

Now getting into the Installation part

Navigate to the Official website of Apache Kafka and download the latest version and extract the zip file and extract the tar file inside it.

Make a new Kafka folder and copy the files

where the BIN folder contains all the libraries of Kafka and zookeeper.

For Windows -> Windows Batch files

For Linux -> Shell Script files

we have a config folder where all the property files are present. Here we need to edit two files

Goto file server.properties edit the log directory c:/kafka/kafka-logs

and next zookeeper.properties dataDirectory dataDir=c:/kafka/zookeeper-data

now all the edit changes are done…now open the command prompt and follow the steps

First, we should start the zookeeper server and defaultport:2181

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Open new terminal to start ApacheKafka server

.\bin\windows\kafka-server-start.bat .\config\server.properties

New Terminal to create a new topic (example)

.\bin\windows\kafka-topics.bat — create — zookeeper localhost:2181 — replication-factor 1 — partitions 1 — topic example

the command to list the topics

.\bin\windows\kafka-topics.bat — list — zookeeper localhost:2181

New Terminal to add messages to the particular topic example

.\bin\windows\kafka-console-producer.bat — broker-list localhost:9092 — topic example

> hello world!!
>happy learnings!!!

New Terminal to create the consumer that fetches the messages from the particular topic example

.\bin\windows\kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic example

— from-beginning

executing the above command will get the following messages the are present in the topic (example)

like

hello world!!

happy learnings!!!!

there you go …we had successfully fetched the message from a particular topic example by creating a producer and consumer

I hope with this blog you got the basic idea of the kafka workflow

Happy Learnings!!!