Apache Kafka : Developing Producer & Consumer in Java

Mar 31, 2024

In this article we will see how to implement "Kafka Producer" and "Kafka Consumer" in plain Java project.

1) Download and Install Kafka

Download the latest Kafka release and extract it:

% wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.12-3.2.0.tgz
% cd kafka_2.12-3.2.0

Run the following command in order to start ZooKeeper:

% bin/zookeeper-server-start.sh config/zookeeper.properties

Open another terminal session and run following command in order to start Kafka:

% bin/kafka-server-start.sh config/server.properties

2) Producer

2.1) Create Java Project

Run following command in order to create a "Java Projects" with "Maven":

mvn archetype:generate -DgroupId=com.cb -DartifactId=kafka-producer-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

2.2) Dependencies

Make sure to add required dependencies in "pom.xml" as shown below:

2.3) Kafka Producer

Here is the code for sample "Kafka Producer", we need to pass "Properties's" instance in "KafkaProducer" constructor.

Some of the "mandatory" properties includes - "bootstrap.servers", "client.id", "key.serializer" and "value.serializer".

2.4) Main

This is plain old java driver class; all the execution starts from here.

3) Consumer

3.1) Create Java Project

Run following command in order to create a "Java Projects" with "Maven":

mvn archetype:generate -DgroupId=com.cb -DartifactId=kafka-consumer-java -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

3.2) Dependencies

Make sure to add required dependencies in "pom.xml" as shown below:

3.3) Kafka Consumer

Here is the code for sample "Kafka Consumer", we need to pass "Properties's" instance in "KafkaProducer" constructor.

Some of the "mandatory" properties includes - "bootstrap.servers, "group.id", "enable.auto.commit", "auto.commit.interval.ms", "session.timeout.ms", "key.deserializer" and "value.serializer".

3.4) Main

This is plain old java driver class; all the execution starts from here.

4) Testing

Here, we are all done with "Producer" and "Consumer" project setup, lets start both applications one bye one.

4.1) Producer Logs

Message with id: '11' sent to partition(0), offset(95) in 59 ms
Message with id: '39' sent to partition(0), offset(96) in 5 ms
Message with id: '1' sent to partition(0), offset(97) in 6 ms
Message with id: '35' sent to partition(0), offset(98) in 5 ms
Message with id: '36' sent to partition(0), offset(99) in 6 ms
Message with id: '41' sent to partition(0), offset(100) in 5 ms
Message with id: '8' sent to partition(0), offset(101) in 6 ms
Message with id: '28' sent to partition(0), offset(102) in 4 ms
Message with id: '29' sent to partition(0), offset(103) in 4 ms
Message with id: '19' sent to partition(0), offset(104) in 6 ms
Message with id: '28' sent to partition(0), offset(105) in 5 ms
Message with id: '41' sent to partition(0), offset(106) in 6 ms

4.2) Consumer Logs

Received message for key: 11, at offset 95
Received message for key: 39, at offset 96
Received message for key: 1, at offset 97
Received message for key: 35, at offset 98
Received message for key: 36, at offset 99
Received message for key: 41, at offset 100
Received message for key: 8, at offset 101
Received message for key: 28, at offset 102
Received message for key: 29, at offset 103
Received message for key: 19, at offset 104
Received message for key: 28, at offset 105
Received message for key: 41, at offset 106

Source Code: Producer, Consumer