Kafka producer using Java

Updated On February 12, 2021 | By Mahesh Mogal


In this article, we will learn how to write Kafka producer in Java language. We will create a test topic then we will build our producer and set its properties. After that, we will start sending messages to that producer. We will also see how we can use Keys to send messages to the same partition in a particular topic.

Creating a topic

In Kafka, Producers send records to the topic. So we need one topic to which our producer sends records. You can use this code to create a new topic in the Kafka server. In the next steps, it is assumed that you have created the topic with name myTopic and we are going to use that to send messages from the producer.

Kafka Producer

Now we can start creating our own Kafka producer in Java. Let us see, how to set up Kafka Producer and set its properties.

In the above code, we can see that we have set up properties for the producer. We need to pass the list of the bootstrap server so that the producer can connect to them and start sending messages. In this example, we are using a producer which uses a key as well as messages in String format so that we are using String Serializer. CLIENT_ID_CONFIG property, we are setting a simple name to our producer in the Kafka server. In case of failure to send a message to Kafka topic, we want to try sending that message again. By setting RETRIES_CONFIG property, we can guarantee that in case of failure this producer will try sending that message two more times. You can learn more about Kafka Producers and their configuration here.

Sending messages

We need to create ProducerRecord to send messages to Kafka topic.

Here we have created producer record and we are sending our first message to Kafka topic. You can notice that we have not passed any key in this producer record. In the absence of the key, this message will be sent to any partitions on that topic. If we want to send the same type of messages to the same partitions on that topic then we need to pass key with that message.

All messages with key = 1 will be assigned to the same partition of this topic. Send function of producer returns record metadata which has information to which partition our message went. We can read that using the following code.

Conclusion

In this tutorial, we have created one simple Kafka producer in Java and understand its different configurations. you can download the code used in this article at git repository. In the next article, We will learn how to write consumer using Java and read messages from Kafka topics. Till then, Happy learning!!

Kafka Producers with java

Mahesh Mogal

I am passionate about Cloud, Data Analytics, Machine Learning, and Artificial Intelligence. I like to learn and try out new things. I have started blogging about my experience while learning these exciting technologies.

Stay Updated with Latest Blogs

Get latest blogs delivered to your mail directly.

Recent Posts

Where and Filter in Spark Dataframes

In this blog, we will learn how to filter rows from spark dataframe using Where and Filter functions.

Where and Filter in Spark Dataframes
Read More
Distinct Rows and Distinct Count from Spark Dataframe

Getting distinct values from columns or rows is one of most used operations. We will learn how to get distinct values as well as count of distinct values.

Distinct Rows and Distinct Count from Spark Dataframe
Read More
Sorting in Spark Dataframe

In this blog, we will learn how to sort rows in spark dataframe based on some column values.

Sorting in Spark Dataframe
Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram
Share via
Copy link
Powered by Social Snap