Kafka producer in .NET

Kafka sample producer service in .NET C#

Introduction

Apache kafka is a popular choice in event messaging systems. Kafka is a highly scalable, distributed event streaming platform that uses publisher-subscriber model. However, the intent of this article is not to discuss what is kafka in deep or its capabilities. So, lets just jump into  some practical example of kafka producer in .net application. Our aim is to show how to use kafka to send messages.

Here, First we are going to create a simple kafka producer which can send messages through kafka and we are going to write a console application in C# and .NET 7(not a requirement). In addition to that we are going to setup the docker to run kafka, zookeeper and some other useful tools.

Setup kafka on docker

Use this command to download the docker-compose yml file.

curl --silent --output docker-compose.yml https://raw.githubusercontent.com/confluentinc/cp-all-in-one/6.1.1-post/cp-all-in-one/docker-compose.yml
once downloaded, go to folder and run the below command.

docker-compose up -d

This will download and run all the related containers in docker. Once done, you can see the running containers in docker.

However, for this example, we are going to use only broker, zookeeper and control-center. Hence, you may edit the docker-compose.yml and remove unnecessary services.

Creating the .NET console application

Open Visual Studio and create a console application. For this example I have created a console application on top of .NET 7. However, for this example, the .NET version doesn’t really matter. 

This is a simple application that reads user input until the user type “exit”. Then it sends the user input to kafka service to publish the message to broker.

Then install the Confluent kafka nuget package.

NuGet\Install-Package Confluent.Kafka -Version 1.9.3

Program.cs (here i’m just using top level statements)

using KafkaProducer;

string input = string.Empty;
var producer = new ProducerService();

do
{
    Console.WriteLine("Enter message");
    input = Console.ReadLine() ?? string.Empty;
    producer.SendMessage(input);
}
while (!input.Equals("Exit", StringComparison.CurrentCultureIgnoreCase));

Console.WriteLine("Exiting....");
Environment.Exit(0);

Next we can add the simple Producer Service. Please note that here we are just using hard coded values just sake of keeping the app as much as simple. (Topic and Bootstrap server values are hardcoded)

ProducerService.cs

using Confluent.Kafka;
namespace KafkaProducer
{
    internal class ProducerService
    {
        public ProducerService() { }

        private readonly ProducerConfig config = new ProducerConfig
        { 
            BootstrapServers = "localhost:9092" 
        };

        private readonly string topic = "kafkaproducer.samplemessage";
        public void SendMessage(string message)
        {
            using (var producer = new ProducerBuilder<Null, string>(config).Build())
            {
                try
                {
                    producer.ProduceAsync(topic, new Message<Null, string> { Value = message })
                        .GetAwaiter()
                        .GetResult();
                }
                catch (Exception e)
                {
                    Console.WriteLine($"Something went wrong: {e}");
                }
            }
        }
    }
}

Now we can run the application and enter some messages.

Its all good and we have a very simple kafka message producer. In fact, The next step is to verify the messages whether we have them on kafka or not.

To do this we can use this awesome UI from confluent which we are already running on docker.

Type the url in browser: http://localhost:9021/clusters

You can see our healthy kafka cluster is running on docker.

Now click on cluster and go to “Topics”. Our topic is “kafkaproducer.samplemessage” which we have hard coded in Producer service.

Once you got to messages tab, use our producer app to send some new messages. UI will show you the latest messages.. Isn’t it awesome? 🙂

You can find the full source code here.