Kafka consumers play a vital role in the functionality of a Kafka cluster. Kafka uses the publisher-subscriber mode where the publishers can write the messages on a given topic. The part of consumers is to subscribe to a given topic and read the messages from that topic.

Apart from reading the messages from a Kafka topic, Kafka consumers are responsible for a variety of other functionalities such as:

Data processing and transformation – Kafka consumers can perform the data processing and conversion on the messages that they read from Kafka topics before sending them to downstream systems or applications. Such data processing operations include data filtering, data aggregation, data joining from multiple topics, and more.

Real-time stream processing – Kafka consumers can also work as real-time stream processors. These can allow the consumers to continuously consume the messages from one or more topics and perform the streaming operations on them such as grouping, windowing, and more.

Scalability – Kafka consumers are designed to be highly scalable and can be deployed and distributed to handle large volumes of messages and achieve high throughput and low latency.

Data synchronization – Lastly, we can use the Kafka consumers for data synchronization between different systems or databases. By consuming messages from Kafka topics, the consumers can update or replicate the data in other systems or databases in real time.

It is, therefore, an understatement to say that the consumers are a significant building block within a Kafka cluster.

It is essential to understand how to configure a basic consumer application in Kafka using the CLI tools. Join us in this tutorial to explore how to do that.

Requirements:

1. Apache Kafka
2. Terminal Access
3. Sudo Permissions

Create a Test Kafka Topic

Before writing the messages to a topic, we must ensure that the topic exists on the server. We can do this using the “kafka-topics.sh” command line utility.

An example command is shown in the following:

The provided command uses the “kafka-topics.sh” utility to create a new a new topic called “users”. We use the –bootstrap-server command to specify the address to the Kafka broker. We also use the –replication-factor argument to set the replication of the topic. Finally, we use the –partition argument to allow us to specify the number of partitions that are assigned to the Kafka topic.

Start Producing the Messages

Once we verify that the topic exists on the server, we can start writing the messages to the topic using the “kafka-producer” utility.

The command syntax is as shown in the following:

If you are running an Apache Kafka 2.4 or less, you need to substitute the –bootstrap-server option with the –broker-list option as shown in the following syntax:

An example demonstration command is as shown in the following:

This command should open an input in the console which allows us to write the messages to the “users” topic.

We can write a new message on each line as shown in the following:

To stop writing the messages to the topic, use the CTRL + C key combination to send a SIGKILL signal to the producer process.

Kafka Consumer Application Using CLI

Once we created a Kafka topic and used a Kafka CLI producer to write some basic messages to the topic, we can proceed and use a basic consumer to read the messages.

To create a basic consumer application using the Kafka CLI tools, use the following command:

The parameters from the previous command are expressed as follows:

<bootstrap_server> – This represents the address to one or more Kafka brokers in the <hostname>:<port> format. You can specify multiple brokers by separating them in commas.

<topic_name> – This specifies the name of the Kafka topic from which we wish to consume the messages.

We can also specify some options to the command to modify the consumer behavior. Some standard options include the following:

–from-beginning – This option tells the consumer to start reading the messages from the beginning of the topic. By default, the consumer only reads the messages that are produced after it starts running.

–max-messages <n> – This option specifies the maximum number of messages that a consumer reads before exiting. By default, the consumer runs indefinitely.

–property <key>=<value> – This parameter allows us to specify the additional properties for the consumer. For example, we can set the deserializer class for the message key and value.

We can use the previous commands to set up a basic Kafka consumer from CLI. For example, to read all the messages from the “users” topic that we created earlier, we can run the following command:

This should read the messages from the beginning of the topic.

Conclusion

You now learned how to configure a basic Kafka consumer application using the CLI tools. After that, you can explore the documentation to learn more.