In Kafka, a consumer group refers to a set of pre-defined consumer applications working together to consume a specific set of topics. Each consumer in the group contains a unique identifier which is known as a “consumer ID”. This unique “consumer ID” value allows a given consumer application to join a consumer group without providing conflicts with the existing applications.

Hence, when a consumer joins a group, it is assigned with a “group ID” which identifies the group that it has joined, and the “consumer ID” which identifies the consumer within the group. These two values help the cluster to keep track of the available consumer applications, what group they belong to, and what data they hold.

We can specify the “group ID” when creating the consumer application instance. In the case of a “consumer ID”, this value is automatically generated by the Kafka broker once the consumer joins a given group. Once a consumer instance is restarted, Kakfa assigns it a new “consumer ID” value during the rejoin with the group. This allows the consumer to proceed and resume the message consumption at the last offset.

In this basic tutorial, we will explore how we can create a consumer group and set the group ID to allow the consumers to join the group.

Create a Consumer Group

We can create a consumer group by adding multiple instances of the Kafka consumer application. We can then use the consumer groups to subscribe to a similar topic. To set the “group ID” of the consumer group, we can edit the configuration file for the consumer and set the “group.id” property.

You can edit the config/consumer.properties file and set the “group.id” value as shown in the following entry:

Once you set the consumer group ID, you can ensure that the consumer application joins the group by setting the “group ID” as a property. However, using the Kafka console consumer application, you can use the following command:

Keep in mind that for console consumers, the consumer group is assigned automatically with the cons0le-consumer prefix followed with the suffix of the PID. However, you can specify the “group ID” with the –group option.

Set the Consumer Group ID as a Property

We can set the properties of the consumer application, including the “group ID” of the consumer group that we wish to join.

Although this varies depending on your target programming language and library, the following shows a fundamental example of how to set the “group ID” in a Java-based consumer application:

In this case, we use the properties.put(“group.id”) and the ID of the group that we wish to join. Once we run the application, the consumer joins the same group and have the messages distributed among the consumers in the app.

Conclusion

You came across the methods of creating a consumer group in Apache Kafka that involves creating multiple instances of a Kafka Consumer and subscribing to the same topics.

You also learned how to specify the group ID, either using the “group. id” configuration property when creating the consumer instance or by setting it as a property in your consumer application. Using the consumer groups, you can ensure that the messages are evenly distributed among the consumers in the group, allowing you to process large volumes of data in parallel.