One key side of making certain a well-managed and maintained a Kafka cluster is monitoring the variety of messages in a subject. By maintaining a tally of the message depend, you may guarantee your system is working easily and detect potential points.
There are a selection of the way to get the message depend for a subject, and we’ll do our greatest to cowl the preferred methods. A few of these strategies embody utilizing Kafka’s built-in command-line instruments, leveraging the Client API, or utilizing third-party instruments. After all, every methodology has its execs and cons, which we’ll additionally briefly check out.
On this article, we’ll talk about varied strategies that will help you get the variety of messages in a Kafka matter with ease.
Utilizing Kafka Command-Line Instruments
Kafka comes bundled with a set of helpful command-line instruments that can be utilized for quite a lot of totally different duties, like creating matters, and managing and monitoring your cluster. These instruments are very handy with regards to retrieving details about your cluster, like message counts. Right here we’ll deal with utilizing these built-in command-line instruments to get the variety of messages in a subject.
To get the variety of messages, you should use the kafka-run-class.sh script together with the kafka.instruments.GetOffsetShell class. The command appears like this:
$ ./bin/kafka-run-class.sh kafka.instruments.GetOffsetShell --broker-list <your_broker_list> --topic <your_topic_name> --time -1
Exchange <your_broker_list> with the listing of your brokers, and <your_topic_name> with the identify of the subject you wish to fetch the message depend for. The --time -1 parameter tells the instrument to fetch the newest offsets for every partition within the matter.
After executing the command, you will see output formatted like this:
<your_topic_name>:<partition_id>:<latest_offset>
For instance, it’d seem like this:
my_topic:0:450
my_topic:1:500
my_topic:2:550
To get the full message depend, it is advisable to add up the <latest_offset> values for every partition. Understand that this methodology exhibits the depend of messages at present accessible within the matter, and any messages which have been eliminated because of retention insurance policies, for instance, is not going to be included.
Utilizing the Kafka Client API
One other method to getting the variety of messages in a Kafka matter is by utilizing the Kafka Client API. The Client API is a programming interface that lets you get messages from matters in varied programming languages. On this part, we’ll present easy methods to use the Client API in Python to depend the messages in a subject. You may adapt this methodology for different languages, reminiscent of Java, as wanted.
First, you will want to put in the kafka-python library if you have not already. You are able to do this utilizing pip:
$ pip set up kafka-python
Subsequent, import the mandatory libraries in your Python script:
from kafka import KafkaConsumer
import time
Subsequent we’ll have to configure the patron properties. Exchange <your_broker_list> with the listing of your brokers and <your_topic_name> with the identify of the subject you wish to fetch the message depend for.
shopper = KafkaConsumer(
'<your_topic_name>',
bootstrap_servers='<your_broker_list>',
auto_offset_reset='earliest',
enable_auto_commit=False,
group_id=None,
)
With the patron arrange, we will now learn messages from the subject:
now = time.time()
message_count = 0
for message in shopper:
message_count += 1
if (message.timestamp / 1000) - now > 60:
break
This code will depend messages for the subsequent 60 seconds earlier than stopping. You may regulate the time interval as wanted.
Lastly, print the message depend to the console:
print(f"Complete message depend: {message_count}")
Now, once you run the script, it should eat messages from the subject and show the full message depend after the given time interval.
Word: Understand that utilizing the Client API to depend messages may not be essentially the most environment friendly methodology, particularly for matters with numerous messages. This methodology truly requires consuming all messages within the matter, which might take loads of time.
Utilizing Kafka Monitoring Instruments
Along with the command-line instruments and Kafka Client API, it’s also possible to use varied monitoring instruments to get the variety of messages in a subject. These instruments are sometimes user-friendly and have superior options that will help you handle clusters. Some well-liked Kafka monitoring instruments embody Confluent Management Heart, CMAK, and Kafdrop. On this part, we’ll stroll you thru the method of acquiring message depend utilizing Confluent Management Heart and CMAK.
For each Confluent Management Heart and CMAK (Cluster Supervisor for Apache Kafka, previously generally known as Kafka Supervisor), you will want to begin by configuring and connecting to your Kafka cluster. Observe the official documentation for every instrument to arrange the connection:
Try our hands-on, sensible information to studying Git, with best-practices, industry-accepted requirements, and included cheat sheet. Cease Googling Git instructions and really be taught it!
As soon as related to your cluster, you may navigate to the subject whose message depend you wish to retrieve.
- In Confluent Management Heart, go to the “Subjects” tab and choose the specified matter.
- In CMAK, select the cluster after which click on on the “Subject Record” hyperlink. Find your matter and click on on it.
After navigating to the subject, you may retrieve the message depend:
- In Confluent Management Heart, test the “Messages” discipline within the “Overview” tab. This discipline ought to present the full variety of messages within the matter.
- In CMAK, search for the “Complete Measurement” discipline within the “Subject Identification” part on the subject’s element web page. This discipline shows the full variety of messages within the matter.
Conclusion
On this article, we checked out a couple of methods to get the variety of messages in a Kafka matter. We began with Kafka’s built-in command-line instruments, in addition to the Client API and different well-liked third occasion instruments like Confluent Management Heart and CMAK.
Finally, the most effective methodology for getting the variety of messages in a subject depends upon your particular use-case and infrastructure. Make sure to weigh the professionals and cons of every method, like efficiency and ease-of-use
