![publish with kafka tool publish with kafka tool](https://res.cloudinary.com/practicaldev/image/fetch/s--cJDkEEjG--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://i1.wp.com/www.idata.co.il/wp-content/uploads/kafka-administration-and-monitoring-free-graphical-interface-tools/kafkatool.jpg)
In a nutshell, one Kafkacat instance reads messages from a Kafka topic, then the messages are transformed into the required format and fed to another kafkacat instance, which writes them to another topic.
PUBLISH WITH KAFKA TOOL CODE
The code is taken from the blog of one of the Kafka ecosystem’s leading promoters - Robin Moffatt.
![publish with kafka tool publish with kafka tool](https://www.mrjamiebowman.com/wp-content/uploads/image-135.png)
I will describe some of the most frequently used ones, and at the end, we will analyze in more detail what, in our opinion, is close to ideal. I would like to point out right away that “on the market” here is not in the sense of “for money”, all the tools discussed below are free. There are a number of solutions on the market that help make the whole process easier. In contrast to the databases like MySQL or MSSQL which come with GUI client applications, Kafka doesn't treat us with such tricks and only offers console utilities with a rather narrow (at first glance) functionality.īut there is good news as well. For example, if we want to solve the problem of the player with id=42, we need to find all messages where he is mentioned (playerId: 42), line them up in a chain, and then see at what point everything went wrong. It turns out that in order to find the right message, you have to read the pack and find among them those that are interesting.
![publish with kafka tool publish with kafka tool](https://i.stack.imgur.com/ROax0.png)
It is also possible to specify a time as the starting point, but then you can still only read all the messages in order. We need to know the offset (for simplicity this is the order number in the topic) from which we want the messages. However, this gets complicated by the fact that you can read only sequentially from Kafka. That means a lot of things, but what we find interesting is that messages are not deleted from threads after the recipients have read them, and technically you can read them again at any time and see what's inside. To begin with, Kafka is not just a message broker, as many people think and use it, but a distributed log. But, as usual, the details are the most interesting part. From a remote perspective, the solution seems to be simple: you need to get the relevant messages from Kafka and see what is wrong with them. This is especially interesting in cases when something does not go according to plan: it is important to understand at what stage everything treads awry. These services are owned by different teams, and it is very useful to see what is contained in a particular message. From input to output, a message can go through a dozen services that filter and transform it, transferring it from one topic to another. My name is Sergey Kalinets, I am an architect at Parimatch Tech, and in this article, I want to share our experience in the field of message search in Kafka.įor our company, Kafka is the central nervous system through which microservices exchange information.