Kafka is cropping up everywhere in the enterprise, typically for log distribution or at the core of an application made up of loosely coupled services. The concept is very simple; a process pushes a message to Kafka topic (think of a message queue), and other processes pull that message from the topic.
As a network engineer, why should you care about Kafka? There are two good reasons:
If you work in an enterprise, it's likely that very soon the syslog data coming from switches, routers, etc. will be sent to their destination via Kafka
Many modern business applications are built from loosely coupled microservices that communicate via Kafka; when that doesn't work or goes slow, the app owner may turn to you for answers
The good news is that the concepts and network protocol are quite straightforward.
In this first video we look at the concepts and then see Kafka in action. In the next video, we'll take a look at the packets.
Author: Paul is Head of Site Reliability Engineering at a financial services company. He started in the IT industry in 1977 as a hardware engineer, moved to software engineering and then to network management before founding Advance7, a performance and stability consultancy. Paul is an amateur coder and has authored some Wireshark plugins. This is a personal blog and in no way represents the stance, opinion or strategy of his employer.