Apache Kafka

Apache Kafka

What is it?

Apache Kafka is an open-source event streaming platform that can transport huge volumes of data at very low latency.

Companies like LinkedIn, Uber, and Netflix use Kafka to process trillions of events and petabtyes of data each day.

Kafka was originally developed at LinkedIn, to help handle their real-time data feeds. It’s now maintained by the Apache Software Foundation, and is widely adopted in industry (being used by 80% of Fortune 100 companies).

Learning How to Learn Kafka

The complexity of Kafka can feel overwhelming, especially if you’re a beginner and doesn’t know where to start. An effective approach is a breadth-first learning strategy, starting with foundational concepts and expanding into practical applications. This way, you’ll cover key areas first, gaining enough understanding to tackle your specific role needs and dive deeper as you go.

Core Kafka Concepts

The essential components of Kafka include topics, partitions, offsets, producers, consumers, consumer groups, and brokers. In:

  • 1.3. Kafka Quickstart: Let your hand dirty by deploying Kafka on your local using Docker Compose.

Choosing Your Path

Developer Path

If you’re in a development role, your goal is to use Kafka for applications, potentially handling use cases like:

  1. Ride-Hailing Services (e.g., Uber): Use Kafka to track real-time vehicle locations, ride requests, and status updates. A producer publishes location and status messages, and consumers subscribe to receive updates as they happen.
  2. Order Management (e.g., Shopee): Kafka can process real-time order data, managing inventory updates and customer notifications. By publishing order events to topics, you can enable different services to update customer order statuses without blocking each other.
  3. Fraud Detection (e.g., Stripe): Kafka enables data aggregation from various sources, which can be analyzed in real time for fraud detection. Events generated by different services—like user login and transaction events—can be consumed and analyzed in real-time, helping identify unusual patterns.

In a development context, your focus is on how Kafka can provide reliable, real-time data streams to power your application’s business logic. Start by building out these use cases with Kafka, implementing topics, partitions, producers, consumers, and consumer groups. You’ll get comfortable handling data flow within Kafka, enabling you to support more complex workflows as your understanding grows.

Operations Path

In operations, you’ll be managing Kafka’s infrastructure to ensure it remains fast, reliable, and scalable. The focus will be on:

  • Broker Health: Ensuring brokers are up and healthy, with replication set up correctly for fault tolerance.
  • Monitoring and Scaling: Monitoring Kafka metrics (e.g., lag in consumer groups, partition distribution) and scaling appropriately.
  • Data Retention Policies: Setting up retention policies to balance disk usage and performance, ensuring that data does not grow indefinitely and consume resources unnecessarily.
  • Migration and Upgrading: Learn how to remove and adding new broker.
  • Security and Access Control: Managing access and securing Kafka, ensuring only authorized producers and consumers can interact with topics.

For operations, after understanding Kafka’s core components, focus on Kafka’s distributed nature and how to monitor and scale Kafka effectively. Explore best practices for configuring Kafka, setting retention policies, and enabling security controls. You’ll also want to dive into tooling for monitoring Kafka, like Prometheus, Grafana, or Kafka Manager.

Going Deeper: Kafka Internals

Once you’re comfortable with the fundamentals, you can dive into Kafka’s inner workings. Here are a few advanced topics to explore:

Storage Mechanisms

Kafka stores topics in the filesystem with a log-segmented storage model, dividing data into smaller files for better performance and easier management. Understanding the internal file structure and how logs are managed can be beneficial for troubleshooting data issues and optimizing storage.

TCP Protocol and Serialization

Kafka uses TCP for communication, and producers serialize messages to byte arrays before sending them to brokers. On the consumer side, these byte arrays are deserialized back into objects. Understanding the serialization-deserialization process is key if you’re optimizing data transfer or debugging message formats.

Replication and Fault Tolerance

Kafka uses replication to ensure data availability. Each partition can have multiple replicas, stored across brokers to protect against failures. Understanding how Kafka’s leader-follower model works will help you manage and monitor replica health and ensure your data’s reliability.

Learning Kafka with a breadth-first approach involves starting with its core components and applying them to real-world use cases. Once you have a solid grasp of Kafka’s main concepts, focus on topics specific to your role—whether it’s developing applications or managing Kafka infrastructure. Finally, as you gain experience, delve into Kafka’s internals to understand the underlying mechanics. With this step-by-step approach, you’ll be able to make the most of Kafka’s capabilities effectively and efficiently.