Why Asynchronous Communication

A message queue decouples the producer of a message from its consumers. The producer doesn’t wait for the consumer; the broker stores the message and hands it out. This buys three things:

• Burst resilience — when a service spikes, messages buffer instead of the system falling over.
• Independent scaling — producers and consumers scale separately; add consumers when behind.
• Failure isolation — a consumer crash doesn’t lose work; the message waits and is redelivered.

The price: no synchronous guarantee of completion, and the semantics of delivery become your responsibility.

Point-to-Point vs Publish/Subscribe

If you need exactly one handler per task → queue. If you need every interested system to receive every event → topic. Kafka merges the two: a topic can be consumed by many consumer groups, each getting its own copy — and within a group, partitions are shared (competing consumers).

Kafka: The Append-Only Log at Scale

Kafka’s model is a distributed, ordered, append-only log:

• Topics — named streams of records.
• Partitions — a topic is split into ordered partitions. The order guarantee is per-partition, not per-topic. Producers pick a partition (by key hash, or round-robin) so related events land in the same partition and stay ordered.
• Consumer groups — consumers in a group each own a subset of partitions (never more than one consumer reads a given partition at a time within the group), so scaling = adding consumers up to the partition count.
• Offsets — each consumer tracks its position (offset) per partition. Replaying = resetting the offset. This makes Kafka replayable and, together with a retention window, a durable event store — the foundation for event sourcing.

Because partitions are the unit of both ordering and scaling, partition count is a design decision: too few limits your consumer parallelism, too many adds overhead and rebalance churn.

Delivery Semantics: At-Most-Once to Exactly-Once

How many times a message gets processed is the most consequential queue decision:

At-least-once + idempotent consumers is the industry default for good reason: it’s simple and safe. Exactly-once is a marketing-adjacent phrase that really means “the pipeline as a whole is designed so the effect happens once” — typically via idempotent downstream effects, transactional outboxes, or Kafka’s transactional producer + idempotent sink.

Ordering and Duplicates: The Two Traps

• Ordering — if the contract is “messages for one entity in order,” the producer must key on that entity so it stays in one partition, and the consumer must not parallelize across that partition. Parallelism and strict ordering are in direct tension.
• Duplicates — under at-least-once, a crash between “message delivered” and “offset committed” causes a redelivery. The consumer’s write must be idempotent (dedupe by message ID, or make the effect naturally idempotent — an INSERT ... ON CONFLICT DO NOTHING, a dedupe table).

Stream Processing

Once events live in an append-only log, you can process the stream: join, aggregate, window, and react in real time. Frameworks like Flink, Kafka Streams, and Spark Streaming treat the log as the source of truth and emit derived streams (aggregations, alerts, joins). The key mental shift: the log is the database — the stream of events is authoritative, and state is just an aggregation of the log with a time window.

The Visualizer

Use the message-queue visualizer above to step through three delivery scenarios on a three-partition topic. Compare at-most-once (messages may vanish on consumer failure), at-least-once (redelivery on crash — watch the duplicate), and exactly-once (acknowledgment coordinated so the effect happens once). Watch how consumers map to partitions and how a failed consumer’s partitions get rebalanced.

Practice Trajectory

1. For a job queue vs an order-events broadcast, pick point-to-point or pub/sub and justify.
2. Key 10 order events by order_id into a 3-partition topic and verify every event for one order lands in the same partition.
3. Trace at-least-once: consumer crashes after processing but before offset commit — where does the duplicate come from, and how does idempotency absorb it?
4. Design a consumer group for a 6-partition topic with 2 consumers; then explain what happens when you add a 3rd.
5. Write a Kafka producer that keys by user ID and a consumer that dedupes by message ID; reason about ordering guarantees.

When It’s the Right Tool