Kafka Is Not an Event Store

Every time a team sits down to build an event-sourced system, someone asks the same question: we already run Kafka, and Kafka is an append-only, ordered log, so isn't that exactly what Event Sourcing needs? The resemblance is real, and it is tempting. It is also superficial, and mistaking one for the other tends to get expensive months later, once the system is in production and the workarounds have quietly piled up.

So rather than arguing that Kafka is bad, because it is not, we want to do something more useful. We will first pin down what an event store actually has to do, then hold Kafka against that list and see where it stands. This is the first post in a small series that asks the same question of one candidate at a time. And for Kafka, the answer is the one that surprises people the least once they can see it laid out: it is an excellent tool, just not for this particular job.

The Log That Looks Like an Event Store

Start with Kafka's own vocabulary, because that is the model a Kafka user thinks in. Events are published to topics, each topic is split into a number of partitions, and every event sits at an offset within its partition. Consumers read forward from an offset, events stay around for as long as the configured retention allows, and you can replay a partition by rewinding. Append-only, ordered, replayable: on the surface, this reads like a description of Event Sourcing.

That is exactly why the analogy is so seductive. Event Sourcing also treats an append-only, ordered sequence of events as the ground truth, and it also reconstructs state by reading those events back. Draw both on a whiteboard and you get the same picture: a log you append to and read from. If the shapes match, why buy a second thing?

Because "looks like" is not "is". The whiteboard hides every guarantee that actually matters, and the guarantees are the whole point. The question is not whether Kafka can hold events – anything can hold events – but whether it can hold them the way an event-sourced system depends on. To answer that, we need a yardstick.

What an Event Store Has to Do

Here is what a store has to provide to serve as the backbone of an event-sourced system. This is the list we will hold every candidate against, in this post and in the ones that follow.

• A durable source of truth. Events are not data in transit on their way somewhere else; they are the truth itself, kept for as long as the system lives.
• Append-only, immutable storage. Not by convention or team discipline, but as a guarantee the store enforces, so that no past event can ever be rewritten or quietly dropped.
• Atomic writes across multiple events. A single decision that produces several events is persisted as one indivisible unit, so you never store half of it.
• Optimistic concurrency per stream. The ability to append events only if that stream has not changed since you last read it, which is how invariants survive concurrent writes.
• Two different read patterns. Reading the entire history in order, to build and rebuild read models, and reading a single entity's events cheaply, to reconstruct that one entity's current state.
• Subscriptions as a first-class feature. Observing new events as they arrive, with a seamless transition from replaying the past to following the present.

That is the yardstick. Six capabilities, and an event store has to deliver all of them together, not most of them with effort. Now let us hold Kafka against it, criterion by criterion.

Transport, Not Truth

The deepest mismatch shows up at the very first item on the list. Kafka is built to move events from producers to consumers, and it is superb at that. But retention is a configuration knob, not a promise of permanence. Set it to seven days, or let log compaction collapse a keyed topic down to the latest value per key, and older events are simply gone. You can, of course, configure retention to keep everything indefinitely – but permanence you have to remember to switch on is not the same as permanence by design. A store whose contents can expire by policy is not where you keep your only copy of the truth.

Picture the consequence in concrete terms. A consumer that goes offline for a week comes back to rebuild its read model, only to find that the events it needed have already aged out of retention – the history it was supposed to replay is not there anymore. An event store never puts you in that position, because keeping the full history is its reason to exist, not a setting you can accidentally misconfigure.

There is a subtler point hiding underneath, and it is one of the most essential rules of event-sourced systems: you store the event as the truth first, and publishing it is a separate, downstream concern. In Kafka, those two steps are the same step. Publishing is the write. There is no moment where the event has been made true and you then decide, separately, who gets to hear about it. The truth and the transport collapse into a single act.

That collapse is the root of most of what follows. Kafka is a pipe, and a pipe is a wonderful thing to have – but it is not the place you keep the record that everything else depends on. The source of truth has to live somewhere with stronger promises, and once it does, Kafka's role changes from "the store" to "the thing downstream of the store."

Where Are the Guardrails?

The next requirement Kafka drops is enforcing invariants when you write. The core move in Event Sourcing is small and constant: append this event only if the aggregate is still at the version I last saw. That single check is what stops two concurrent commands from both succeeding and leaving an aggregate in a state the business rules say can never exist – two people booking the last seat, an account overdrawn past its limit, an order shipped twice.

Kafka has no optimistic concurrency control at the granularity you need. The closest workaround is a coarse, pessimistic lock at the partition level – and because a single partition holds many entities, it blocks far more than the one you actually care about, while trading away the throughput that made Kafka attractive in the first place. Optimistic concurrency solves the same problem precisely, at the level of one stream, and it scales because conflicts are the exception rather than the rule.

This is one of the places where a purpose-built store earns its keep. In EventSourcingDB you attach preconditions to a write, and they are evaluated inside the very same transaction that appends your events. That means you can enforce a business or consistency rule declaratively, without external locks and without extra roundtrips, and a violated expectation simply aborts the write. If you would like to see how that feels in practice, the preconditions guide walks through it end to end.

Topics Are Not Streams

Now to the mismatch that trips people up most, because it sits right in the vocabulary. Kafka's unit is the topic, a coarse container split into a limited and deliberately small number of partitions. Event Sourcing's unit is the stream – one per aggregate, one per entity – and a real system has as many streams as it has entities, which can easily be millions. EventSourcingDB calls these fine-grained streams subjects, and reasoning in terms of them is what makes per-entity work natural.

Watch what happens when you try to map one onto the other. Reconstructing a single aggregate means loading exactly one stream. In Kafka, "give me every event for order 4711" leaves you two bad options: put one topic or partition per entity, which does not scale to millions of them, or share a topic across entities and scan it, filtering out everything that is not order 4711, which is linear in the size of the whole topic. Both are wrong answers to the single most common read in an event-sourced system.

And it gets worse when you need the other read pattern at the same time. Projections need the full history in a single, well-defined order. Across partitions, Kafka gives you order within a partition but no global order for free; recovering one means adding your own sequencing logic on top. So the two reads pull against each other: partition by entity and you lose easy global order, keep a shared log and you lose cheap per-entity access. A topic is a coarse pipe; a stream is a fine-grained, addressable unit of truth. They are simply not the same shape.

What You End Up Building

Step back and watch what a team actually does when it insists on Kafka as the store anyway, because the pattern is remarkably consistent. To get concurrency, they add an external database to hold and check version numbers. To read a single entity efficiently, they add an index or a secondary store keyed by entity id. To make sure events are never lost to retention, they add another durable store behind Kafka. And to keep that store and Kafka in agreement, they reach for the outbox pattern.

Piece by piece, they rebuild an event store around Kafka – the very thing they set out to avoid needing. The outbox in particular is a tell. We have argued before that you don't need an outbox when storing an event and publishing it are the same atomic act; the outbox is precisely the workaround you reach for because Kafka splits those two apart. Its presence is a symptom, not a solution.

The result is an architecture with more moving parts, more places to get consistency subtly wrong, and more operational surface than the purpose-built store the team was trying to avoid buying. The complexity you were trying to skip comes back, wearing a different hat – and now it is spread across several systems that all have to agree.

The Right Log for the Right Job

Come back to the question we started with. Kafka is an append-only, ordered log, and Event Sourcing wants an append-only, ordered log, so the confusion is understandable. But the two logs are built for opposite ends of the system. Kafka's log is optimized to move events between services quickly and at scale. An event store's log is optimized to keep events as the truth, protect the invariants as they are written, and hand them back by entity and in order. Those are different jobs, and optimizing hard for one means giving ground on the other.

That is the whole point, stated plainly: a great messaging platform and the best event store are not the same tool, and asking one to be the other is where the trouble starts. Kafka shines exactly where an event store should not be – as the transport between systems, downstream of the store, carrying events that have already been made true. Used there, it is not a compromise; it is the right tool doing what it is best at.

It is worth being fair about the one item on our list Kafka handles gracefully: following a stream of new events as they arrive is its native strength. But even there, it follows partitions, not the fine-grained per-entity streams an event-sourced system reconstructs from. That is the tell in miniature – Kafka fits so naturally on the transport side and so awkwardly on the storage side because every one of its strengths is aimed at moving events, not at keeping them as the truth.

This is the first entry in a series that holds different candidates against the same six-item list. Next, we will turn it around and point it at the tools people reach for from the opposite direction: the general-purpose databases they already trust for everything else. If you are weighing these trade-offs for your own architecture and want to go deeper on Event Sourcing, CQRS, and Domain-Driven Design, we gather that thinking in one place at cqrs.com – a good next stop once you have decided that the log on your whiteboard deserves a store built for it.