Event Sourcing was coined around 2004 by Greg Young and Martin Fowler, while Durable Execution is somewhat a new kid in the block – firstly used around 2022 by Maxim Fateev, although the concept itself is a bit older and known by different names (workflow orchestration, reliable execution, etc.). Older than either, and quietly holding up much of the infrastructure we lean on every day, are Consensus Protocols, for which we owe Leslie Lamport a lot – most famously Paxos, conceived in the late 1980s and published in 1998. While marketed to different audiences and used to solve different kinds of problems, all three employ the same underlying mechanism: a Distributed Log. In this article, we dive into the details of the Distributed Log and how it empowers Consensus, Event Sourcing, and Durable Execution.

Append-Only Log – what is it?

Let’s start with the boring part, because the boring part is the whole trick. A log is perhaps the simplest data structure there is: an ordered, append-only sequence of records. You add to the end, and you read from wherever you like towards the end. That’s it. No in-place updates, no deletions, no clever tree balancing. Jay Kreps wrote a now-famous piece arguing that you can’t really understand databases, replication, consensus, or version control without first understanding this humble thing. I tend to agree.

It looks like this:

Fig. 1: An append-only log. Records are added at the tail, and each one gets a unique, increasing index.

Each record gets a unique, monotonically increasing index (you’ll also hear offset, sequence number, log entry number – same idea). The index does double duty: it identifies the record, and it defines a notion of time. Everything to the left happened before everything to the right. We didn’t have to invent a clock to get ordering – the position in the log is the ordering.

Two properties make this little structure so powerful. Let’s take them one at a time.

Immutability

Once a record is written, it never changes. We don’t update it, we don’t reach back into the middle of the log and tweak record #2 because we changed our mind, and we don’t quietly delete it later when it becomes inconvenient. The log only ever knows facts, and facts don’t get edited or erased – new facts get appended.

The log is the source of truth. Everything else (the current state, the views, the projections) is derived from it.

This is the part people underestimate. If the log holds the complete, ordered history of what happened, then any state you care about is just a function of that history. The current state is a fold over the records: you replay the entries one by one through a deterministic apply function, threading the accumulated state through as you go.

Lose the state? Replay the log, and you get it back, bit for bit, provided the apply function is deterministic. Hold onto that determinism requirement – it is going to follow us around for the rest of the article like a stray dog.

Immutability is also why people get the urge to rewrite history (and why they can’t). I wrote about that temptation in Bi-Temporal Event Sourcing, who cares? – the short version is that you don’t delete the wrong fact, you append a correcting one. The timeline only grows forward.

Append-Only nature

The second property is that all writes happen at one place: the tail. This sounds like a limitation, and it is, but it’s the kind of limitation that gives back more than it takes.

• Writes are cheap. Appending to the end means sequential I/O. Disks (even the spinning ones we like to pretend no longer exist) are dramatically happier writing sequentially than seeking all over the place. No random writes, no in-place mutation, no fragmentation drama.
• Ordering is free. Because there is exactly one place to write, the order in which records arrive is the order in which they are stored. I made this argument in detail in Ordering in Event-Sourced Systems: we cannot trust timestamps across machines, so we simply accept the order of receipt as THE order. The tail is where that decision gets made.
• The tail is the single point of synchronization. Concurrency in a log is almost embarrassingly simple. There is one contended location – the end – and that’s where you detect and resolve conflicting appends. Everywhere else, there is nothing to contend over, because nothing else ever changes.
• Readers carry their own cursor. A reader just remembers an index and asks for “everything after this.” Many readers, each at their own pace, none of them stepping on each other or on the writer. The writer doesn’t even need to know they exist.

There’s a subtlety hiding in that last point, though. A reader sometimes wants a slice: the stream of a single command/view model, one key, everything tagged a certain way. Scanning the whole log and discarding most of it would be silly, so we build indices over the log – secondary structures that map a stream id, a key, or a tag to the matching positions, giving a different view that carves the one stream into substreams (one command model’s history, one key, one tag). This is also what makes Dynamic Consistency Boundaries practical: instead of nailing the consistency boundary to a fixed aggregate, we define it dynamically as the set of events matching some query, and we index events by their tags – so that both reading the relevant slice and checking for conflicting appends at the tail stay fast, with no full scan required.

Distributed Implementations

A single log on a single disk is wonderful right up until that disk (or the machine holding it) decides to take the rest of the week off. Surviving in the wild with a single replica is maybe too brave. So we replicate: every node in the cluster keeps a copy of the log.

And here is where the easy part ends and the interesting part begins. With one writer and one disk, the order was trivial – whatever order the records arrived in. With many nodes, we have a new question:

How do all the nodes agree that record #3 is record #3, and not record #4 on some other node that saw things in a different order?

This is no longer a storage problem. It is an agreement problem. We need every node to converge on the exact same sequence of records, despite messages arriving out of order, nodes crashing mid-write, and the network misbehaving.

So who does the agreeing?

Append-Only log at the heart of Consensus Protocols

This is my favorite part, because it’s where the abstraction shows its hand. Consensus protocols – Raft, Multi-Paxos, and friends – are, at their core, machinery for agreeing on the contents of a replicated log. Raft doesn’t even hide it – the central data structure in the paper is literally called the replicated log.

How does this work?

A single leader accepts, appends, and replicates each entry to the followers. An entry is considered committed once a quorum (a majority) of nodes has durably stored it. Once committed, every node applies the entry, in log order, to its local state machine. If every node runs the same deterministic state machine and feeds it the same entries in the same order, every node ends up in the same state. Kreps calls this the State Machine Replication principle, and it’s the quiet engine under an enormous amount of infrastructure. If you zoom into the internals of databases, you’ll see that at the very core is a log (which some of them call CDC – Change Data Capture) which is replicated across the cluster. There are even databases that provide read access to this log. Another important point to note is that not all databases use Consensus to replicate the log.

Compaction

Now for the uncomfortable question the append-only, immutable, never-delete log invites: if we only ever add, doesn’t it grow forever?

Yes. Storage costs money, and replaying from the dawn of time to rebuild state only gets slower. The solution is Compaction.

Periodically, we snapshot the state of the state machine up to a certain log index N. Up to N, we can safely trim the log because we’ll derive the current state of the state machine by loading the snapshot and replaying the log from N onward.

So: can an append-only log really be append-only forever? When the history is the point, yes!

Event Sourcing in a Nutshell

With the log understood, Event Sourcing is almost an anticlimax – in the best way.

The traditional approach stores the current state and mutates it in place. A bank account row shows balance = 100, you withdraw 30, and now the row shows balance = 70. The old value is gone, and the why is gone. Event Sourcing refuses to throw that away. Instead of storing the balance, you store the events that produced it:

AccountOpened(initial = 0)
MoneyDeposited(amount = 100)
MoneyWithdrawn(amount = 30)

An account modeled as a stream of events rather than a mutable balance.

The current state is, once again, a left fold over those events. Want the balance? Replay the stream and apply each event. Want to know how you got there? It’s right there in front of you, in order.

Where does this live? In an Event Store – which, if you’ve been paying attention, is just an append-only, immutable, ordered log specialized for domain events. It carries total ordering, it accepts appends at the tail (which, by the way, the event-sourcing world perversely calls the head of the stream (weird, huh?)) and it detects conflicting appends right there at the tail.

And because the truth is the log rather than some particular shape of state, you get things that are awkward-to-impossible with mutate-in-place storage almost for free:

• A complete audit trail, by construction. You didn’t add logging; the facts are in the storage.
• Temporal queries – “what did this look like last Tuesday?” – because you can fold up to any index.
• Multiple read models from one source of truth (hello, CQRS). Different consumers, different cursors, different projections – exactly the decoupled readers the log gave us.
• Debugging by replay. Reproduce a bug by replaying the events that caused it.

When streams get long, you snapshot the command model’s derived state and replay from there – the very compaction story from above. Do note that the history in the event store is not affected!

Event Sourcing, then, is not a new mechanism. It’s the distributed log, pointed at a business domain.

Durable Execution – Event Sourcing in disguise

Which brings us to the new kid. Durable Execution sells a genuinely magical-sounding promise: write your business process as ordinary code – functions, loops, if statements, even a sleep for thirty days – and the runtime guarantees it runs to completion. Worker crashes, deploys, restarts, a process that outlives the machine it started on – none of it matters. The function just… finishes.

How? Take a guess.

Every meaningful step the workflow takes – invoking an activity, starting a timer, receiving a signal – and, critically, its result, gets appended to a log. Temporal calls it the Event History, Restate calls it a Journal, and the shape is the same across the field. Append-only, immutable, ordered. Sound familiar?

Now the clever bit: replay. When a worker picks up a workflow after a crash – possibly on an entirely different machine – it does not restore memory from a snapshot. It runs your workflow code again from the very beginning. But this time, whenever the code reaches a step that already happened, the runtime doesn’t re-execute the side effect – it feeds back the recorded result from the log. The code marches deterministically back through the exact same branches, rebuilding its local variables and its position, until it catches up to where it crashed. Then it carries on as if nothing happened.

Read that again and line it up with what we’ve been doing all article:

The workflow’s state – where it is in the code, what its variables hold – is a fold over an append-only log of step results. Recovery is replay. The program is a command model; the recorded step results are its events.

It’s Event Sourcing. The “command model” just happens to be the execution state of your program rather than a bank account, and the “events” are the outcomes of the steps rather than MoneyWithdrawn.

And once you see that, workflows turn out to be a wonderful fit for event sourcing – reliable business processes want an audit trail, crash recovery, and room to scale, which is exactly the bundle an event-sourced log hands you for free. Axoniq’s Workflow Engine (in preview at the time of writing) leans into this directly: the steps a workflow takes are themselves recorded as events, in the same event store (the same log) that already holds your business events. There’s no separate sidecar journal for the workflow; domain facts and workflow progress live in one place.

That shared event store is exactly what lets you wire the two together: a workflow can suspend on a business event and resume once the event is appended, so publishing an event can both update read models and unblock workflows waiting on it. Since workflow events are part of the event store, you can build various read models based on them – how many workflows completed successfully? How many are stuck on this step? etc.

Recap

Strip away the marketing and three very different-looking technologies collapse onto one substrate:

• Consensus protocols use a replicated log to agree on an order, then fold it into a replicated state machine.
• Event Sourcing uses an append-only log of domain facts, then folds it into the command model state and any number of read models.
• Durable Execution uses an append-only log of step results, then folds it (by re-running your code) back into a live, resumable program.

The principle behind all of them is the same: an append-only log. Each and every technology described uses the characteristics of a log: immutability, append-only nature, determinism, while applying log entries (although this is not the log property per se).