Making a Ledger, a Billing System and an Audit Agree on Every Balance

Two systems that read the same events should end up in the same state. They often don’t, and the reason is rarely the code. It’s the order. A withdrawal that comes before a deposit gets declined; the same withdrawal after it goes through. If the ledger and billing see those two in a different order, they disagree on the balance, and both are right about what they saw.

This worked example has four branches and 300 accounts. Over four hours they produce 100,929 transactions: deposits, withdrawals that are declined when the balance is short, and 929 reversals that undo an earlier transaction and name it as their parent. The branches’ clocks are up to 0.8 seconds apart, so 77 reversals are stamped before the transaction they undo. The ledger and billing each get the transactions through their own queue, with their own delays. Later an auditor replays the four branch logs. Those were exported at least once, so 2,510 of their lines are there twice.

Same Rules, Different Books

Both systems run the same account logic. They take transactions as they come. At the end, 107 accounts have different balances in the ledger and in billing. The ledger declined 4,010 withdrawals and billing 4,005.

Most of the damage is from reversals. Billing’s queue is slower and more uneven, and 107 times a reversal reached it before the transaction it undoes. With nothing to undo yet, the reversal did nothing, and the original went through afterwards as if it had never been reversed. The ledger had the same problem once. Sorting by timestamp wouldn’t have saved either of them: 77 reversals are stamped early, so a sort puts them first too.

Same Sequence, Same Books

Put Inorderly in front of each system, with a 10-minute window:

inorderly -window 600000 -in ledger.ndjson -out ledger.out.ndjson -rejects ledger.rejects.ndjson
inorderly -window 600000 -in billing.ndjson -out billing.out.ndjson -rejects billing.rejects.ndjson

Both give the same sequence. Every reversal comes straight after the transaction it names, including the 77 with early clocks. The ledger and billing run their logic on it and end with the same balance on all 300 accounts and the same 4,019 declined withdrawals. A SHA-256 of every balance matches across both.

The Audit Gets the Same Hash

The auditor doesn’t have either queue, only the four branch logs, each in its own branch’s order and with the export’s repeated lines left in:

inorderly -window 600000 -in branch-1.ndjson -in branch-2.ndjson -in branch-3.ndjson -in branch-4.ndjson -out audit.out.ndjson -rejects audit.rejects.ndjson

The 2,510 repeats are dropped, and the output is the same sequence the ledger and billing used, byte for byte. All three print one replay hash: sha256:8fb91786…b151. That one line is the audit evidence. If the auditor’s hash matches the one the ledger recorded on the day, the ledger processed exactly these transactions in exactly this order, and anyone can rerun it to check.

Choosing the Window

The window has to cover the slowest path in. Here the live queues are seconds apart, and the branch logs, read a line from each in turn, drift further apart in time, because the branches don’t do equal business. Ten minutes covers both. Anything later is refused and listed in the rejects file, so a late transaction never changes the order of ones already booked. It gets dealt with on its own, with a record of why.

Run It Yourself

The example is in the Inorderly repository: go run ./examples/usecases -dir uc audit writes the queue files and the branch logs and checks everything above. The numbers come from the recorded run. Deterministic event ordering explained has the background, and the other worked examples cover retried webhooks, two redundant market data feeds and sensor telemetry.