How Inorderly Was Tested With Shuffled Arrivals and Killed Runs

Inorderly makes one promise above the rest: the same events give the same output, byte for byte, however they arrived. That’s an easy thing to say and an easy thing to get slightly wrong, so most of the tests do nothing but try to break it. They shuffle events, repeat them, cut the stream with snapshots, and kill the process in the middle of a run.

Every result below comes from the files in test/results, recorded from version 0.1.1 on a two-core 2.1 GHz Xeon cloud machine running Linux, with Go 1.24. The same tests run on every push, on Linux, macOS and Windows.

Thousands of Shuffles

The main test is a search. It makes a small, deliberately messy set of events: shared timestamps, logical counters, parents that come later, parents that never come, parents in cycles, events that name themselves, and a mix of ids and sequence numbers. It orders the set once in time order. Then it feeds the same set in 20 other arrival orders, with every event sent up to three times, and with snapshots taken and restored at random points along the way. Every order has to give the same hash.

The recorded run did that for 10,000 sets, which is 200,000 arrival orders. One output per set, every time.

The search earned its place early. In its first few hundred sets it caught a case where an event held a long time could find its parent already forgotten under the memory limit, so the list of missing parents depended on timing. The fix keeps any released event that a held event might still name. After that, the long search found nothing.

Killed Thirty Times

The crash test doesn’t fake a crash. It writes 240,351 lines across two files, including 300 late events and one line of garbage, and runs inorderly on them with checkpoints. Then it kills the process with SIGKILL, with no chance to clean up, at a random moment after each new checkpoint, and starts it again with the same flags. It does that 30 times and lets the 31st run finish.

The output file, the rejects file and the replay hash all match a run that was never interrupted. That works because every checkpoint writes the output to disk before recording how far it got, and a restart cuts the files back to that point and writes the rest again, the same. The release smoke check does a shorter version on every binary, including on Windows, where the kill is TerminateProcess.

This test has found a real bug. Recording the results again on a busy machine, one kill landed in a short gap: after the run had written a compacted copy of its record log, but before the checkpoint that names it. The restarted run took that file for a leftover, then wrote its own next log under the same name and deleted it at the following checkpoint, so the next restart refused to carry on. Nothing wrong was ever written; the run stopped instead. Version 0.1.1 fixes it, and a new test walks through that exact sequence on every push.

The Demo

inorderly demo is the same idea at a bigger size, and anyone can run it. The recorded run made 100,000 events and fed them in 20 shuffled orders, each event sent one to five times: just over six million arrivals, four million of them repeats, and one output. A run that saved and restored its whole state twelve times along the way matched it too.

It also checks the parts of the promise one by one:

• Parents first. 13,916 events named a parent. All of them came out after it, including the 3,919 whose clocks said they were older than their parent.
• Orphans reported. 1,044 events named a parent that never came. Every one was reported with the missing parent’s name.
• Late events refused. 1,000 events were added too late to fit. All 1,000 were refused, and the output of everything else didn’t change by a byte.
• Two feeds merged. Two copies of a 100,000-tick feed each lost about 5% of their ticks at random. Merged, every tick that reached either copy came out once, in sequence order, and the 247 ticks lost from both were reported as exactly 247 gaps.

Another Language

A producer written in Python, with its own JSON encoder, its own key order and its own escapes, writes 20,000 sensor readings in four shuffled orders with repeats. All four give the same output as the sorted one. That’s the canonical form doing its job: the same event written by two different encoders is the same bytes.

A Second Pair of Eyes

Before release, an independent review went through the code looking for any way to make the output depend on arrival order. It wrote its own fuzzer, which picks arrival orders differently from the tests and jumps the watermark as far as it can, and ran 20,000 more rounds with no mismatch. It did find other things to fix, and they’re fixed:

• identities that weren’t valid UTF-8 could break a snapshot;
• many events waiting on one missing parent took time that grew with the square of their number;
• a long run of events, each with a clock behind its parent’s, could make sort keys grow without limit;
• a restart read the whole history of released events.

Each now has a test that would catch it coming back.

Speed

On that same two-core machine, one Orderer took about 344,000 arriving events a second, two-thirds of them repeats, and wrote about 115,000 ordered events a second. That’s 2.9 microseconds per arriving event, including turning its JSON into the canonical form. Shared cloud machines vary from day to day; the recording for 0.1.0, on the same kind of machine, measured about 428,000. Your numbers will depend on your machine, your events, your window and how many parents they name.

Four Worked Examples

The recording also runs four realistic examples, each delivered in two or three arrival orders and set beside the consumer people usually write by hand: retried webhooks, two lossy market data feeds, sensor telemetry and a ledger audit. Each gives one output in every arrival order, through the library and again through the command.