Pages on Inorderly.com

About

Mon, 01 Jan 0001 00:00:00 +0000

inorderly.com is the home of Inorderly, a small Go engine for deterministic event ordering. It takes events that arrive late, twice or out of order and releases them as one sequence. Give it the same events again, in any order and with any repeats, and it writes the same sequence, byte for byte.

Why It Exists

Events rarely arrive the way they happened. A webhook provider retries, so an order shows up twice. Two services stamp events with clocks that disagree, so a payment looks older than the order it pays for. A feed handler sees tick 88 before tick 87. Most teams patch this with a dedupe table, a sort buffer and a timeout, and the patch mostly works.

Contact

Mon, 01 Jan 0001 00:00:00 +0000

inorderly.com would like to hear from:

Anyone trying Inorderly out. What worked, what got in the way, and what you used before.
Anyone who found a bug. Above all, two runs over the same events that gave different output.
Anyone with an idea for something Inorderly should do next.

Write to info@inorderly.com.

Bug reports are also welcome on GitHub. It helps to include what inorderly version prints, your operating system, the flags you used, and a few made-up events that show the problem. Please don’t send real customer data.

Download Inorderly

Mon, 01 Jan 0001 00:00:00 +0000

Each archive below holds the inorderly command with the license files and README. There’s nothing else to install. The links always point to the latest release.

System	Download
Linux, x86-64	inorderly_linux_amd64.tar.gz
Linux, ARM64	inorderly_linux_arm64.tar.gz
macOS, Apple silicon	inorderly_darwin_arm64.tar.gz
macOS, Intel	inorderly_darwin_amd64.tar.gz
Windows, x86-64	inorderly_windows_amd64.zip

Checksums are in SHA256SUMS. Every release is listed on GitHub.

Install

On Linux or macOS:

tar -xzf inorderly_linux_amd64.tar.gz
sudo mv inorderly /usr/local/bin/
inorderly version

The binaries are pure Go and need no system libraries. The macOS ones aren’t signed by Apple, and macOS may ask you to allow them the first time. You can also clear the flag yourself:

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

This takes about five minutes. You’ll feed the same events to Inorderly in different orders, with repeats, and see the same output and the same hash come back every time. You need the inorderly command from the download page.

1. Write Some Events

Inorderly reads one JSON event per line. Save these five lines as events.ndjson. They’re an order, its payment and its shipment from three services, a second order, and the first order again, the way a retried webhook would send it:

Reference

Mon, 01 Jan 0001 00:00:00 +0000

The command, its files and the Go packages, for Inorderly 0.1. The contract on GitHub has the exact ordering rules.

The Command

inorderly [flags]           read NDJSON events, write them in order
inorderly version           print the version as JSON
inorderly demo [flags]      run the self-checking demo and print its report

Flags

Flag	Default	Meaning
`-in FILE`	standard input	An input file. Repeat it to read several files a line at a time in turn: one line from the first, one from the second, and so on. `-` is standard input.
`-out FILE`	standard output	The ordered output.
`-window N`	0	How far, in event-time units, an event may arrive behind the newest time seen.
`-hold N`	the window	How long past its own time an event waits for a parent.
`-remember N`	0 (always)	How long behind the watermark released events are remembered, for duplicates and parents.
`-rejects FILE`	none	Late, conflicting and invalid lines, one JSON object each.
`-state DIR`	none	Checkpoints. A run killed part way carries on from the last checkpoint when started again with the same flags. Needs `-in` files and an `-out` file.
`-checkpoint-every N`	10000	Input lines between checkpoints.
`-stats FILE`	standard error	Where the summary goes.
`-strict`	off	Exit with status 3 if any line was late, conflicting or invalid.

Input

One event per line, as JSON. Blank lines are skipped. Lines may end in \n or \r\n, and the last line needs no line ending. A line longer than 16 MiB is invalid. Unknown fields, a missing producer or time, a fraction where an integer belongs, a key repeated in one data object, or anything after the event on the same line make the line invalid.