Inorderly 0.2 Orders Events Straight From Kafka, NATS and Redis Streams

The events that need ordering mostly don’t live in files. They sit in a Kafka topic, a NATS stream or a Redis stream, and until now getting them through Inorderly meant a script in front of it and another one behind. That works until the run gets killed half way, and then nobody can say which lines made it out.

Inorderly 0.2 talks to the brokers itself. -in and -out take a URL as well as a file: kafka://, nats:// or redis://, with TLS and passwords. The clients are written against Go’s standard library, so it’s still one binary with no dependencies, and there are no consumer groups to set up. Inorderly keeps where it got to in each input in its own checkpoint.

inorderly -in kafka://kafka-1:9092/payments -in redis://cache:6379/orders \
  -out nats://nats:4222/ordered -window 500 -state run.state

Killed Half Way, With Brokers on Both Ends

A file can be cut back to the last checkpoint after a crash. A Kafka topic can’t. So every checkpoint also notes how far the output had got, in the broker’s own terms. Started again, Inorderly reads back every line it had written past that point, checks that the positions follow on with nothing missing or repeated, and checks each line against the one it makes now. Then it carries on without writing those lines a second time. When a run ends, it reads back what it wrote and checks it the same way.

The tests kill a run with SIGKILL ten times, at random moments, with a broker on the input side and on the output side, then let it finish. On Redis, Valkey, NATS and Kafka alike, the output ends up byte for byte what a run that was never interrupted writes. Other tests plant a gap, a stranger’s line or an emptied stream in the output between runs. Each time the restarted run stops and says what it found, rather than write around it.

The independent review before this release is why those checks exist. Its first round found that a Kafka write retried after a timeout could land twice, that a gap left by a failed write could go unnoticed after a restart, and that a NATS hiccup could end a read early. Its second round found that an output emptied between runs slipped through. All four are fixed, and each has a test.

With -follow, a run keeps reading as messages come and never flushes. SIGTERM writes a checkpoint and stops it, and the next run carries on with whatever was still being held.

One Slow Producer

A single window for every producer means one slow source forces a wide window on everyone. -window-for sensor-7=60000 gives that one producer a window of its own. Its events are late only past that, while the others keep the tight bound. Release still waits for the widest window, which is what keeps the output the same whatever the arrival order. That claim got the same treatment as the rest: 200 random event sets, 20 arrival orders each, with snapshots taken and restored along the way, one output per set.

Late events can also go to a file of their own now. -late late.ndjson writes each one as a plain event line, ready to feed into another run with a wider window, or into whatever reconciles them.

The download page has the binaries, and the reference has the URL forms and the limits. The code is on GitHub.