Ordering Retried Webhooks So Payments Never Come Before Their Orders

A shop takes its events as webhooks. Checkout sends the order. Payments sends the payment, and later a refund if there is one. The warehouse sends the shipment. Three services, three clocks, one provider in the middle that retries any delivery it isn’t sure about.

This worked example runs 20,000 orders through that setup. Every order gets a payment that names the order as its parent, and then either a shipment or, for about one order in twelve, a refund. The payments service’s clock runs up to a second and a half behind checkout’s, which is plenty: 3,487 payments are stamped before the order they pay for. The provider retries about 15% of deliveries, some of them twice, up to a minute and a half later. The same 60,000 events are then delivered three times, each time with different network delays and different retries.

What the Usual Consumer Does

The simplest consumer handles each webhook as it lands: add the payment to the day’s takings, subtract the refund. On the first delivery run that counts $350,006.23 too much, because 10,920 deliveries were retries. It also meets 205 events before the thing they belong to, a payment before its order or a shipment before its payment, which in real code means a lookup that fails and a retry queue of its own.

So the consumer gets a dedupe table and a sort. Drop anything with an id already seen, then sort by timestamp. The totals are right now. The order still isn’t. Sorting by time puts all 3,487 early-stamped payments in front of their orders, because the timestamps say so. The clock is wrong, and the sort believes it.

What Inorderly Does

The events go in with a 60-second window:

inorderly -window 60000 -in deliveries-1.ndjson -out deliveries-1.out.ndjson -rejects deliveries-1.rejects.ndjson

All 60,000 come out once each. The 10,920 retries in the first run are dropped as repeats, and so are the 10,678 and 10,793 in the other two. Revenue from the output is $1,986,777.90 in all three, which is what was actually paid less what was refunded. Every payment comes straight after its order, every refund and shipment after its payment. That includes the 3,487 payments whose clocks said they came first. Each one names its order as a parent, so it waits for it.

The three runs give one output, and one replay hash: sha256:357e625b…f610a. That’s the part a dedupe table and a sort can’t give you. Run the same day again next month, with whatever retries the provider sends that time, and you get the same bytes.

Choosing the Window

The window has to cover how far behind the newest event a first delivery can arrive. Here that’s under three seconds, so 60 seconds is generous. Retries don’t need to fit inside it: a retry of an event that’s already out is a repeat, and it’s dropped whenever it comes, as long as Inorderly still remembers the event. The default is to remember everything, and -remember puts a limit on that for long runs.

If a first delivery ever comes later than the window, it’s refused and written to the rejects file, never slipped into the middle of output that has already gone. That keeps the output the same. It also gives you one small file to look at instead of a total that’s quietly off.

Run It Yourself

The example is in the Inorderly repository. go run ./examples/usecases -dir uc webhooks writes the three delivery files and checks everything above. Every number here comes from the recorded run. The quick start has the same idea at five lines, and the other worked examples cover two redundant market data feeds, sensor telemetry and a ledger that has to agree with billing and an audit.