Ordering Sensor Telemetry From Devices With Bad Clocks and Late Uploads

Sensors are the worst-behaved producers there are. Their clocks drift. Some sit behind a gateway that forwards readings in batches. Some lose their connection for a while, keep what they measured, and send it all at once when they’re back. A few readings just never arrive.

This worked example has 200 devices sending a temperature reading every 10 seconds for three hours, 216,000 readings in all. Their clocks are up to 3 seconds out. Fifty of them sit behind five gateways that forward every 45 to 90 seconds. Twenty go offline for 2 to 12 minutes and ten for 25 to 40 minutes. 634 readings are lost on the way. Each reading carries its device’s sequence number. The readings are uploaded three times, with different network delays and gateway timing each time.

The job is the obvious one: an average temperature for every minute.

The Minute That Closes Too Soon

A hand-written aggregator keeps a bucket per minute, by the device’s own time, and closes it once readings from a minute past its end have come in. Anything for a closed minute is thrown away. It’s a reasonable design, and it’s what most streaming code does.

It dropped 4,183 readings on the first upload run, 4,335 on the second and 4,298 on the third. The gateway batches are the trouble. Whether a batch lands before or after its minute closes depends on when the gateway happened to flush, so 157 of the 182 minute averages came out different between the first run and the second. Same readings, different numbers. Nothing records which readings were dropped.

Ordered First, Then Averaged

Put Inorderly in front with a 15-minute window, and average its output instead:

inorderly -window 900000 -in uploads-1.ndjson -out uploads-1.out.ndjson -rejects uploads-1.rejects.ndjson

The gateway batches are well inside the window, and so are the backlogs from the short outages. They’re all put in time order with everything else. The long outages are another matter. Readings more than 15 minutes old when they finally arrive are refused as late, 972 of them, all from the ten devices that were off longest, and each one goes to the rejects file with its device and line.

214,394 readings come out, in the same order on every run, with one replay hash: sha256:3ba8ff69…5251. The minute averages taken from that output are the same every time, because they’re computed from the same bytes.

Nothing goes missing silently either. Each device numbers its readings, so Inorderly reports every hole in each device’s numbering: 639 gaps covering 1,606 readings, which is exactly the 634 lost in transit plus the 972 refused. A missing reading is listed with its device and its number, whatever the reason it’s missing.

Choosing the Window

The window is a trade. A longer one accepts older backlogs and holds output back for longer. Fifteen minutes here means a minute’s average is final about fifteen minutes after the minute ends. If the averages are for a dashboard, take a short window and send the rejects to a slower correction job. If they’re for billing or compliance, take a long one. Either way, the same readings give the same result.

Clock offsets of a few seconds don’t need fixing first. Each device’s readings stay in its own sequence order, and readings from different devices are ordered by their stamped times, which is as good as those clocks allow, and the same on every run.

Run It Yourself

The example is in the Inorderly repository: go run ./examples/usecases -dir uc telemetry writes the three upload files and checks everything above. The numbers come from the recorded run. The other worked examples cover retried webhooks, two redundant market data feeds and a ledger that has to agree with billing and an audit.