Merging Two Lossy Market Data Feeds Into One Tick Stream With Exact Gaps

Exchanges send market data twice, over two lines usually called A and B. Each line drops a packet now and then, and rarely the same one. Take both, keep the first copy of each tick, and you get a fuller feed than either line alone. That’s the theory. The merge code is where it gets awkward.

This worked example sends 150,000 ticks for three symbols over two lines. Each line loses about 2% of its ticks at random. Line A also goes quiet for four seconds and line B for three, at different times. Line B runs a few milliseconds behind A. In the end 143,458 ticks came on both lines, and 84 came on neither.

The ticks are merged three times: live twice, with different line delays, and once more from the two capture files read side by side, the way you’d replay a day.

First Seen, First Sent

The usual merge passes each tick on the first time its sequence number turns up. It’s fast and it drops the repeats. It also sends ticks out of sequence. When line A misses tick 88 and has already delivered 89, the 88 from line B arrives a few milliseconds later and goes out after 89. That happened 383 times in the first live merge and 425 times in the second. The output differs from one run to the next, so a replay of the day can’t be checked against what was sent live.

And it can’t say what’s missing. A tick that neither line delivered looks just like a tick that hasn’t arrived yet.

One Stream, in Sequence

Inorderly reads the same arrivals with a ten-second window. For the capture files it reads both, a line from each in turn:

inorderly -window 10000 -in line-a.ndjson -in line-b.ndjson -out capture.out.ndjson -rejects capture.rejects.ndjson

Each tick’s identity is its symbol plus its sequence number, and the copy from line B is the same event as the copy from line A, so the second one is dropped. 149,916 ticks come out, each once, every symbol’s ticks in sequence order. The 84 lost on both lines are reported as 84 gaps, by symbol and sequence number, which is exactly what a recovery request to the exchange needs.

Both live merges and the replay from the capture files give the same output, with one replay hash: sha256:6fa20783…e96b.

Why Ten Seconds

Live, the two lines are milliseconds apart and a much smaller window would do. The capture files are another matter. The command reads them a line at a time in turn. After line A’s four-second outage its file is a few hundred ticks short of line B’s, so the two drift apart in time while they’re read. Ten seconds covers that comfortably. A tick that ever arrives later than the window is refused and written to the rejects file rather than slipped in out of place.

Ticks with the same timestamp are ordered by symbol, then sequence number. That’s a fixed rule, published in the contract, so two systems merging the same lines put those ties the same way round.

Run It Yourself

The example is in the Inorderly repository: go run ./examples/usecases -dir uc feeds writes the live and capture files and checks all of the above. The numbers come from the recorded run. The demo runs a single-symbol version too, and the other worked examples cover retried webhooks, sensor telemetry and a ledger that has to agree with billing and an audit.