Stream or batch: choose from the decision

A pipeline becomes expensive when every table is treated as if a trader were staring at it. Most are not. Current, our practice fabric, splits the world in two. The warehouse gets the tested, documented, slightly late truth. A narrow stream carries the events whose age is part of the decision: an outage, a threshold, a spike that looks like fraud.

The design conversation we insist on is a list. For each number on the page your team actually opens: who acts, what they do, and how old the number may be before the action is wrong. That list retires more Kafka clusters than any benchmark.

Tests sit on the boundary, not in a quarterly cleanup. A source that changes shape should fail the job and tell a person. A green dashboard over a silent failure is how operators learn to export CSV files and keep a private sheet, the beginning of the end of the warehouse.

Models belong downstream of those tests. A forecast on an untested feed is a confident rumour. Gauge does not train until the series has a contract. That sequencing feels slow in week one and obvious the first time a source breaks on a Friday.