03, Data fabric
Current
Batch for the warehouse, a stream for the few decisions that cannot wait, and tests that fail the pipeline before a dashboard lies.
Decision. Which number is allowed to be an hour late, and which is not.
How it is shaped
01
Decisions
List the questions first. Freshness follows the decision, not a brochure about streaming.
02
Contracts
Schemas and tests at the boundary. A broken source stops, loudly.
03
Lanes
Batch as the default. A stream only where latency is part of the requirement.
What runs
- A smaller twin runs on our own sources.
- Failed tests page a person. They do not vanish into a green dashboard.
- Lineage a new engineer can read.
Stack, kept short
- Ingestion
- Quality tests
- Warehouse
- Streaming lane
Use this when the warehouse, the ops feed, and the model are currently three unhappy neighbours.
Borrow this shape