Strong foundations

Data is a consumer of technology

Every other consumer of a product system meets it at a comfortable level of abstraction. The mobile app talks to an API. A partner integration receives a webhook. Support tooling reads a curated view. Each of them is handed a stable contract, and whatever churn happens underneath is somebody else's problem.

The data team is different, and the difference is structural rather than cultural. It consumes the same systems from underneath, reading tables, change logs, replicas and blobs, because that is the only place where the full history lives. It is a technology consumer whose needs are unusually complex and whose point of contact sits at an unusually low level of abstraction. Nobody chose that arrangement; it is what analytics requires. The consequence is that internal implementation details of a product system are, in practice, the data team's public interface.

Reconstructing events from state

Product systems are usually built to answer one question well: what is true right now? An order row holds the current status, a user row the current plan, an inventory row the current count. State is what the application needs to render a screen and enforce a rule.

Analytics asks the opposite question: what happened, in what order, and what changed between then and now? Cohorts, funnels, churn, attribution, lifetime value, any model with a time dimension: all of it is built from events, not from current state. So the data team spends much of its effort reconstructing an event stream that the source system never emitted, by diffing snapshots, reading change logs, and inferring transitions from timestamps that happen to survive.

That reconstruction is only as good as the traces left behind. Where the system recorded the path, it works. Where the system recorded only the destination, no amount of engineering downstream will recover the journey.

What gets lost, and why it never comes back

Information loss in a data model is quiet and permanent. There is no error, no failed build, no alert. A row is updated, the previous value ceases to exist, and nobody notices until months later when someone asks a question that would have needed it.

The usual culprits are unremarkable in isolation. A status column mutated in place, so the transitions between states are gone. A hard delete, so cancelled records vanish rather than being marked cancelled. A missing updated_at, so a change can be detected but not dated. An identifier that is reused after deletion, so two different entities collapse into one history. A denormalised field overwritten on write, so the value at the time of the transaction is replaced by the value today. An enum whose meaning is redefined without a version, so the same code means different things before and after a release nobody recorded.

Each of these is a reasonable local decision. Together they decide which questions the business will be able to answer for the rest of the product's life.

The cost lands somewhere else

The asymmetry is what makes this hard to manage. The team that takes the decision rarely pays for it. Overwriting a column is the simplest thing to write, ships fastest, and behaves correctly for the feature it was written for. The bill arrives on another team's desk, one or two quarters later, as a pipeline full of heuristics, a metric that cannot be computed historically, or a model that has to be trained on a shorter window than anyone wanted.

This is why exhorting people to 'think about data' does not work. The feedback loop is too slow and too indirect to teach anyone anything. What works is putting the consumer in the room while the decision is still cheap to change.

Collaboration, concretely

Strong foundations are less a technology choice than a working agreement between the technology and data teams. In practice it looks mundane: the data team reviews schema changes the way a downstream service would review an API change, and gets notice of migrations before they ship rather than after they break something. Entities that matter analytically carry immutable history, whether an append-only log, change data capture, or explicit events, instead of only current state. Deletes are soft where the record has analytical meaning. Timestamps are recorded in UTC, on every state change, including the ones the feature does not need. Identifiers are stable and never recycled. And when a system genuinely cannot carry its own history, someone owns emitting the events that carry it instead.

None of this asks product engineers to build a warehouse or to learn dimensional modelling. It asks for the same courtesy any team extends to a consumer it knows is there: a contract, a version, and a heads-up before it changes.

Design for the reader you cannot see

The practical test at design time is short. If this row is updated tomorrow, can we still tell what it looked like today, and when it changed? If this record is deleted, does the fact that it existed survive? If this identifier is reused, will two histories merge? If this enum gains a value, will last year's rows still mean what they meant?

Four questions, asked while the migration is still a pull request, prevent most of the archaeology that data teams otherwise perform for years. They cost minutes at design time and are effectively impossible to answer retroactively.

Foundations are a shared artefact

The instinct is to treat the data platform as the foundation, and the product systems as an input that arrives however it arrives. It is the wrong boundary. The foundation starts at the point where information is first written down, which is inside the product, in the technology team's schema, and the quality of everything downstream is capped there.

So the work is not to build a better pipeline around whatever the source produces. It is to treat the data team as the demanding, low-level, high-complexity consumer it actually is, and to design the systems it reads with that consumer in mind. Do that and analytics stops being archaeology. Skip it and every model, dashboard and agent downstream is quietly limited by a decision nobody remembers taking.