02 · What I do # Data architecture Data strategy · 6 min read Architecture is the set of decisions that are expensive to reverse. Not which dashboard tool to buy, but where data lands, in what format, on whose compute, and behind which interfaces. Those choices are cheap on a whiteboard and brutally costly in year three, which is why the useful question is never what the stack should look like today. It is which questions the business will need answered over the next few years, and which of today's shortcuts would make answering them impossible. In short - Architecture is the set of decisions that are expensive to reverse. Everything else is implementation. - Start from the use cases the business will have, not only the ones it has, and size for the next order of magnitude rather than the next three. - Keep layers replaceable through open formats and portable logic, and take coupling deliberately where leaving would cost a sprint rather than a project. ## Start from the use cases, not the tools The stack is downstream of the workload. Before naming a single technology, the list worth writing is the one nobody usually writes: what does this system have to answer, for whom, how fresh, how often, and how far back. Monthly management reporting and a real-time fraud check are not the same problem wearing different logos. One tolerates a night of latency and reads two years of history; the other needs a decision in milliseconds and cares about the last ten minutes. Then repeat the exercise for the roadmap. Not fantasy scenarios, but the things the business has already said it intends to do: launch in a second market, sell to enterprise customers who will demand audit trails, put a model in front of users, charge by consumption. Each of those quietly imposes a requirement on the architecture, and each is far cheaper to accommodate as a design assumption than as a migration. ## Sizing the future honestly Two failure modes, equally common. The first is building for a scale that never arrives: a distributed cluster, a streaming platform and a feature store for forty gigabytes of data and six analysts, at which point most of the team's effort goes into operating machinery rather than answering questions. The second is refusing to look up: a single production replica quietly carrying the entire analytical load until the day it stops, usually the day the company gets its first serious customer. The discipline is to estimate orders of magnitude rather than exact numbers, and to design for the next one, not the next three. Where is data volume in a year at current growth? How many concurrent consumers? How long must history be retained, and is that a product decision or a regulatory one? Getting within a factor of ten is enough to make the structural choices correctly, and pretending to more precision than that is theatre. ## Lake, warehouse, or lakehouse The three answers optimise for different things. A warehouse gives structure, governance and fast SQL over modelled data, at the cost of paying to structure everything you keep. A lake gives cheap storage and format freedom, at the cost of discipline: without contracts and catalogues it becomes a place where data goes to be forgotten. A lakehouse tries to keep the cheap storage and add transactional tables and schema enforcement on top of it. In practice the boundary matters less than it used to. Open table formats let storage and query engine be chosen separately, so the same files can be read by more than one engine and the engine can change without moving the data. That single property is worth more than most feature comparisons, because it converts an irreversible decision into a reversible one. The question to hold onto is not which architecture is fashionable, but which parts of this you would be able to replace in two years without a rewrite. > Architecture is the set of decisions you cannot cheaply undo. Everything else is implementation. ## Single database, or distributed processing Most companies need far less distribution than they think. A single well-tuned relational database, or a columnar warehouse with elastic compute, will carry an organisation a long way, and it will do so with an operational burden roughly one person can hold. Distributed processing earns its place when the data genuinely exceeds what one machine can process in an acceptable window, when transformations are complex enough that expressing them in SQL becomes the bottleneck, or when the same engine has to serve batch and machine learning pipelines over the same files. What is rarely priced in is the operating cost of that power: cluster tuning, dependency management, job orchestration, debugging failures that only appear at scale, and the hiring profile that comes with all of it. The right question is not whether the workload could run on a distributed engine, but whether the team can run the engine on a Tuesday when something breaks and the board deck is due Wednesday. ## Modularity, and where coupling is acceptable A modular architecture is one where each layer can be replaced without touching the others: ingestion, storage, transformation, serving, orchestration. The way to get there is boring and reliable. Keep data in open formats. Keep transformation logic in something portable rather than buried in a vendor's proprietary interface. Let layers talk through explicit contracts instead of reaching into each other's internals. Coupling is not always wrong; it is a trade with a price tag. A managed service that saves a quarter of engineering effort may be worth being locked into for three years. The mistake is making that trade without noticing. For each dependency, the honest question is what it would take to leave: a weekend, a sprint, or a project nobody will ever fund. Take the tight coupling where the answer is a sprint, and stay loose where the answer is a project. ## Cloud agnostic, or honest about the trade Being genuinely cloud agnostic means giving up the managed services that make a small data team productive, and paying for that abstraction forever, in complexity and in salary. It is the right call for some: regulated industries with sovereignty requirements, companies whose customers demand it contractually, workloads with negotiating leverage at renewal. For everyone else, the pragmatic position is to pick one cloud and stay portable where portability is cheap. Open storage formats, containerised workloads, transformation logic that is not written in a dialect only one vendor speaks, and infrastructure defined in code. That combination keeps the exit possible without paying the agnosticism tax every day. What matters is that the decision is explicit and written down, so that the next architect inherits a choice rather than an accident. ## The infrastructure follows the business Data teams are used to the idea that analysis should serve business objectives. Infrastructure is held to the same standard less often, and it should be. Latency targets should come from the decisions they enable, not from what the technology can do. Retention should come from what regulation and the product require. Spend should be traceable to the value of the workloads consuming it, which means cost per pipeline, per domain, per team, rather than one invoice nobody can decompose. It also means sequencing. If the company's next twelve months are about entering a new market, the architecture work that matters is multi-region data handling and localisation, not the streaming platform someone read about. Architecture that follows the business is not architecture that reacts to it; it is architecture that has already read the roadmap and made room. ## Deciding, and revisiting None of these questions have universal answers, which is precisely why they need to be answered explicitly, with the reasoning recorded. A short decision record for each structural choice, stating the context, the options and why one was taken, is worth more than any diagram. It tells the team a year later whether a constraint still holds or whether it quietly expired. Then revisit on a schedule rather than in a crisis. Growth invalidates assumptions, prices change, engines improve, the business turns. An architecture designed to be replaced one layer at a time can absorb that. One designed to be right forever will simply be wrong, expensively, and all at once. Written by Gonzalo Rodriguez Pardo, Chief Data Officer