This topic is the architecture of the analytical stack — the systems an organisation builds to query “what happened”, “what’s happening”, and “what will happen” with data from many sources.

The Historical Evolution

The current consensus architecture is a lakehouse — file store of open formats underneath, query engines on top — but the historical design debate still informs each organisation’s shape.

Kimball vs Inmon

Two philosophies of data warehouse architecture, each named after its originator:

The practical reality: most modern data platforms are Kimball-shaped at the consumption end (semantic layer reads from star-schema models) and Inmon-shaped at the data foundation (raw data normalised into a “clean” Bronze/Silver layer). The two are complementary, not exclusive.

Data Warehouse vs Data Lake

Each has its characteristic failure mode: the warehouse’s is strictness ossifying schema creation; the lake’s is chaos preventing reuse. The lakehouse is the synthesis that aims for both.

The Lakehouse

A lakehouse is the architecture that puts warehouse guarantees on top of lake storage:

                      Query engine (Spark, Trino, Snowflake, ...)
                              ↓
                      Open table format (Iceberg / Delta / Hudi)
                              ↓
   ┌──────────────┴───────────────┐
   │    Object storage (S3, GCS)    │   ← table data in Parquet / ORC
   └───────────────────────────────┘

The three properties that distinguish it from raw S3-with-Parquet files:

1. ACID transactions — multi-table writes are atomic via a snapshot mechanism. Inserting five files partitioned across two tables either all commit or all roll back.
2. Schema management — the table format tracks schema history; you can time-travel to the table as of 2024-03-12 10:00:00.
3. Catalog integration — a metastore (Hive Metastore, Glue Catalog, Unity Catalog, Nessie) provides SQL-discoverable table names.

The data lives in object storage (cheap, durable, write-once). The metadata lives in the table format (snapshot history, manifest of data files). The query engine reads the manifest, prunes files by predicate, scans only relevant Parquet files.

Iceberg, Delta, Hudi

Three open table formats dominate:

Each is technically interchangeable; the choice is downstream of the ecosystem you’ve standardised on. Iceberg has the broadest engine support; Delta has the smoothest Databricks experience; Hudi is the right call for streaming-heavy workloads with upsert semantics.

ETL vs ELT

The cloud warehouse made ELT dominant: storage is cheap enough to land raw, compute is idle enough that transformations can run as SQL inside the warehouse, and the analytics engineer is more productive writing SQL than writing Python pipelines.

The shift to ELT pairs naturally with tools like dbt — a SQL-first transformation layer that runs as queries, with tests and lineage. The transformation step is the warehouse’s job; the warehouse’s strength is exploited, not avoided.

Practice Trajectory

1. Pick an analytical pipeline in your organisation. Identify which steps would today be ETL vs ELT; identify which would benefit from the shift.
2. Sketch a Bronze / Silver / Gold lakehouse architecture for clickstream + payments. Name the open format at each layer; explain what each layer’s contract is.
3. Pick a current “data swamp” problem in your stack. Identify which lakehouse property (ACID, schema history, catalog discoverability) would mitigate it most.
4. Compare costs: storing 10TB warehouse-native vs S3-lakehouse. Where does the trade-off tip — at what table size, query frequency, or freshness requirement?
5. Run dbt against a small dataset; identify one test, one model, and one lineage graph edge. Note what becomes visible that wasn’t before.

When It’s the Right Tool