The Problem with Two-Tier Data Architectures
For the past decade, many enterprises have operated a two-tier data architecture: a data lake for raw, unstructured, and semi-structured data, and a data warehouse for curated, structured data serving business intelligence workloads. This architecture made sense when BI and data science were separate disciplines with different tooling and different data requirements. But as AI and ML workloads have moved from the research lab to production, the separation has become a source of friction: data scientists need access to the raw data in the lake, BI teams need the governed, curated data in the warehouse, and maintaining two copies of data with different governance models is expensive and error-prone.
What the Lakehouse Pattern Solves
The data lakehouse pattern addresses this by combining the low-cost, flexible storage of a data lake with the data management and performance capabilities of a data warehouse — in a single architecture. The key enabling technologies are open table formats (Delta Lake, Apache Iceberg, Apache Hudi) that add ACID transactions, schema enforcement, and time travel capabilities to object storage. This means you can store data in open formats on commodity cloud storage while still getting the reliability and query performance that BI workloads require.
Choosing Your Table Format
The three dominant open table formats — Delta Lake, Apache Iceberg, and Apache Hudi — each have different strengths. Delta Lake, developed by Databricks, has the strongest ecosystem integration with Spark and the broadest adoption in the Databricks community. Apache Iceberg has strong support from AWS (through Glue and Athena) and is the preferred format for organisations running on AWS. Apache Hudi, developed at Uber, has particular strengths for streaming ingestion and incremental processing use cases. For most organisations, the choice is driven by cloud provider and existing tooling rather than fundamental technical differences.
The Medallion Architecture
The medallion architecture — organising data into bronze (raw), silver (cleansed and conformed), and gold (business-ready) layers — has become the dominant data organisation pattern in lakehouse implementations. Bronze stores raw data exactly as received from source systems, providing a complete audit trail and enabling reprocessing. Silver applies data quality rules, deduplication, and schema standardisation. Gold contains business-level aggregations and metrics optimised for consumption by BI tools and AI models. The layered approach makes data lineage transparent and enables different teams to work at the layer appropriate to their needs.
Governance in the Lakehouse
Data governance is one of the most important and most frequently underestimated aspects of a lakehouse implementation. Open table formats provide the technical foundation for governance — schema enforcement, access controls, audit logging — but governance is ultimately a people and process challenge as much as a technical one. Unity Catalog (Databricks), AWS Lake Formation, and Microsoft Purview provide centralised governance capabilities for lakehouse environments, but they require investment in data cataloguing, classification, and access policy design to deliver their value.
Migration Path from Existing Architectures
For organisations migrating from existing data warehouse or data lake architectures, we recommend a phased approach. Start by establishing the lakehouse foundation: choose your table format, set up the medallion architecture, and implement governance tooling. Then migrate new data sources directly to the lakehouse rather than the legacy architecture. Gradually migrate existing data warehouse workloads, starting with those that will benefit most from access to raw data or that have high storage costs in the warehouse. Retire the legacy architecture only when all workloads have been successfully migrated and validated.