Building a Modern Data Platform: The Complete Guide

A modern data platform is no longer just a warehouse for business intelligence. In an era where every company is becoming an AI company, the data platform has evolved into the central nervous system for the entire organisation. It powers not only executive dashboards but also customer-facing product features, operational applications, and the retrieval-augmented generation (RAG) systems that define modern AI.

Getting the architecture right from day one is the difference between a platform that accelerates the business and one that becomes a costly, brittle liability. In our engagements, we see too many teams either over-engineer a solution for problems they don’t have or under-invest in the foundational layers of governance and testing, leading to a crisis of trust in their data.

This guide provides a reference architecture for a modern data platform in 2026. We will cover the core layers, ingestion patterns, governance, cost controls, and AI-readiness, along with a pragmatic, phased plan to build it. This is the blueprint we use to help clients from funded startups to established enterprises build data capabilities that last.

What a modern data platform must do in the AI era

The requirements for a data platform have expanded significantly beyond traditional analytics. Yesterday's data warehouse was successful if it could populate a weekly sales dashboard. Today's platform must operate at a much higher level of utility, reliability, and scope. The work of building and maintaining this foundation is the core of data engineering.

A truly modern platform must be able to:

1. Serve Multiple Consumers: The platform is no longer a back-office function serving only analysts. It must serve three distinct audiences:

   • Analytics: Business intelligence (BI), ad-hoc querying, and strategic analysis.
   • Applications: Powering features in production software, such as recommendation engines, personalisation, or reporting inside a SaaS product.
   • AI/ML: Providing training data for traditional ML models and, increasingly, clean, structured, and unstructured data for generative AI and RAG systems.

2. Handle Diverse Data Types: SQL-based analytics on structured data is table stakes. A modern platform must be a polyglot, natively handling unstructured data (PDFs, images, audio), semi-structured data (JSON logs), and geospatial data alongside traditional relational tables. The rise of multimodal AI models makes this a non-negotiable requirement.

3. Embed Governance by Design: Governance can no longer be an afterthought. A modern platform has security, compliance, quality, and discovery built into its core. This includes data contracts at the point of ingestion, automated quality checks, column-level lineage, and role-based access controls that persist from the storage layer to the consumption layer. For any company operating in Europe, this is essential for GDPR compliance.

4. Balance Performance and Cost: The shift to cloud-native, consumption-based pricing models offers incredible scale but also introduces the risk of runaway costs. The platform's architecture must provide mechanisms for monitoring, forecasting, and controlling spend without sacrificing the performance required by its users.

Reference architecture: bronze, silver, gold and the semantic layer

The most resilient and scalable architecture we implement today is the multi-layered data lakehouse. This pattern, popularised by Databricks, combines the low-cost, flexible storage of a data lake with the transactional guarantees and performance of a data warehouse. It organises data into distinct quality zones, making pipelines more robust, debuggable, and easier to govern.

                                │◄───────────────────────────┐
                                │    SEMANTIC LAYER          │
                                │ (Metrics: dbt, Cube)       │
                                │                            │
┌───────────────────┐           ├───────────────────┬────────┴────────┬───────────────────┐
│   BI & Analytics  │           │   AI / ML Models  │   Operational   │  Customer-Facing  │
│   (Tableau, PBI)  │           │   (RAG, Forecast) │   Applications  │    (APIs)         │
└───────────────────┘           └───────────────────┘  (Internal Apps)└───────────────────┘
        ▲                                 ▲                   ▲                 ▲
        │                                 │                   │                 │
┌───────┴─────────────────────────────────┴───────────────────┴─────────────────┴───────┐
│   GOLD  │ Business-ready, aggregated data. Star schemas, feature stores, app views.   │
│ (Warehouse: Snowflake, Databricks SQL, BigQuery)                                      │
└───────────────────────────────────────────────────────────────────────────────────────┘
                                        ▲
                                        │ (Modelling, Aggregation - dbt, Spark)
┌───────────────────────────────────────────────────────────────────────────────────────┐
│   SILVER│ Cleaned, conformed, validated data. Columnar format (Delta Lake, Iceberg).    │
│ (Lakehouse Storage: S3, ADLS, GCS + Compute Engine)                                   │
└───────────────────────────────────────────────────────────────────────────────────────┘
                                        ▲
                                        │ (Cleansing, Structuring - Spark, Python)
┌───────────────────────────────────────────────────────────────────────────────────────┐
│   BRONZE│ Raw, immutable source data. Landed as-is. Schema-on-read.                   │
│ (Data Lake Storage: S3, ADLS, GCS)                                                    │
└───────────────────────────────────────────────────────────────────────────────────────┘
        ▲                                 ▲                   ▲                 ▲
        │                                 │                   │                 │
┌───────┴───────────┐           ┌─────────┴─────────┐ ┌───────┴───────────┐┌────┴────┐
│  Databases (CDC)  │           │ SaaS APIs (Batch) │ │ Event Streams     ││  Files  │
│  (Postgres, MySQL)│           │ (Salesforce, Hubspot) │ │ (Kafka, Kinesis)  ││ (PDF, CSV)│
└───────────────────┘           └───────────────────┘ └───────────────────┘└─────────┘

### Bronze: The Raw Landing Zone

The first layer, Bronze, is an immutable, append-only store of raw data. Data is ingested from source systems and lands here with minimal to no transformation.

• Purpose: To create a permanent, auditable record of the source data. If a bug is discovered in a downstream pipeline, you can always rebuild the Silver and Gold layers from the raw Bronze data without having to re-fetch from the source system (which may no longer be possible).
• Format: Data is often stored in its native format (JSON, CSV, Avro) or a simple columnar format like Parquet.
• Structure: Typically organised by source system and ingestion date (e.g., s3://bucket/bronze/salesforce/account/2026/05/20/).
• Key Principle: Schema-on-read. You don't enforce a rigid schema here; you capture the data as it arrives, warts and all.

### Silver: The Enriched and Conformed Layer

The Silver layer is where the data begins to take shape. It represents a cleaned, validated, and conformed version of the truth.

• Purpose: To provide a reliable, single source of truth for major data entities (e.g., customers, products, orders). Multiple raw sources may be joined and deduplicated to create one Silver table.
• Transformations: Data cleansing (handling nulls, standardising formats), data type casting, deduplication, and joining to create a base "conformed" model.
• Format: Always a transactional, columnar format like Delta Lake or Apache Iceberg. These formats provide ACID transactions, time travel (querying the data as it was at a specific point in time), and performance optimisations.
• What we would NOT do: We advise against putting heavy business-specific aggregations in the Silver layer. It should remain focused on clean, entity-level data, not report-specific views. Keeping it relatively un-aggregated maximises its utility for unforeseen future use cases.

### Gold: The Business-Ready Product Layer

The Gold layer is the "data product" layer, optimised for specific business use cases. This is what end-users and applications consume.

• Purpose: To provide high-performance, easy-to-query datasets for analytics, ML, and applications.
• Structure: Data is often denormalised and aggregated into wide tables or star schemas for BI tools. For ML, it might be structured as feature tables. For applications, it could be specific views designed to back an API.
• Consumers: BI analysts, data scientists, and increasingly, backend application services. High-quality software engineering is key to building reliable APIs on top of Gold tables.

### The Semantic Layer

The semantic layer is not a data storage layer but a logical abstraction that sits on top of the Gold layer. It provides a consistent, business-friendly view of the data.

• Purpose: To define key business metrics and dimensions in one place. For example, the definition of "Monthly Active User" or "Net Revenue" is coded once and reused across all tools (BI, AI, etc.), eliminating inconsistency.
• Tools: dbt's Semantic Layer, Cube, AtScale, or custom solutions.
• Benefit: Decouples downstream tools from the physical structure of the Gold tables. You can refactor your data models without breaking every dashboard in the company.

Batch, streaming and CDC — choosing per use case

Not all data needs to be real-time. Choosing the right ingestion pattern for each source is a critical architectural decision that balances latency, cost, and complexity. A well-designed platform often uses all three. For a deeper dive into these patterns, our guide on Data Pipeline Architecture Explained provides additional detail.

### Batch: The Workhorse

Most data analytics does not require sub-second latency. Batch processing, where data is collected and processed in discrete chunks (e.g., every hour or every 24 hours), remains the most common, cost-effective, and reliable pattern. It's ideal for ingesting data from SaaS APIs (like Salesforce), file drops, and any source where near-real-time updates don't justify the complexity.

### Streaming: For Real-Time Needs

When business operations depend on immediate data, streaming is necessary. This involves processing data event-by-event as it arrives.

• Use Case: A fintech company needs to detect fraudulent transactions in real-time. An e-commerce platform wants to update inventory levels instantly as items are sold.
• Technology: Apache Kafka, AWS Kinesis, or Google Pub/Sub for the event bus, with processing engines like Apache Flink or Spark Streaming.
• Trade-off: Streaming systems are significantly more complex to build, operate, and scale. They require careful state management and higher operational overhead. Only use streaming when a clear business case for sub-minute latency exists.

### Change Data Capture (CDC): The Modern Standard for Databases

CDC is a technique for efficiently capturing row-level changes (inserts, updates, deletes) from a source database's transaction log and replicating them to a target, like a data warehouse.

• Advantage: It's far more efficient than repeatedly querying a production database or performing full table dumps. It puts minimal load on the source system and delivers changes with low latency (seconds to minutes).
• Technology: Tools like Debezium, Fivetran, or Striim are leaders in this space. They read from the logs of databases like PostgreSQL, MySQL, and SQL Server.
• Our approach: For any production OLTP database, CDC is now our default recommendation for ingestion. It provides a near-perfect replica in the Bronze layer, enabling both analytical use cases and a reliable disaster recovery source.

Orchestration, testing and data contracts

A data platform without robust orchestration and testing is just a collection of fragile scripts. Applying software engineering best practices to data pipelines is what creates a reliable, production-grade system.

• Orchestration: Modern workflow orchestrators like Dagster, Airflow, and Prefect are essential. They go far beyond cron, providing dependency management, automatic retries, backfills, observability, and alerting. In our engagements, we favour orchestrators that are data-aware (like Dagster), as they allow for more explicit modelling of the data assets being produced.

• Testing: Data quality must be automated and continuous. We implement several layers of testing:

  • Unit Tests: On the transformation code itself (e.g., using Python's pytest on a Spark transformation function).
  • Data Quality Tests: On the data assets produced by a pipeline. Tools like dbt and Great Expectations allow you to declare assumptions about your data (e.g., user_id must be unique and not null) and fail the pipeline if they are not met.

• Data Contracts: This is a crucial emerging concept for preventing data quality issues at the source. A data contract is a formal agreement between a data producer (e.g., a backend service team) and a data consumer (the data platform) that defines the schema, semantics, and quality guarantees of the data being exchanged. It is enforced by automated checks in the producer's CI/CD pipeline. When a producer attempts to deploy a breaking change (e.g., renaming a field), the build fails, forcing a conversation with downstream consumers. This prevents the "silent breakage" that plagues so many data platforms.

Governance: catalogue, lineage, access, retention

As a platform scales to serve more domains and users, governance becomes the primary enabler of trust and security.

• Data Catalogue: A catalogue answers the question, "What data do we have, and what does it mean?" Tools like DataHub, Amundsen (open source), or commercial offerings like Alation and Collibra scan your data assets and provide a searchable inventory. Good catalogues allow for business context, ownership, and quality metrics to be attached to each data asset.

• Data Lineage: Lineage tools automatically track the flow of data from source to consumption, often at the column level. When a user in a BI tool asks, "Where did this 'revenue' number come from?", lineage provides a visual map back through the Gold, Silver, and Bronze layers to the source system. This is indispensable for debugging, impact analysis, and regulatory audits.

• Access Control: Access to data must be managed centrally and consistently. The principle of least privilege should apply. Modern platforms like Snowflake and Databricks provide robust role-based access control (RBAC) that can secure data down to the row and column level. For instance, you can create a policy where managers can only see salary data for their direct reports within the same table.

• Data Retention & Deletion: Policies for data retention and deletion are critical for both cost management and compliance (e.g., GDPR's "right to be forgotten"). The layered architecture helps here. A deletion request can be processed by dropping the user's data from the Silver and Gold layers, while an anonymised record might be retained in Bronze for historical integrity if regulations permit.

AI-ready extensions: embeddings, chunking, permission-aware retrieval

The modern data platform must be architected to support generative AI workloads, particularly RAG. This introduces new types of data and new pipeline requirements. For a deeper look, see our analysis of how companies prepare their data for generative AI.

• Embeddings Pipelines: RAG systems work by searching over vector embeddings—numerical representations of your documents. Your data platform needs pipelines to:

  1. Generate these embeddings using models like OpenAI's text-embedding-3-large or open-source alternatives.
  2. Store them, typically in a specialised vector database (e.g., Pinecone, Weaviate) or increasingly, in native vector types within data warehouses like Snowflake or PostgreSQL with pgvector.
  3. Keep them in sync as source documents are added, updated, or deleted.

• Chunking and Metadata Management: Raw documents (like long PDFs) are too large to be used directly in RAG prompts. They must be split into smaller, semantically coherent "chunks". The data platform must manage this process, storing the chunks and, crucially, their metadata (e.g., source document ID, page number, author). This metadata is vital for citing sources in AI-generated answers. This is a core competency of our AI engineering practice.

• Permission-Aware Retrieval: This is the most complex challenge. A RAG system that tells a junior employee about upcoming M&A plans is a catastrophic failure. The AI's retrieval process must respect the user's permissions. This means the access control layer defined in your governance framework must be integrated into the retrieval step. When a user asks a question, the RAG system must first filter the searchable chunks based on that user's permissions before performing the semantic search. This cannot be bolted on; it must be designed in from the start.

Cost control and the warehouse bill nobody expected

Cloud data platforms offer incredible power, but their consumption-based pricing models can lead to staggering bills if not actively managed. We've seen multiple clients experience "bill shock" as their platform's usage grows.

A common scenario we encountered involved a mid-sized European insurer. Their data warehouse costs, primarily on Snowflake, grew from €20,000/month to over €75,000/month in a single quarter after onboarding a new product analytics team. Our investigation revealed two main culprits:

1. A few widely used dbt models were written inefficiently, causing full table scans over years of historical data on every run.
2. The team's BI tool was configured to refresh dashboards every 15 minutes, re-running these expensive queries constantly against the largest compute warehouse.

Our remediation focused on three areas:

• Query Optimisation: We refactored the dbt models to use incremental processing and better clustering keys, reducing the data scanned per run by over 90%.
• Compute Tiering & Caching: We instructed the BI tool to use a smaller, dedicated warehouse for its queries and implemented several key Gold tables as pre-aggregated materialized views.
• Monitoring & Alerting: We set up Snowflake resource monitors to automatically send alerts and suspend compute when daily credit usage exceeded a defined budget.

The result was a reduction in the monthly bill to a stable ~€35,000, while actually improving dashboard load times for users. This experience highlights the need for proactive cost governance, including query monitoring, budgeting, and educating users on the cost implications of their work.

A phased 12-month build plan with checkpoints

Building a comprehensive data platform is a marathon, not a sprint. We always advocate for a phased approach that delivers value incrementally. Trying to build the "perfect" enterprise-wide platform in one go is a recipe for a multi-year project that delivers no value until the end, by which time requirements will have changed. The way how we work is to deliver value in slices.

Here is a realistic 12-month plan for a company starting from scratch.

### Phase 1 (Months 1-3): Foundation & First Value

The goal is to solve one high-value problem to build momentum and secure stakeholder buy-in.

• Focus: Answer one critical business question (e.g., "What is our daily customer acquisition cost by channel?").
• Tasks:
  • Set up core cloud infrastructure (storage, warehouse, orchestrator).
  • Ingest 1-2 essential data sources (e.g., PostgreSQL database via CDC, Google Ads API via batch).
  • Build the Bronze -> Silver -> Gold pipeline for just this data.
  • Connect a BI tool and build the first dashboard.
• Example Cost: For a Series A startup, this initial phase might cost between €70,000 - €100,000 in engineering effort (e.g., 2 senior engineers for 3 months) plus cloud/licensing costs.

### Phase 2 (Months 4-6): Scale & Harden

With initial value delivered, the focus shifts to making the platform robust and scalable.

• Focus: Onboard a second business domain (e.g., Product) and industrialise the development process.
• Tasks:
  • Ingest product usage data from an event stream.
  • Integrate data quality testing (dbt tests, Great Expectations) into the pipelines.
  • Set up CI/CD for automated testing and deployment of data models.
  • Begin implementing basic data lineage and documentation.

### Phase 3 (Months 7-12): Govern & Extend for AI

Now the platform is stable and trusted, you can broaden its reach and capabilities.

• Focus: Implement enterprise-grade governance and extend the platform to serve AI use cases.
• Tasks:
  • Roll out a data catalogue and train business users on how to discover data.
  • Implement fine-grained role-based access controls.
  • Build the first embedding and chunking pipelines for a RAG pilot project.
  • Establish formal data ownership and stewardship programs.

Frequently asked questions

### What is a modern data platform?

A modern data platform is a governed, layered system designed to serve a wide variety of consumers beyond traditional analytics. It ingests raw data from many source systems, transforms it through validated and modelled layers (often Bronze, Silver, Gold), and serves it to BI tools, operational applications, and AI retrieval systems. Crucially, it has lineage, testing, and access control built into its architecture, ensuring the data it provides is not just available, but also reliable, secure, and trustworthy.

### How long does it take to build a data platform?

The timeline varies with scope, but you should aim to deliver a first useful slice of value in 8 to 12 weeks. This initial phase typically focuses on answering one critical business question. Building a fully governed, multi-domain platform that serves the entire enterprise is a longer journey, often taking 9 to 18 months. The key is to build it incrementally, prioritising work per business question or use case, not by trying to onboard every source system at once.

Key takeaways

• Modern data platforms serve AI and applications, not just BI. The architecture must support unstructured data, low-latency APIs, and AI-specific workloads like embedding generation.
• The Bronze, Silver, and Gold lakehouse is the dominant architectural pattern. It provides robustness, auditability, and a clear separation of concerns that scales effectively.
• Governance and testing are not optional extras. They must be designed in from the start to build trust and ensure security and compliance. Data contracts are a key part of this.
• Choose the right ingestion pattern for the use case. Batch is the default, but CDC is the modern standard for databases, and streaming should be reserved for true real-time needs.
• Cloud data warehouse costs require active management. Consumption-based models can be costly without query monitoring, optimisation, and clear budgeting.
• Build incrementally. Start small by answering one high-value business question in 2-3 months and expand from there.

Building a data platform that can truly accelerate your business requires getting the foundational architecture right. It is a complex undertaking that sits at the intersection of data, software, and AI engineering.

If you are planning a new data platform or looking to modernise an existing one, a thorough architecture assessment is the critical first step. We can help you design a blueprint that is scalable, cost-effective, and ready for the demands of AI.