What Is Data Engineering and Why Does Every AI Company Need It?

Data engineering is the practice of designing, building, and maintaining the systems that collect, store, and prepare data for analysis and machine learning. It's the essential, often invisible, infrastructure that underpins every successful data-driven organisation. While data scientists and AI models generate the headlines, data engineers build the roads, bridges, and pipelines that deliver the fuel they run on. Without robust data engineering, analytics are slow and unreliable, and artificial intelligence initiatives fail to move from promising prototypes to production-ready systems that deliver business value. It is the disciplined, systems-thinking approach to managing an organisation's most valuable asset: its data.

The components of a modern data platform

A modern data platform isn't a single piece of software but an interconnected system of tools and processes. At Golux Group, we think of it in five distinct layers, each with a specific job to do. Understanding these layers helps to demystify the stack and clarifies where investment is needed.

Ingestion

This is the process of moving data from its source into a central storage system. Sources can be anything from internal production databases (like PostgreSQL or MongoDB), SaaS applications (like Salesforce or HubSpot), event streams (like Kafka), or even simple CSV files from a partner.

• The Challenge: Sources are numerous and heterogeneous. APIs change, schemas drift, and networks fail. Ingestion needs to be reliable, resilient, and observable.
• Common Approaches: Managed ETL/ELT services like Fivetran or Airbyte offer hundreds of pre-built connectors, significantly accelerating development. For bespoke or very high-volume sources, custom-built connectors using Python or event streaming platforms like Kafka are common. The pattern we see most often is a mix: using managed services for common SaaS tools and building custom ingestion for core, proprietary systems.

Storage

Once ingested, data needs a home. Historically, this was a data warehouse, optimised for structured, relational data. Today, the dominant paradigm is the data lakehouse, which combines the low-cost, flexible storage of a data lake with the performance and management features of a data warehouse.

• The Challenge: Storage needs to be cost-effective for vast volumes of data, from structured tables to semi-structured JSON and unstructured text or images. It must also be fast enough to support interactive queries and model training.
• Common Platforms: The market is led by three major cloud platforms: Snowflake, Databricks, and Google BigQuery. All three separate storage from compute, meaning you can scale each independently, and all support SQL as the primary interface, which has democratised access.

Transformation

Raw data is rarely useful. It contains duplicates, errors, and inconsistencies. It's structured for operational efficiency, not analytical insight. The transformation layer is where this raw data is cleaned, modelled, and reshaped into assets ready for consumption. This is the "T" in ELT (Extract, Load, Transform), the dominant architectural pattern today.

• The Challenge: Transformations must be reliable, testable, and documented. When a number in a report is questioned, you must be able to trace its lineage back to the source.
• The Key Tool: The near-universal tool for this layer is dbt (Data Build Tool). It allows engineers to build and manage complex data transformation pipelines using simple SQL, combined with software engineering best practices like version control, testing, and documentation. For more complex transformations involving unstructured data or very large datasets, Apache Spark remains a powerful option. A well-structured transformation layer is often the most significant value driver in a data platform, as explored in our guide to building a modern data platform.

Serving

The serving layer is where value is delivered to the business. It's how end-users—whether they are analysts, data scientists, or AI applications—access the prepared data.

• The Challenge: The serving layer must cater to different needs. An analyst using a BI tool needs aggregated, easy-to-query tables. A real-time personalisation engine needs low-latency access to user profiles. A generative AI application needs access to a vector database for semantic search.
• Common Approaches:
  • Business Intelligence (BI): Tools like Tableau, Power BI, or Metabase connect directly to the data warehouse.
  • Machine Learning: Data scientists often pull data into notebooks (e.g., Jupyter) for exploration and model training. Production models might access data via a dedicated "feature store."
  • AI Applications: Retrieval-Augmented Generation (RAG) systems often query a vector database (like Pinecone or Weaviate) that has been populated by a data engineering pipeline.

Governance

Data governance is the framework for managing data as a strategic asset. It encompasses security (who can see what), privacy (compliance with GDPR, etc.), quality (ensuring data is accurate and trustworthy), and lineage (tracking data from source to destination).

• The Challenge: Governance was often an afterthought. In the age of AI, it is a prerequisite. You cannot responsibly deploy an AI system on data that you cannot secure, audit, or trust.
• Common Approaches: Governance can be implemented using features within the data platform itself (e.g., Snowflake's column-level security and data masking) or through dedicated governance tools like Atlan or Collibra, which provide a unified data catalogue and lineage graph.

The table below compares three common approaches to building out this stack, which we frequently discuss with clients.

Our recommendation for most organisations starting today is a hybrid approach. It balances the speed of managed services for non-critical components with the control and cost-effectiveness of open-source software for the core transformation and storage layers.

Why AI multiplies the cost of bad data

The old adage "Garbage In, Garbage Out" has been a truism in analytics for decades. With artificial intelligence, and particularly with large language models (LLMs), this is no longer sufficient. It's more accurate to say "Garbage In, Catastrophe Out." AI systems act as multipliers, amplifying the consequences of poor-quality data across the organisation at a scale and speed humans cannot manage.

Let’s consider a concrete example. We worked with a Series A logistics platform in Germany aiming to build an AI-powered pricing engine. Their goal was to provide dynamic quotes for shipments based on historical data, capacity, and route information. Their core shipment data was spread across a production PostgreSQL database and thousands of historical spreadsheets.

During initial analysis, we found that approximately 10% of their historical shipment records had incorrect or missing gross_weight_kg data, often defaulting to 1. A traditional analyst might spot and filter these outliers. An AI model, however, learns from them.

The initial model, trained on this uncleaned data, began systematically underpricing heavy shipments, assuming they were light. The business impact was direct and severe:

1. Direct Financial Loss: In a single month of testing, the model quoted on 500 shipments where the weight was incorrectly learned. The average pricing error was €80 per shipment. This resulted in a direct loss of €40,000 in one month.
2. Wasted Engineering and Cloud Costs: Before the data quality issue was identified as the root cause, the machine learning team spent six weeks attempting to "fix the model"—tuning hyperparameters, trying different algorithms, and re-training.
   • Cost of 2 senior ML engineers for 1.5 months: 2 * €12,000/month * 1.5 months = €36,000
   • Cost of wasted GPU training runs on a cloud provider: ~€10,000
3. Erosion of Trust: The sales team quickly lost faith in the AI tool, reverting to manual quoting and refusing to adopt the new system. The entire project, an investment of over €250,000, was at risk of being written off.

The total quantifiable cost of this one data quality issue exceeded €86,000 in less than two months. The fix was not a more complex model; it was a robust data engineering pipeline. We implemented a system using dbt to automatically run data quality tests on every batch of new data. Shipments with weights outside a logical range (e.g., gross_weight_kg < 5 or gross_weight_kg > 40000) are now automatically flagged and quarantined for manual review before they ever reach the model training dataset. This foundational work is crucial for any organisation looking to prepare their data for generative AI and other advanced applications.

A five-question maturity test for your data engineering

As a leader, you don't need to know how to configure a Spark cluster. You do need to know whether your data infrastructure is an asset or a liability. Ask your technical leadership these five questions to get a quick read on your organisation's data engineering maturity.

1. Time-to-Insight: "If we sign a new SaaS partner today, how many person-weeks would it take for their data to be available for querying in our BI tool?"
2. Trust & Lineage: "If I see a key metric in our quarterly board report, can you show me, in under 30 minutes, exactly which source systems it came from and the transformations applied?"
3. Access & Security: "Is our process for granting access to sensitive customer data automated, auditable, and role-based? Or does it involve a manual request to an engineer?"
4. Reliability & Observability: "When a data pipeline fails overnight, who is automatically alerted, and do we have a clear Service Level Objective (SLO) for how quickly it must be fixed?"
5. Efficiency: "What percentage of our data scientists' and analysts' time is spent finding, cleaning, and validating data versus building models and creating insights?"

Based on the answers, you can place your organisation on the following maturity spectrum.

Organisations in the "Lagging" column are not ready for significant AI investment. Their projects will be slow, expensive, and likely to fail. Those in the "Leading" column have the foundation to rapidly build and deploy trustworthy AI at scale.

Core roles in a modern data team

As the data stack has matured, the roles have specialised. Confusing them is a common hiring mistake. While in a small startup one person might wear all three hats, understanding the distinct functions is key to building a scalable team. There's a lot of nuance in the distinctions between data engineering, data science, and machine learning, but here we focus on the roles that build the platform itself.

Data Engineer

The data engineer is the traditional owner of the data platform. They are systems builders concerned with the reliability, scalability, and efficiency of the data infrastructure.

• Primary Focus: Building and maintaining robust, scalable data pipelines. Infrastructure as Code (Terraform), orchestration (Airflow, Dagster), and data processing frameworks (Spark, Kafka).
• Core Skills: Strong software engineering fundamentals, Python, Java or Scala, SQL, deep knowledge of cloud infrastructure (VPC, IAM, Kubernetes), and distributed systems.
• Key Question: "How do I move 10 terabytes of data from A to B reliably, every hour, for the lowest possible cost?"

Analytics Engineer

This is a newer role, popularised by dbt, that sits between data engineers and data analysts. They focus on the transformation layer, turning raw, messy data into clean, well-documented, and business-ready data models.

• Primary Focus: Curating the data that the rest of the business uses. They are experts in the business logic.
• Core Skills: Exceptional SQL, dbt, data modelling techniques (e.g., Kimball), version control (Git), and strong communication with business stakeholders.
• Key Question: "What is the single source of truth for 'monthly active users', and how can I model it so it's easy for everyone to use and trust?"

Platform Engineer

Platform engineers build the internal developer platforms that both data engineers and software engineers use. In a data context, they might manage the Kubernetes cluster that Airflow runs on, or build the CI/CD pipelines used to deploy dbt projects.

• Primary Focus: Reducing cognitive load for other engineers by providing stable, self-service platforms.
• Core Skills: Deep infrastructure expertise (Kubernetes, cloud networking), CI/CD tools (Jenkins, GitLab CI), and a product mindset for internal tools.
• Key Question: "How can I provide a paved road for data engineers to deploy a new pipeline with all our standard security, logging, and monitoring baked in?"

What we would NOT do is hire a data scientist and expect them to build production data infrastructure. While they can often build a prototype pipeline in a notebook, they typically lack the software engineering and systems design skills to create something that is reliable, scalable, and maintainable. This mismatch in expectations is a primary reason why so many AI projects remain stuck in the lab.

What good looks like in 90 days

Moving from a "Lagging" to a "Leading" data culture is a journey, but meaningful, value-driven progress can be made in a single quarter. A focused 90-day engagement is not about boiling the ocean; it's about delivering a production-grade data pipeline for one high-impact business problem, establishing a pattern that can be replicated across the organisation.

Here is a typical timeline for a foundational AI engineering project:

Month 1: Audit, Design, and Foundation (Weeks 1-4)

• Activities: Stakeholder interviews to identify a critical business use case (e.g., customer churn prediction, claims summarisation). Audit of existing data sources and infrastructure. Design of a target-state architecture.
• Deliverables: A detailed architecture diagram for the new pipeline. A prioritised backlog of work. Core infrastructure provisioned via code (e.g., a new Snowflake account, dbt project, and GitHub repository).

This target architecture often follows a modern ELT pattern, which prioritises speed and flexibility.

       Source Systems         Ingestion           Storage & Transformation (Lakehouse)       Serving Layer
      ┌────────────────┐     ┌───────────┐         ┌──────────────────────────────┐        ┌──────────────────┐
      │ ERP (SAP)      ├─►───┤ Fivetran  ├───►─────┤                              │────►──┤ BI (Power BI)    │
      └────────────────┘     └───────────┘         │  Snowflake / Databricks      │        └──────────────────┘
                                                   │                              │
      ┌────────────────┐     ┌───────────┐         │ ┌─────────┐   ┌──────────┐   │        ┌──────────────────┐
      │ CRM (Salesforce) ├─►───┤ Airbyte   ├───►─────┤ │ Raw Data│ ► │   dbt    │ ► │────►──┤ RAG AI App       │
      └────────────────┘     └───────────┘         │ └─────────┘   └──────────┘   │        └──────────────────┘
                                                   │   (Tables)     (Models)    │
                                                   └──────────────────────────────┘

This diagram illustrates a clean separation of concerns, a core tenet of modern data pipeline architecture.

Month 2: Build, Test, and Automate (Weeks 5-8)

• Activities: Develop the ingestion pipelines for the chosen sources. Build the dbt transformation models to clean, join, and aggregate the data. Implement data quality tests (e.g., uniqueness, not-null constraints, accepted values) within dbt. Set up orchestration (e.g., Airflow or dbt Cloud) to run the pipeline on a schedule.
• Deliverables: A fully automated, end-to-end data pipeline. A suite of data quality tests that run with every execution. A documented data model in dbt.

Month 3: Deliver Value, Document, and Handover (Weeks 9-12)

• Activities: Connect the new, clean data models to an end application—either a BI dashboard demonstrating a new KPI or a proof-of-concept AI model. Conduct training sessions with the client's team. Finalise documentation.
• Deliverables: A live dashboard or working AI prototype powered by the new pipeline. A runbook for maintaining and extending the pipeline. An empowered client team.

Let's take a second example: a mid-sized European insurance company wanting to use Generative AI to summarise complex claims documents. Their data is locked in three separate legacy systems.

A 90-day project with a budget of €120,000 (e.g., two senior data engineers for three months) could achieve the following:

• Unify and de-duplicate customer and claims data from the three source systems into a single "source of truth" table in a data warehouse.
• Create an API that allows the AI application to securely retrieve all relevant documents and metadata for a given claim ID.
• The business outcome: The average time for a claims handler to get up to speed on a complex claim drops from 60 minutes to 15 minutes. With 20,000 complex claims per year and a fully loaded handler cost of €60/hour, the annual saving is: 20,000 claims * 0.75 hours/claim * €60/hour = €900,000 This demonstrates a clear, compelling ROI on the initial data engineering investment within the first year.

Frequently asked questions

Why does AI need data engineering?

AI models are increasingly becoming commodities, but your proprietary data is your unique advantage. Data engineering is what makes that data usable and trustworthy for AI. The quality of retrieval, enforcement of permissions, freshness of information, and auditable lineage all originate from the data platform. These factors determine whether the output from an AI system is accurate, secure, and ultimately, trustworthy enough to be used in a business context. Without solid data engineering, even the most advanced AI is simply a powerful guess.

Key takeaways

• Data engineering is foundational. It is the disciplined work of building the infrastructure that makes all other data functions, including AI, possible.
• AI multiplies the cost of bad data. Seemingly small data quality issues can lead to significant financial losses, wasted effort, and a complete loss of trust in AI systems.
• The modern data stack is modular. It consists of five key layers: ingestion, storage, transformation, serving, and governance. Understanding these helps focus investment.
• Maturity is measurable. Simple, business-focused questions about data speed, trust, security, and reliability can quickly reveal the health of your data infrastructure.
• Specialised roles are critical. Data engineers, analytics engineers, and platform engineers have distinct skills. Assigning the wrong tasks to the wrong role is a recipe for failure.
• Focus on value in 90 days. Don't try to boil the ocean. A successful data engineering initiative starts by solving one high-impact business problem, creating a repeatable pattern for success.

Investing in AI without first investing in data engineering is like building a skyscraper on sand. The initial progress may seem fast, but the entire structure is at risk of collapse. For many organisations, the first step is a clear-eyed assessment of their current data architecture to understand its readiness for AI. If you are looking to de-risk your AI investments and build a foundation for long-term success, a strategic review can provide the clarity needed to move forward. For a deeper conversation on your specific needs, our AI consulting team is available.