How Companies Prepare Their Data for Generative AI

Generative AI's potential to transform how businesses access and use their internal knowledge is immense. The vision is compelling: an internal assistant that can answer complex questions about operations, customers, and policies by drawing on the company's collective intelligence stored in documents, databases, and applications. Yet, a significant gap exists between this vision and the reality of most corporate data landscapes.

The common assertion, "we have lots of data," is frequently mistaken for readiness. In our engagements, we find that while volume is rarely the issue, the state of that data is almost always the primary obstacle. Raw, unstructured, siloed, and ungoverned data is not an asset for Large Language Models (LLMs); it is a source of risk, inaccuracy, and project failure. Without a deliberate and systematic preparation programme, feeding this data to an LLM is like asking a brilliant but uninformed consultant to give strategic advice after handing them a messy pile of unsorted paperwork.

This article outlines the structured, six-step process we use to create AI-ready data. It's a programme focused on turning digital clutter into a reliable, high-performance knowledge base that can power accurate, secure, and genuinely useful generative AI applications. This is not about exotic algorithms, but about the disciplined practice of data engineering applied to the specific needs of LLMs.

Why 'we have lots of data' is not readiness

Enterprise data is typically fragmented across dozens of systems, each with its own format, permission model, and access patterns. Confluence pages, SharePoint documents, Jira tickets, Salesforce records, and SQL databases all hold valuable pieces of the puzzle. An LLM cannot natively navigate this complexity.

AI readiness is not about data volume. It is about:

• Accessibility: Can the system programmatically access the content?
• Quality: Is the content clean, current, and not contradictory?
• Structure: Is the data broken down into logical, digestible units?
• Context: Is each piece of data enriched with metadata (e.g., source, author, date, permissions)?
• Security: Can the system enforce the exact same access controls as the source system?

Answering "no" to any of these questions introduces risks. An LLM using out-of-date information will give wrong answers. An LLM that cannot parse a PDF will have a knowledge gap. An LLM that bypasses your existing permissions creates a catastrophic security breach. The work of creating AI-ready data is the work of addressing these five dimensions systematically.

Step 1 — Inventory and classify sources

The first step is to create a comprehensive inventory of all potential knowledge sources. You cannot build a pipeline for data you don't know you have. This process involves collaborating with business and IT leaders to map the enterprise data landscape.

For each source, we document its type, format, owner, and strategic value. This allows us to prioritise the integration effort. Not all data is created equal; the goal is to start with sources that offer the highest potential return on the effort required to process them. A company's internal wiki and core policy documents are often much more valuable starting points than an archive of old project plans.

In our experience across various sectors, a prioritisation matrix often looks something like this:

This inventory forms the roadmap for the data integration effort. We would not, for example, recommend starting with network file shares. The effort to extract text and infer meaning from a decade of poorly named Word documents and spreadsheets is enormous, and the value is often questionable. Start with structured, high-value sources like Confluence and SharePoint.

Step 2 — Permissions and row/document-level access

This is the single most critical step for any enterprise generative AI project. A system that allows a junior employee to ask questions and receive answers from the board's confidential M&A documents is not just a failure; it is a disaster.

The core principle is that the AI system must respect the source-of-truth permissions. A user's query should only return results from documents and data they are already authorised to see.

A common but dangerous anti-pattern is to use a single, highly privileged service account to scrape all data, embed it, and load it into a single, monolithic vector index. This effectively creates a "super user" database that bypasses all existing security controls.

The correct approach requires propagating the end-user's identity throughout the query process.

1. User Authentication: The user logs into the AI application via your standard Single Sign-On (SSO) provider (e.g., Azure AD, Okta).
2. Identity Propagation: The application backend receives the user's identity token.
3. Permissions-Aware Retrieval: When the user asks a question, the query to the knowledge base must be filtered based on that user's permissions.

There are two primary models for implementing this filtering:

• Pre-filtering (Separate Indexes): You create separate indexes for each distinct permission group (e.g., "Engineering," "Finance," "Executive"). At query time, you select the appropriate index based on the user's group membership. This is fast at query time but leads to an explosion of indexes, making it complex and costly to manage. It's only viable for companies with a very small number of coarse-grained permission groups.
• Post-filtering (Metadata Filtering): This is the more common and scalable approach. During the indexing process, each chunk of data is tagged with metadata that specifies who can access it (e.g., allowed_users: ["user:123"], allowed_groups: ["group:abc"]). At query time, the search query includes a filter clause that matches the user's identity against this metadata.

For example, a query to an OpenSearch or Pinecone index would include a vector query for semantic matching and a metadata filter like {"term": {"allowed_groups": "engineering-dept"}}.

Implementing this correctly is non-trivial. It requires careful mapping of source system ACLs (Access Control Lists) to a standardised metadata format and ensuring your indexing pipeline can reliably apply these tags. This is a foundational element of building a secure and compliant modern data platform.

Step 3 — Normalisation, chunking and metadata

Once a document is accessed and its permissions are understood, it must be processed for the LLM. Raw documents, especially binary formats like PDF and DOCX, are not directly usable. This stage involves three key activities: normalisation, chunking, and metadata enrichment.

Normalisation

This is the process of converting all source documents into clean, plain text. For a PDF, this involves OCR (Optical Character Recognition) and layout analysis to extract text in a logical reading order. For a Confluence page, it means parsing HTML and stripping out irrelevant UI elements. For a spreadsheet, it could mean serialising each row into a descriptive sentence. The goal is a consistent text format, regardless of the source.

Chunking

LLMs have a finite context window—the amount of text they can consider at once. You cannot feed a 200-page document into a prompt. The document must be split into smaller, semantically meaningful "chunks."

The quality of chunking has a direct impact on the quality of retrieval.

• Poor Chunking: A fixed-size chunk that splits a sentence or table mid-way provides incomplete context.
• Good Chunking: A chunk that corresponds to a single paragraph, a section with its heading, or a table with its caption provides coherent, self-contained context.

There is no single best chunking strategy; it depends on the source data's structure.

• Fixed-Size: The simplest method (e.g., 1000 characters per chunk with 100 characters of overlap). It's fast but naive.
• Recursive: A multi-pass approach that tries to split on paragraphs, then sentences, then words, to maintain semantic boundaries. This is a good general-purpose starting point.
• Semantic/Content-Aware: The most effective method. It uses the document's structure (e.g., Markdown headings, HTML tags, document outlines) to create chunks. For example, a chunk might be a single section of a technical document, from ### Sub-heading to the next. This requires more sophisticated parsing logic but yields superior results.

Metadata Enrichment

Every chunk must be treated as a database record, not just a blob of text. It should be stored with a rich set of metadata that provides essential context for retrieval, filtering, and citation.

A minimal set of metadata for each chunk should include:

• document_id: A unique identifier for the source document.
• source_uri: A direct link back to the original document or page.
• source_type: e.g., 'Confluence', 'SharePoint', 'Jira'.
• chunk_id: The sequence number of the chunk within the document.
• created_at, modified_at: Timestamps from the source document.
• permissions: The access control metadata discussed in Step 2.
• title, author: Document-level metadata.

This metadata is not optional. The source_uri is essential for providing citations. The timestamps are critical for managing freshness. The permissions are non-negotiable for security.

Step 4 — Embeddings, indexes and hybrid search

With data chunked and enriched, the next step is to make it searchable. Modern enterprise search for AI goes beyond traditional keywords. This is where Retrieval-Augmented Generation (RAG) comes in. RAG is a design pattern where the LLM's knowledge is augmented at query time with relevant information retrieved from your private knowledge base.

The core components of a RAG retrieval system are embeddings and indexes.

• Embeddings: An embedding is a vector (a list of numbers) that represents the semantic meaning of a piece of text. An embedding model (e.g., from OpenAI, Cohere, or open-source models like Sentence-Transformers) is used to convert each text chunk into a vector. Chunks with similar meanings will have vectors that are "close" to each other in multi-dimensional space.
• Vector Index: A specialised database, often called a vector database (e.g., Pinecone, Weaviate, or features within PostgreSQL/OpenSearch), is used to store these vectors. It can efficiently find the "nearest neighbours" to a query vector, enabling semantic search.

However, relying solely on vector search is a common mistake. Vector search excels at finding conceptually similar results but can struggle with queries that depend on specific keywords, product codes, or acronyms.

For example, a user query for "Project Firefly Q3 report" might be missed by a pure vector search if the term "Firefly" is uncommon in the training data, but a keyword search would find it instantly.

This is why we almost always implement hybrid search. Hybrid search combines the results from a traditional keyword search (like BM25) and a modern vector search, fusing the results to get the best of both worlds.

A typical hybrid search pipeline looks like this:

User Query -> [Query Transformation] -> (Keyword Query) + (Vector Query)
                                             |                 |
                                             v                 v
                                     [Keyword Index]     [Vector Index]
                                     (e.g., BM25)        (e.g., HNSW)
                                             |                 |
                                             v                 v
                                     [Candidate Chunks]  [Candidate Chunks]
                                             |                 |
                                             +------[Fusion]-----+
                                                      |
                                                      v
                                              [Ranked Chunks]
                                                      |
                                                      v
                                 [Prompt Engineering: Context + Query]
                                                      |
                                                      v
                                                    [LLM] -> [Answer] + [Citations]

This architecture ensures that the LLM is fed the most relevant possible context, dramatically reducing the likelihood of "hallucinations" and improving the factual accuracy of the answer.

Step 5 — Freshness, deletion and right-to-be-forgotten

A knowledge base is only as good as it is current. Data in an enterprise is constantly changing: wiki pages are updated, policy documents are superseded, project statuses change. A robust data pipeline must be designed to handle these changes efficiently.

This is fundamentally a data engineering challenge. The goal is to move from slow, nightly batch updates to a more responsive, event-driven architecture.

• Batch Processing: The simplest approach. The system re-scrapes and re-indexes all sources on a schedule (e.g., every 24 hours). This is easy to build but results in high data latency. An answer might be based on yesterday's information, which is unacceptable for many use cases.
• Event-Driven Updates: A superior approach. The source systems (e.g., Confluence, SharePoint) emit events (via webhooks or message queues) whenever a document is created, updated, or deleted. These events trigger a serverless function or a microservice that processes only the changed document, updating the index in near real-time. Designing such systems is a core competency; you can learn more about the patterns in our guide to data pipeline architecture.

A real-world example illustrates the importance of this. For a European insurer we worked with, claims handlers needed instant access to the latest policy updates. Their legacy search tool updated overnight. A change to a claims handling procedure made at 9 AM would not be reflected until the next day, creating a risk of processing claims incorrectly. An event-driven pipeline we helped design, listening to SharePoint webhooks, reduced this data latency from 24 hours to under two minutes.

Furthermore, the pipeline must handle deletions. This is not just for freshness but also for compliance. Regulations like GDPR grant individuals the "right to be forgotten." If a document containing personal data is deleted from the source system, it must also be purged from the AI's knowledge base. An event-driven pipeline can receive a document_deleted event and trigger a process to remove all associated chunks and vectors from the search index.

Step 6 — Measuring retrieval quality before building UI

How do you know if your data preparation and retrieval system is any good? You measure it.

A common failure pattern is for teams to spend months building a beautiful chat interface on top of a mediocre retrieval system. Users are initially impressed by the UI, but quickly become frustrated by irrelevant or inaccurate answers. The project then gets stuck in a cycle of random prompt tuning, trying to fix a problem whose root cause is poor data retrieval.

We insist on a "measure first" approach. Before any significant front-end development, we establish a quantitative benchmark for the quality of the RAG retrieval system.

1. Create an Evaluation Set: Working with business stakeholders, we create a "golden set" of representative questions. For each question, a subject matter expert identifies the exact source document(s) or chunk(s) that contain the correct answer. This set might contain 50-200 question/answer pairs.
2. Define a Metric: We use standard information retrieval metrics to score our system's performance against this golden set. Common metrics include:
   • Hit Rate: For a given question, was the correct chunk present in the top K retrieved results (e.g., in the top 5)?
   • Mean Reciprocal Rank (MRR): Measures how high up the list the first correct chunk appears. Higher is better.
   • Normalized Discounted Cumulative Gain (NDCG): A more sophisticated metric that rewards systems for retrieving multiple relevant chunks and for ranking them highly.
3. Iterate and Improve: We run the evaluation set against our retrieval pipeline and calculate the scores. The initial scores are often lower than expected. This is normal. The scores provide a baseline. We can then systematically experiment with different chunking strategies, embedding models, and hybrid search configurations, re-running the evaluation after each change to see if the score improves.

For a Series A logistics platform, we established a golden set of 50 common carrier compliance questions. The business goal was to enable support staff to answer customer queries without escalating to the legal team. We set an initial target of Hit Rate@5 > 90%. The first implementation, using a simple chunking strategy, scored only 65%. By iterating on a content-aware chunking strategy for their legal documents and implementing a tuned hybrid search, we improved the score to 92%. This data-driven result gave the leadership team the confidence to invest in building the full user-facing application.

Governance and audit for regulated industries

For companies in regulated sectors like finance, insurance, and healthcare, simply providing an answer is not enough. The system must be able to prove why it gave that answer. This requires strong governance and auditability features built into the core architecture.

The key is traceability. For every single response generated by the LLM, the system must log and be able to present:

1. The original, verbatim user query.
2. The exact set of document chunks retrieved from the knowledge base.
3. Direct links (citations) back to the source documents for each of those chunks.
4. The final prompt sent to the LLM (which includes the retrieved context).
5. The raw response from the LLM.

This audit trail is not a "nice-to-have." It is a requirement for demonstrating compliance, investigating errors, and defending the system's decisions to auditors or regulators. When a user asks, "What is our liability in scenario X?" and the AI gives an answer, the business must be able to trace that answer back to a specific clause in a specific, version-controlled legal document.

Our approach to AI engineering for enterprise clients prioritises this traceability from day one. It influences the choice of databases, the logging infrastructure, and the API design. A system without this capability is unsuitable for any serious enterprise use case where accuracy and accountability are paramount.

Frequently asked questions

How do we make our documents usable by an LLM?

To make documents usable, you must first convert them from their native format (like PDF, DOCX, or HTML) into clean, plain text. Then, these long texts must be broken down into smaller, semantically coherent chunks. Each chunk should be enriched with metadata, including its source, author, and critically, the permissions that govern who can see it. Finally, these chunks are indexed using a hybrid approach that combines traditional keyword search with modern vector search to ensure they can be accurately retrieved in response to a user's query.

How much does it cost to prepare data for GenAI?

The cost varies with the number and complexity of data sources, but a focused proof-of-concept is a manageable investment. For example, a 3-month project to build an AI-ready knowledge base from a primary source like Confluence (approx. 50,000 pages) can be estimated. Assuming a senior Golux team of two engineers, the engagement cost would be around €150,000. Cloud infrastructure costs for this phase are typically modest, often under €2,000 per month. This investment validates the entire technical and business approach before a larger, more costly rollout.

Can we just connect ChatGPT to our data?

Directly connecting a public LLM like ChatGPT to a raw data source is not feasible and highly insecure. The LLM has no way to navigate your internal systems, parse proprietary formats, or respect your company's access control policies. The process described in this article—building a secure retrieval pipeline that fetches relevant, permission-checked information and provides it to the LLM as context within a prompt—is the standard, secure architecture for this type of application. This RAG pattern ensures the LLM's response is grounded in your data and respects your security.

Key takeaways

• Readiness is not volume. AI-ready data is clean, accessible, structured, secure, and enriched with metadata. Raw data is a liability.
• Permissions are non-negotiable. The AI system must enforce the same document- and row-level permissions as the source systems. Failure to do so is a critical security breach.
• Chunking quality determines retrieval quality. Invest time in content-aware chunking strategies over naive, fixed-size approaches.
• Use hybrid search. Combining keyword and vector search provides the best of both worlds, balancing keyword precision with semantic recall for more accurate retrieval.
• Measure retrieval before building UIs. Establish a quantitative evaluation set and metrics (like Hit Rate or NDCG) to prove your retrieval system is effective before investing in front-end development.
• Prioritise governance and traceability. For any serious enterprise use case, the system must be able to cite its sources and provide a complete audit trail for every answer it generates.

Preparing your enterprise data for generative AI is the foundational work that separates successful, high-value AI applications from disappointing science projects. It requires a systematic approach that blends data engineering discipline with a deep understanding of how language models process information.

If you are planning an enterprise generative AI initiative and need to ensure your data foundation is solid, our senior teams can provide a detailed architecture assessment to de-risk your project and accelerate your path to production. Please get in touch to discuss how our AI consulting can help you build with confidence.