Generative AI Development: From Idea to Production

Pilot purgatory: Why many GenAI projects never reach production

A compelling generative AI demo is seductively easy to build. In a few days, a single engineer can chain together a large language model (LLM) API and a vector database to create a chatbot that answers questions over a few PDF documents. The results look magical. Stakeholders are impressed. A pilot is launched.

And then, nothing.

Across our engagements, we observe a consistent pattern: a significant number of generative AI initiatives stall after the initial prototype. They enter a state of "pilot purgatory"—too promising to cancel, but too fragile and unpredictable to release to real users. The demo that worked 90% of the time on ten curated documents fails spectacularly on a thousand real-world files. The costs are unclear, the failure modes are alarming, and the path to a robust, production-grade system is ambiguous.

This happens because building a production AI system is not about finding the perfect prompt. It is a rigorous software engineering discipline. It requires a systematic approach to data, evaluation, safety, cost management, and operations. The "magic" of the prototype must be replaced by the measurable reliability of a production service.

This guide is our playbook for navigating that journey. It outlines the six essential stages for taking a generative AI application from a promising idea to a production system that delivers durable business value.

Stage 1: Problem framing and value hypothesis

The most common failure mode for any technology project is solving a problem that doesn't matter. With generative AI, the allure of the technology can often obscure the business case. The starting point must not be "let's use an LLM," but "what is a high-value, narrowly-defined problem we can solve with it?"

We guide our clients to frame the initiative around a clear value hypothesis. This is a simple, falsifiable statement that connects the AI capability to a business outcome. The structure is:

If we can [achieve a specific AI capability], then we will [produce a specific business outcome], which we will measure by [a specific, quantifiable metric].

Let's consider two examples from our work.

Example 1: A European insurer

• Hypothesis: If we can automatically summarise complex claims histories and highlight non-standard clauses, then we will reduce the time claims adjusters spend on manual review, which we will measure by a 15% reduction in average claims handling time.

Example 2: A Series B B2B SaaS platform

• Hypothesis: If we can provide an AI-powered chat interface that answers user questions using our technical documentation, then we will improve user self-service, which we will measure by a 20% reduction in support tickets related to "how-to" questions.

This framing does three crucial things:

1. It defines success: The metric (handling time, ticket volume) becomes the north star for the project. Every technical decision should be weighed against its impact on this metric.
2. It scopes the work: The capability is specific ("summarise claims histories," not "revolutionise insurance"). This focuses the initial effort on a manageable slice of the problem. Many clients find it useful to explore what's possible when building custom AI solutions before committing to a specific path.
3. It creates a business case: The hypothesis directly links the engineering effort to a tangible financial or operational outcome, justifying the investment.

What we advise against is broad, undefined goals like "improve productivity" or "enhance customer experience." These are laudable aims, but they are not actionable engineering targets. A successful AI engineering project starts with a precise definition of the job to be done.

Stage 2: Data access, permissions, and retrieval design

For most enterprise use cases, the power of generative AI is not in the model's pre-trained knowledge, but in its ability to reason over your proprietary data. The dominant architectural pattern for this is Retrieval-Augmented Generation (RAG). In a RAG system, the user's query is first used to retrieve relevant information from a private knowledge base; that information is then "augmented" into the prompt sent to the LLM, giving it the context it needs to formulate a precise, factual answer.

Building the RAG pipeline is where most of the real engineering work lies. It is fundamentally a data engineering challenge.

                     +---------------------------+
User Query --------> |   1. Input Processing     |
                     | (Guardrails, Intent Det.) |
                     +-------------+-------------+
                                   |
                                   v
+------------------+     +---------------------+     +--------------------------+
| 2. Query         | --> | 3. Vector Database  | --> | 4. Retrieve Top-K Chunks |
|    Embedding     |     | (e.g., Pinecone)    |     +--------------------------+
+------------------+     +---------------------+
                                   |
+----------------------------------+----------------------------------+
| 5. Permissions & Re-ranking                                         |
|    - Filter chunks user is not allowed to see                       |
|    - Re-rank for relevance                                          |
+----------------------------------+----------------------------------+
                                   |
                                   v
+----------------------------------+----------------------------------+
| 6. Prompt Construction                                              |
|    - System Prompt (Instructions, Persona)                          |
|    - Retrieved Chunks (Context)                                     |
|    - User Query                                                     |
+----------------------------------+----------------------------------+
                                   |
                                   v
                     +-------------+-------------+
                     |    7. LLM Call            |
                     | (e.g., Claude, GPT-4o)    |
                     +-------------+-------------+
                                   |
                                   v
+------------------+     +---------------------+
| 8. Output        | --> | 9. Final Response   |
|    Processing    |     | to User             |
|  (Guardrails)    |     +---------------------+
+------------------+

The diagram above illustrates a production-grade RAG pipeline. Key considerations here include:

Data Ingestion and Chunking

Your raw data (from Confluence, SharePoint, Salesforce, etc.) is rarely in an LLM-friendly format. It needs to be extracted, cleaned, and broken into smaller "chunks."

• Chunking Strategy: Simply splitting documents every 1,000 characters is a poor strategy. You need semantic chunking that respects document structure (paragraphs, sections, tables). The quality of your retrieval depends heavily on the quality of your chunks.
• Embedding: Each chunk is converted into a vector embedding—a numerical representation of its semantic meaning—and stored in a vector database. The choice of embedding model is a critical, and often overlooked, decision that impacts relevance and cost.

Permissions

This is the single biggest blocker we see in enterprise projects. An LLM application must respect user-level data permissions. A junior analyst asking about company financials should not see the same documents as the CFO.

Implementing this is non-trivial. The simplest approach is post-retrieval filtering: retrieve a set of candidate chunks from the vector database, then check each one against an access control list (ACL) service to see if the current user has permission. This is secure but can be inefficient if many of the top-retrieved documents are filtered out. More advanced techniques involve storing permissions metadata alongside the vectors themselves.

Retrieval Strategy

A simple vector search is a good starting point, but production systems often require more sophistication. This might include:

• Hybrid Search: Combining semantic (vector) search with traditional keyword (lexical) search to get the best of both worlds.
• Re-ranking: Using a secondary, lightweight model to re-rank the initial set of retrieved documents for relevance to the specific query before passing them to the main LLM. This can improve quality and reduce the number of tokens sent to the expensive model.

Stage 3: Prototype with a scoring harness from day one

How do you know if changing a prompt, a chunking strategy, or an LLM made the system better or worse? In a traditional software, you have unit tests and integration tests. In generative AI, the equivalent is an evaluation harness.

Building without an evaluation harness is like coding without a compiler. Every change is based on gut feel and a few anecdotal examples. This is unsustainable and unsafe for a production system.

An evaluation harness consists of three components:

1. The "Golden Set": A curated, version-controlled set of test cases. This includes representative user queries and, for some tests, the ideal or "golden" answers. This set should cover typical use cases, edge cases, and known failure modes.
2. The Metrics: A suite of scoring functions that measure the quality of the model's output for each test case.
3. The Runner: An automated script that runs every query in the golden set through the AI system and calculates the scores for all metrics.

The results are tracked over time. When you make a change, you run the evaluation harness. If the scores improve, you can merge the change with confidence. If they regress, you know you've broken something.

Common Evaluation Methods

Choosing the right metrics is key. A single metric is never enough; you need a balanced scorecard.

In practice, a production-grade harness uses a combination of these. For example: rule-based checks for safety, embedding distance for semantic similarity on core use cases, and LLM-as-Judge for overall quality assessment.

Stage 4: Guardrails, red-teaming, and failure handling

A production system will be pushed in ways you don't expect. Users will ask inappropriate questions, try to jailbreak the system, or input malformed data. You must design for this reality. This involves layers of defense, often called "guardrails."

• Input Guardrails: These check the user's query before it's processed. They can block prompts containing PII, toxic language, or known prompt injection attacks. A simple but effective input guardrail is to use a fast, cheap model to classify the user's intent and block anything that falls into a forbidden category.
• Output Guardrails: These check the LLM's response before it's shown to the user. They can scan for hallucinations (by checking if the output is grounded in the provided context), PII that the model might have leaked, or toxic content. A common technique is to ask a second LLM: "Based on the provided sources, is the following statement factually correct?"
• Content Moderation APIs: Services like Azure Content Safety or OpenAI's Moderation endpoint provide dedicated, fine-tuned models for detecting categories of harmful content (hate, self-harm, etc.) and are an essential part of any public-facing application.

Red-Teaming

Beyond automated guardrails, you need a human-in-the-loop process to actively try to break the system. This is red-teaming. A dedicated team (internal or external) should be tasked with systematically attempting to:

• Bypass safety filters.
• Elicit biased or harmful responses.
• Trick the system into revealing confidential information.
• Discover queries that cause it to hallucinate wildly.

The findings from red-teaming sessions are invaluable. They are used to expand the evaluation harness's "golden set" with new failure modes and to refine the guardrails and system prompts.

Graceful Failure

What happens when a guardrail is triggered or the system can't find a good answer? The worst possible response is to generate a plausible-sounding but incorrect answer. A production system must know when to say "I don't know."

Design explicit failure pathways. For instance, if the retrieval mechanism returns no relevant documents, or if the retrieved documents have a low relevance score, the system should bypass the LLM call entirely and respond with a pre-canned message like: "I could not find any information about that in my knowledge base. Could you rephrase your question?" This is far preferable to a confident hallucination.

Stage 5: Cost, latency, and model routing

Once the system is functionally correct and safe, the focus shifts to operational performance: cost and speed. A system that costs €5 per query or takes 10 seconds to respond is not viable for most use cases.

Understanding the Cost Structure

The costs of a GenAI application come from multiple sources:

1. LLM API Calls: Priced per token (input and output). Input tokens are often cheaper than output tokens.
2. Embedding Model API Calls: Priced per token used to create embeddings for new documents and user queries.
3. Vector Database: Typically priced on a combination of data volume and compute/instance hours.
4. Orchestration Compute: The server or serverless function running the application logic (the RAG pipeline itself).

Worked Example: Costing an Internal Knowledge Base Chatbot

Let's model the costs for an AI chatbot used by 500 employees at a European professional services firm. We'll use realistic 2026 pricing estimates.

• Usage: 10,000 queries per month.
• Data: Average query is 50 tokens. The RAG pipeline retrieves 4,000 tokens of context. The LLM generates a 250-token response.
• Model Choice: A balanced Tier 2 model (e.g., Claude 3.5 Sonnet) at €3.00 per million input tokens and €15.00 per million output tokens.
• Embedding Model: A mid-range model at €0.10 per million tokens.
• Vector DB: A managed service costing €250/month.
• Orchestration: A serverless function with negligible cost for this volume.

Calculation per query:

• Input tokens to LLM: 50 (query) + 4,000 (context) = 4,050
• Input cost: (4,050 / 1,000,000) * €3.00 = €0.01215
• Output tokens from LLM: 250
• Output cost: (250 / 1,000,000) * €15.00 = €0.00375
• Embedding cost (for the query): (50 / 1,000,000) * €0.10 = €0.000005 (negligible)
• Total API cost per query: €0.01215 + €0.00375 = €0.0159

Total monthly cost:

• API Costs: 10,000 queries * €0.0159/query = €159
• Vector DB: €250
• Total Monthly Operating Cost: €159 + €250 = €409

This kind of analysis is crucial for building a business case and avoiding budget surprises. You can find a more detailed breakdown in our guide on how much it costs to build an AI application.

The Model Cascade

A powerful technique for managing both cost and latency is a model cascade or router. The idea is to use the cheapest, fastest model that can handle the task.

1. First, a fast, cheap "router" model classifies the user's query. Is it a simple question? A complex one requiring reasoning?
2. Simple queries are sent to a cheap, low-latency model (e.g., Claude 3.5 Haiku).
3. Complex queries are sent to a powerful, more expensive model (e.g., GPT-4o or Claude 3 Opus).

This layered approach ensures a snappy user experience for the majority of queries while reserving the expensive, powerful models for the minority of tasks that truly require them.

Model Selection Trade-offs (Illustrative 2026 Models)

Stage 6: Release, monitoring, and drift response

Releasing a generative AI application is not a single event. A "big bang" launch is too risky. We recommend a phased rollout:

1. Internal Dogfooding: Release to the project team and a small group of internal power users.
2. Internal Beta: Release to a wider internal audience, perhaps a specific department.
3. Canary Release: Release to a small percentage (e.g., 1-5%) of real, external users.
4. Gradual Rollout: Slowly increase the percentage of users who have access, while closely monitoring performance.

Monitoring

Your monitoring dashboard needs to go beyond CPU and memory. For a GenAI app, you must track:

• Quality Metrics: Track user feedback (e.g., thumbs up/down clicks on responses), as well as automated scores from your evaluation harness on a sample of production traffic. Are the responses still helpful?
• Performance Metrics: p50, p90, and p95 latency. How fast is the system, really?
• Cost Metrics: Token consumption per query and total API spend, broken down by model.
• Behavioral Metrics: Are guardrails being triggered? How often does the system refuse to answer? What are the most common topics users are asking about?

Responding to Drift

The world changes. Your data changes. The way users interact with your system changes. This is "drift," and it will degrade your system's performance over time.

• Data Drift: The underlying data in your knowledge base is updated. Old information becomes stale. This requires a robust data ingestion pipeline that can automatically re-index and re-embed updated documents.
• Concept Drift: Users start asking about new topics or in new ways that the system wasn't designed for. This is where monitoring user queries is critical. When you spot a new pattern of questions where the system is performing poorly, those queries need to be added to your evaluation "golden set," and you may need to update prompts or retrieval strategies to handle them.

Drift management is an ongoing process of monitoring, evaluating, and re-tuning.

Operating model: Who owns the system after launch?

A generative AI system is not a traditional software project that you can hand off to a maintenance team. It's a dynamic system that requires continuous ownership and improvement. The biggest mistake we see companies make is failing to plan for this.

A successful operating model requires a dedicated, cross-functional team, even if it's small. This team typically includes:

• Product Owner: Owns the roadmap, prioritises improvements based on user feedback and business value, and manages the backlog.
• Lead Engineer / AI Engineer: Owns the technical architecture, monitors performance, and implements changes to the pipeline (prompts, models, retrieval logic).
• Data Engineer / Analyst: Owns the data ingestion pipelines, monitors data quality, and analyses user interaction logs to identify patterns and drift.

This team is responsible for the feedback loop: monitoring production metrics, identifying areas for improvement, updating the evaluation harness, experimenting with changes, and safely deploying them. This is how we structure our partnerships with clients; how we work is to build the system and empower the team that will own it long-term. Choosing the right partner is key, as they should be able to guide you in setting up this long-term operational structure. For more on this, consider reading about what defines a good AI software development company.

Frequently asked questions

How long does it take to move a GenAI prototype into production?

For a well-defined use case with clean data access, we typically see a 6–12 week timeline from a validated prototype to a v1 production release. The critical path is rarely the core LLM work. The longest poles in the tent are almost always securing data permissions in a corporate environment, building robust data ingestion pipelines, and creating a comprehensive evaluation harness. The prompt engineering itself is often the fastest part of the process once this foundation is in place.

What is an evaluation harness and why is it mandatory?

An evaluation harness is an automated system for testing the quality of your AI application. It consists of a versioned set of test cases (the "golden set") and a suite of metrics that score the AI's outputs for things like accuracy, relevance, and safety. It is mandatory because without it, you are flying blind. Every change to a prompt, model, or data source is just a guess. The harness provides objective, repeatable evidence that a change has made the system better (or worse), enabling you to iterate quickly and safely, and forming the backbone of your CI/CD process for AI.

Key takeaways

• Start with a narrow business problem. Frame your project with a clear hypothesis connecting an AI capability to a measurable business outcome.
• Production GenAI is a data engineering problem. The majority of the work is in building a robust RAG pipeline, with a particular focus on data quality and permissions.
• Build an evaluation harness from day one. You cannot improve what you cannot measure. Automated scoring against a golden set of test cases is non-negotiable for production.
• Design for failure. Implement layered guardrails, conduct adversarial red-teaming, and ensure the system knows when to say "I don't know" instead of hallucinating.
• Actively manage cost and latency. Use model cascades and rigorous cost analysis to ensure the system is economically viable and provides a good user experience.
• Plan for day two. A GenAI system is a dynamic product, not a static project. It requires a dedicated, cross-functional team to monitor, manage drift, and continuously improve it after launch.

Moving a generative AI application from a flashy demo to a reliable production service is a significant engineering challenge. It requires a shift in mindset from prompt-tinkering to systematic, metrics-driven development. The path is complex, but by following a structured process, you can navigate pilot purgatory and deliver real, lasting value.

If you are planning a generative AI initiative and need guidance on establishing a production-ready architecture, our senior teams can provide a thorough assessment and a clear roadmap.