Large Language Models Explained for Business Leaders

Large Language Models (LLMs) have moved from research labs to boardrooms in record time. As a leader, you are likely being asked to formulate a strategy, approve a budget, or assess the competitive threat posed by this technology. The discourse is often clouded by hype, technical jargon, and unrealistic claims.

This guide is designed for a business audience. It explains how these models work in practical terms, what they can and cannot do, how their use is priced, and what governance is required for safe deployment. We will not use complex mathematics, but we will use numbers, because successful AI adoption is an economic and strategic exercise, not just a technical one.

How a large language model actually works

At its core, an LLM is a highly advanced pattern-matching system. Forget notions of consciousness or understanding; it is more helpful to think of an LLM as a system for predicting the next most probable word in a sequence.

Imagine you were given the sentence fragment: "The capital of France is". Your brain, trained on a lifetime of information, immediately suggests "Paris". An LLM does the same, but on a planetary scale. It has been trained on a vast library of text and code from the public internet, books, and other sources. Through this training, it has built a complex statistical model of how words, sentences, and concepts relate to one another.

When you give an LLM a prompt (e.g., "Write a short email to my team about the new quarterly targets"), it performs this next-word prediction process repeatedly:

1. It starts with your prompt.
2. It calculates the most probable word to follow.
3. It appends that word to the sequence.
4. It treats this new, longer sequence as its input and repeats the process, generating the next word.

This loop continues until it reaches a natural stopping point or a predefined length limit. The "magic" is that by simply predicting the next most likely word over and over, the model can draft emails, summarise documents, write computer code, and translate languages. It is a powerful demonstration of how simple rules, applied at immense scale, can produce complex and useful behaviour.

The key takeaway is that the model does not understand that Paris is a city or that France is a country. It only "knows" that in the statistical universe of the text it was trained on, the word "Paris" is overwhelmingly the most probable token to follow "The capital of France is". This distinction is the single most important concept for understanding both the capabilities and the limitations of LLMs.

Capability map: what they are good and bad at

Because LLMs are probabilistic text generators, they excel at tasks that involve manipulating, summarising, or transforming language in ways that are statistically common. They struggle with tasks that require genuine reasoning, precise calculation, or adherence to facts not present in their training data.

In our engagements, we have observed a clear pattern of strengths and weaknesses. Attempting to use an LLM for a task in the "poor fit" column often leads to unreliable outputs, frustrated users, and failed projects.

A successful AI engineering strategy focuses on applying LLMs to the "Strong Fit" column, while using other tools—or human experts—for the "Poor Fit" column. Many high-value business applications, such as internal knowledge assistants, chain together multiple "Strong Fit" tasks to create a powerful composite capability.

Context windows, tokens and what you pay for

Using a commercial LLM via an Application Programming Interface (API) is a metered service, much like electricity or cloud computing. The fundamental unit of consumption is the token.

A token is a piece of a word. For English text, 100 tokens roughly corresponds to 75 words. "Large Language Model" is three words, but four tokens: "Large", "Language", "Mod", "el". Every piece of text you send to the model (the prompt) and every piece of text it generates (the completion) is measured in tokens.

The context window is the model's short-term memory. It is the maximum number of tokens (prompt + completion) that the model can consider at one time. Early models like GPT-3 had a context window of 2,048 tokens. Modern frontier models have context windows of 128,000 tokens or even millions, allowing them to process entire books or long reports in a single request.

You pay for the tokens you use. Pricing is typically split between input tokens (your prompt) and output tokens (the model's response), with output tokens often being more expensive.

Worked Example: Costing a Support Ticket Summarisation

Let's imagine a European insurer wants to use an LLM to summarise complex customer support conversations to reduce handling time for senior agents.

• Scenario: A customer has a long email exchange with a junior agent. The entire thread is 2,000 words. The insurer wants the LLM to produce a 150-word summary of the issue, the steps taken, and the customer's current sentiment.
• Token Calculation:
  • Input text: 2,000 words ≈ 2,667 tokens.
  • System instruction prompt (e.g., "You are an expert insurance agent summariser..."): 100 words ≈ 133 tokens.
  • Total Input: 2,667 + 133 = 2,800 tokens.
  • Total Output: 150 words ≈ 200 tokens.
• Costing (Projected 2026 API pricing for a high-quality model):
  • Input price: €3.00 per million tokens.
  • Output price: €10.00 per million tokens.
• Calculation:
  • Input cost: (2,800 / 1,000,000) * €3.00 = €0.0084
  • Output cost: (200 / 1,000,000) * €10.00 = €0.0020
  • Total cost per summary: €0.0104, or about 1 euro cent.

If the insurer processes 5,000 such tickets per day, the daily cost would be 5,000 * €0.0104 = €52. The annual cost would be approximately €19,000. This can then be compared to the value of the senior agent time saved to calculate a clear return on investment.

Hallucination, grounding and why RAG exists

The single biggest risk in deploying LLMs in an enterprise context is hallucination. This is when the model generates text that is plausible, fluent, and completely false.

Hallucinations are not a bug or a malfunction. They are a direct consequence of how LLMs work. The model is always just predicting the next most probable word. Sometimes, the most statistically likely sequence of words does not correspond to reality. The model will state a fabricated "fact" with the same confidence it uses for a correct one, because it has no concept of truth, only of probability.

For a creative brainstorming partner, this is a feature. For an application that gives financial advice or medical information, it is a critical failure. The solution is grounding.

Grounding is the process of forcing the LLM's output to be based on a specific set of reliable information that you provide. The most common and effective technique for this is Retrieval-Augmented Generation (RAG).

A RAG system turns the LLM from a know-it-all oracle into a skilled reading-comprehension engine. Instead of asking the model a question and hoping it knows the answer from its training data, you first retrieve relevant, factual documents from your own knowledge base (e.g., SharePoint, Confluence, a product database) and then instruct the model to answer the question based only on the documents provided.

Here is a simplified flow of a RAG architecture:

       User Query
           |
           v
+----------------------+
| 1. Retrieval         |  <-- Your private knowledge base
| (Finds relevant docs)|      (e.g., product manuals, HR policies)
+----------------------+
           |
           v
+----------------------+
| 2. Augmentation      |
| (Combine query +    |
|  docs into a prompt) |
+----------------------+
           |
           v
+----------------------+
| 3. Generation        |  <-- Large Language Model (API or self-hosted)
| (LLM answers based   |
|  on provided text)   |
+----------------------+
           |
           v
      Grounded Answer
(with citations to sources)

This approach dramatically reduces hallucinations because the model is no longer required to recall facts from its vast, static training data. It is simply tasked with synthesising an answer from a small, curated, and up-to-date set of information provided in the prompt. This is why building systems with a solid RAG architecture is fundamental for creating AI applications that use your own data. It is the primary method we use when building reliable internal assistants for our clients.

Open weights versus frontier APIs

There are two primary ways to integrate LLM capabilities into your business. You can use a commercial API from a provider like OpenAI, Anthropic, or Google, or you can host an "open-weights" model (often inaccurately called "open source") on your own infrastructure.

This is one of the most significant strategic decisions you will make. It is analogous to choosing between a SaaS product and building an in-house solution. Neither is inherently better; the right choice depends on your specific requirements for performance, cost, control, and talent.

Worked Example: TCO of an Internal Q&A Bot

A Series B logistics platform wants to build an internal bot to answer complex questions about its 200+ microservices. They anticipate 100,000 queries per month from their 150 engineers.

• API-based Approach:

  • Average query/response size: 4,000 tokens.
  • Monthly tokens: 100,000 queries * 4,000 tokens/query = 400 million tokens.
  • Blended cost (projected 2026): €5 per million tokens.
  • Monthly API cost: €2,000.
  • Development team: 2 senior software engineers.
  • 3-Year TCO: (2 engineers * €120,000/yr * 3 yrs) + (€2,000/mo * 36 mo) = €720,000 + €72,000 = €792,000

• Open-Weights Approach:

  • Hardware: Requires a cluster of high-end GPUs. Upfront cost: €150,000. 3-year hosting cost: €90,000.
  • Team: Requires 2 ML engineers and 1 MLOps specialist, who are more expensive.
  • 3-Year TCO: (€150,000 + €90,000 hardware) + (3 engineers * €150,000/yr * 3 yrs) = €240,000 + €1,350,000 = €1,590,000

In this scenario, the API-based approach is significantly more cost-effective. The open-weights option would only make sense if data privacy was an absolute blocker (e.g., data could not leave the EU), or if the company planned to build a deeply customised model that would become a core part of its product offering. Most companies begin with APIs and only consider hosting open-weights models as their scale and sophistication grow. The choice between these paths is a critical part of any AI strategy, and a core focus of our AI consulting engagements.

Governance: policy, logging, data handling

Deploying LLMs without a governance framework is like giving employees company credit cards with no spending policy or expense reports. It is a recipe for data leaks, unexpected costs, and inconsistent, unreliable outputs. A robust governance plan has three pillars.

### Data Handling and Privacy

Your first policy must define what data is permitted to be sent to an LLM, especially a third-party API. As a baseline, Personally Identifiable Information (PII) and sensitive corporate intellectual property should be forbidden. Anonymisation and data-masking techniques can be implemented in the application layer before the data is sent to the model. For RAG systems, access control is paramount. The system must respect the source document's permissions. If a user cannot access a file in SharePoint, the RAG application must not show them content from that file.

### Acceptable Use Policies

Employees need clear guidelines on how to use company-sanctioned AI tools. These policies should specify:

• Permitted use cases: e.g., summarising meeting notes, drafting marketing copy, generating code.
• Prohibited use cases: e.g., entering sensitive client data, making automated financial decisions, generating content that violates copyright.
• Fact-checking requirements: A clear statement that all outputs, especially those containing factual claims, must be verified by a human expert before being used in external-facing materials or critical decisions.

### Logging and Auditing

Every prompt sent to an LLM and every response received should be logged. This is non-negotiable for any serious enterprise application. These logs are essential for:

• Auditing and compliance: Demonstrating that data handling policies are being followed.
• Debugging: Understanding why the model produced a strange or incorrect output.
• Cost control: Monitoring token consumption to prevent budget overruns.
• Improving the system: Analysing user queries to identify common failure points or areas where the RAG knowledge base is deficient.

Without logging, you are flying blind.

Board-level questions before deployment

Before signing off on any significant LLM initiative, the board and executive team should have clear answers to these questions.

1. What specific, measurable business problem are we solving? The goal should not be "to use AI". It should be "to reduce average customer support handling time by 20%" or "to accelerate developer onboarding by providing instant answers to technical questions".
2. How will we measure Return on Investment (ROI)? What are the key performance indicators (KPIs)? Are we measuring cost savings, revenue generation, or improvements in employee productivity or customer satisfaction?
3. Is this a 'vitamin' or a 'painkiller'? Is this a nice-to-have tool that improves convenience, or does it solve a critical business pain point that gives us a competitive advantage? We often advise clients to start with a "painkiller" to demonstrate clear value. One such painkiller we frequently see is using LLMs to build internal AI assistants that people actually use to navigate complex internal knowledge.
4. What is our data strategy for this system? Where will the grounding data for our RAG system come from? Who is responsible for its quality, accuracy, and freshness? A great AI application built on top of a messy, outdated knowledge base will fail.
5. What are the data privacy and security implications? Have we chosen an architecture (API vs. open weights) that aligns with our data-sensitivity requirements and regulatory obligations (e.g., GDPR)?
6. Do we have the talent to build and maintain this? LLM application development requires skills that blend software engineering, data science, and infrastructure management. If you lack this talent in-house, you will need a partner with proven machine learning development expertise.
7. How does this create a sustainable competitive advantage? If we are just wrapping a standard API in a simple interface, our competitors can replicate it in weeks. Is our advantage in our proprietary data, our unique workflow integration, or the network effects of our user base?

Frequently asked questions

Are large language models safe to use with company data?

Yes, with the right architecture and contractual protections, they can be safe for most enterprise use cases. For API-based models, this requires an enterprise agreement with the provider that includes a zero-data-retention and no-training policy, ensuring your data is not stored or used to train their models. The primary risk, however, is rarely the model provider; it is in your own implementation. A secure system uses techniques like permission-aware retrieval in its RAG architecture, ensuring the AI only accesses and presents data that the specific user is already authorised to see.

Key takeaways

• LLMs are powerful pattern matchers, not thinkers. They predict the next most probable word. This explains both their remarkable capabilities and their tendency to invent facts.
• Grounding is non-negotiable for enterprise use. Unconstrained LLMs hallucinate. Use Retrieval-Augmented Generation (RAG) to force models to answer based on your company's verified data.
• The choice between open-weights and API models is a major strategic decision. It is a trade-off between control, cost, and performance. Most businesses should start with APIs.
• Governance is not optional. You need clear policies for data handling, acceptable use, and comprehensive logging to manage risk and cost.
• Start with a specific, high-value problem. Avoid "AI for AI's sake". Focus on a measurable "painkiller" project to build momentum and demonstrate ROI to the business.
• Success requires more than just technology. It requires a clear strategy, high-quality data, and the right engineering talent to execute.

Navigating the LLM landscape requires a clear understanding of both the technology's potential and its practical limitations. By focusing on specific business problems, prioritising data quality, and implementing robust governance, you can move beyond the hype and begin building real, sustainable value.

If you are ready to define a pragmatic AI strategy for your organisation, our senior teams are available for a detailed consultation.