The rise of the company LLM: Why enterprise AI must be built in context
Employees now spend an average of 4.3 hours per week checking the work their AI did for them. That is roughly £14,200 per person, per year, burned on babysitting a system that was supposed to make them more productive. Meanwhile, MIT and RAND Corporation research shows that 85 to 95% of enterprise AI pilots fail to deliver expected outcomes. The technology works, but it severely lacks organisational context.
For the past few years, enterprise AI has been shaped by general-purpose models trained at internet scale. These systems are powerful, flexible, and increasingly easy to access. But inside real organisations, where accuracy, governance, and trust matter, generic AI is already hitting its limits. We expect the next phase to transition from scale to specificity.
Gartner agrees, predicting that by 2027, organisations will deploy small, task-specific AI models at three times the volume of general-purpose large language models. By 2028, more than half of all enterprise GenAI models will be domain-specific, up from roughly one percent in 2023. Gartner named domain-specific language models as the number one rising technology trend for 2026, noting that some deliver four times the efficiency of general-purpose LLMs in cost and latency.
The supervision tax
Generic AI is optimised for breadth. It is built to answer almost any question reasonably well. Enterprises need something different. They need a system that knows the difference between a draft policy and an approved one, that understands which contract terms trigger escalation, and that recognises when a procurement request sits outside delegated authority.
Without that context, every output requires human verification. In low-risk tasks, that is manageable. In document-heavy, regulated environments – finance, legal, procurement, healthcare – it becomes expensive and slow. Forrester found that fewer than one in three AI decision-makers can tie AI value to their organisation’s financial growth. As a result, enterprises are expected to defer a quarter of planned AI spend into 2027 while they reassess where generic approaches fall short. Forrester’s verdict for 2026: AI will trade its tiara for a hard hat.
From sovereign AI to company intelligence
There is growing interest in sovereign AI, models trained at a national or regional level. There may be a future in which these models are specialised by national or industrial capabilities, but the real opportunity lies in organisational SMLs: small language models trained on vertical datasets and grounded in the specific knowledge of a single organisation.
In a recent example, IBM deployed a domain-specific AI system called OLGA at the Stuttgart Higher Regional Court in Germany, where judges faced a backlog of more than 10,000 cases. OLGA categorises cases, extracts metadata from pleading files running to hundreds of pages and allows judges to concentrate on complex legal reasoning rather than repetitive document triage. The court estimates a processing time reduction of over 50%. When institutional knowledge is encoded and deployed with intelligence document processes, the potential for processing reduction and insight extraction is incredible.
Gartner projects the domain-specific AI market will reach $11.3 billion by 2028. The market direction is clear: the winning models are not going to be determined by size of generality of intelligence. It is specificity and organisational context that will best determine output.
Context over prompts
Much of last year’s conversation revolved around prompt engineering, the art of asking the right question. Organisations are now discovering that the more effective approach is to improve what AI knows, not how you ask it.
Gartner formalised this shift in mid-2025, declaring that context engineering has replaced prompt engineering as the priority discipline. The difference is practical. Instead of spending three iterations refining a prompt to get a useful procurement summary, organisations structure their contract libraries, so the AI already understands vendor terms, approval thresholds, and compliance requirements before anyone types a word. The focus moves from crafting clever inputs to curating trustworthy knowledge.
Documents are the missing layer
90% of enterprise data is unstructured. It lives in PDFs, contracts, policies, technical manuals, and project documentation. Yet Deloitte found that only 18% of organisations can leverage this data. The gap between what companies know and what their AI systems can access is enormous.
The PDF Association has highlighted that even PDFs – the dominant enterprise document format – were not designed with AI in mind, with critical information hidden behind compression, visual formatting, and intricate layouts. Converting structured documents into machine-readable formats dramatically improves AI accuracy. This is where the company LLM is born: not in a research lab, but in the intelligence already embedded in an organisation’s document estate, extracted, structured, and made available as a single source of truth.
Quiet, embedded, everywhere
Company-specific models do not replace people. They work in the background: validating documents, enforcing policy, reducing errors, accelerating decisions within defined boundaries. Unlike a generic assistant, they understand who the user is, what role they play, which data they should access, and which rules apply. They improve continuously as new documents are created, reviewed, and approved.
Gartner predicts 40% of enterprise applications will integrate task-specific AI agents by the end of 2026, up from less than five percent today. But they also warn that over 40% of agentic AI projects will be cancelled by 2027 due to unclear value. The pattern is consistent: AI that is not grounded in organisational context does not survive contact with organisational reality.
Generic AI will continue to play an important role in creativity and exploration. But inside the enterprise, where trust and precision determine outcomes, the winners will be organisations that build AI in context – small, governed, and trained on what the business knows.
The rise of the company LLM is not about replacing people with machines. It is about closing the gap between what AI can do and what it understands about the organisation it serves. That gap has always been the real bottleneck. Now, finally, it is closing.
