The AI bill has arrived, and after two years of tool purchases and token burns, boards are asking a simple question: What did we actually get?
The answer is complex. PwC found 56% of CEOs report no significant financial benefit from AI. But WRITER* found that 92% of companies see individual productivity gains, while only 29% report significant organizational ROI. What does this mean for the enterprise? Individual contributors are faster, but the org hasn’t changed. Going a step further, the tools are working fine, but without any shared understanding of the business they’re working in. That’s not an AI problem. It’s a context problem.
This piece draws from Insight’s Onsite Hour context layer series, a weekly virtual event series for portfolio companies created by Insight’s 100+ in-house experts, including sessions led by Travis Kassay, Jack Rohrer, and Jared Brickman.
We’re in the third phase of the AI era
The first phase was the AI mandate. Boards said, “Do AI.” Everyone scrambled.
The second phase was tokenmaxxing. Every function needed an AI plan, so teams bought tools and burned tokens. Leaderboards appeared. Usage metrics climbed.
Now we’re entering phase three: Where’s the ROI? The bill has arrived. Impact is ad hoc, not enterprise-wide. Individual contributors are more productive. But the org as a whole hasn’t changed how it operates.
Most teams get personal productivity gains, but the context layer is what turns that into enterprise gains.
What the context layer actually is
The context layer sits in the middle of your AI stack. Below is your data: the CRM, data warehouse, and ERP. Above it are your AI tools: the assistants, agents, and automations that actually do the work. The context layer reads from your systems and writes back to them. Every tool draws from the same source of truth. “Think about the best new hire that you have on day one,” said Kassay. “Absolutely brilliant, but useless for about a month because they have no context. Every AI session today is that new hire, and most companies rebrief them from scratch every single session.”
“Every AI session today is that new hire, and most companies rebrief them from scratch every single session.”
Without it, the failure mode is predictable. Imagine a strategic account entering its 90-day renewal window. You send an AI agent to investigate and build a renewal plan. Because there’s no context layer, the agent goes directly to raw records — customer relationship management platform (CRM), support system, product usage data — and tries to assemble a picture. The data doesn’t agree. The agent “makes up the definition of ARR on the fly, accidentally including services revenue,” as Brickman described it. “It interprets test users as active users. It re-reasons a whole bunch of stuff, pulling call transcripts and rewriting a renewal playbook from scratch.” The output doesn’t match what the finance agent or account management would say. It’s living in its own bubble.
“The failure was not that the agent lacked access to data,” Brickman said. “It’s that it lacked a prepared understanding of the business.”
How the context layer compounds
Insight Partners has observed a pattern across the portfolio: two companies with the same use case — deal coaching and analysis — making very different bets.
The first company bought point solutions. Disconnected tools that often disagreed with each other. Per-seat pricing that grew expensive fast, millions on a single vendor alone. Fast to deploy, but blind to cross-functional tasks, and capped by vendors’ product roadmaps. New use cases required new contracts.
The second company invested in its context layer. One unified layer feeds every tool. Agents and skills execute the use cases. Built in-house, using existing engineering capacity. Each new use case builds on the last.
Atlan Cofounder and Co-CEO Prukalpa Sankar, who has spent 18 months building context layers with customers, including large enterprises, put the dynamic well: “We were building agents that were single-star players. But dream teams are never built on single stars. Dream teams are actually built on shared context — shared knowledge, shared expertise, shared learning loops.”
Point solutions rent capability on a seat-by-seat basis. The context layer compounds. The advantage grows with each use case added. And as Tricentis’ Principal Architect Jakob Marsala, noted, the financial case isn’t just about use cases: “Every time that reasoning loop runs, that’s burning tokens, and that is runaway AI spend. The more you can shift left on that stuff, it is a very quick financial savings as well.”
What goes inside the context layer
The context layer has four components: a knowledge base, a knowledge graph, a glossary, and memory. Governance holds all of them together. “The trap here is to avoid marketing building its own knowledge base, sales building their own, and product building their own,” explained Rohrer. “That just recreates the same problems and silos that we’ve historically had within companies, and so you want to build toward one shared place from day one.”
Knowledge base. Your company’s knowledge, written down and organized, so AI can use it. Think of everything you’d hand a new hire on day one — company identity, ideal customer profile (ICP) definition, product overview, brand voice, competitive cards. The format matters: not PDFs and spreadsheets, but structured markdown files updated routinely. Best practice is to organize by scope: organization level, function level, and user level. Early on, the knowledge base may be the entire context layer, and that’s a fine place to start.
Knowledge graph. The map of how things in your business connect. Not the data itself, but the relationships: customer → contract → owner → renewal date. With the graph, AI can reason across the business rather than just return summarized answers from a single system. Your CRM and data catalog are already rich starting points.
Glossary. Your company’s dictionary. What counts as annual recurring revenue (ARR), a lead, a churn, a win-back? Which system wins when the numbers disagree? Marsala explained: “There’s a way that everybody does ARR, and then there’s a way your business does it. Claude, OpenAI — probably not going to know what they mean by that.”
The glossary prevents AI from confidently running analysis on the wrong definition and acts as a semantic layer.
Memory. What lets AI keep what it learns instead of starting over every session? Decisions, corrections, and outcomes get written back. Every interaction makes the next one smarter. It starts simply; for example, a well-kept decision log works. Eventually, AI writes back what it learns on its own.
Governance. The ownership and upkeep that keep everything else true. Context without an owner goes stale by default. Every file needs a named owner and an update cadence, with clear triggers: a pricing change, a major account lost, a product launch.
What context layers look like end-to-end
Before the trigger even hits, the CRM has been mined and pulled through a unified ontology. A renewal risk playbook has been assembled. An account-brief skill defines exactly what the agent should pull from the knowledge store — customer segmentation, product definitions, business rules — and what to join with live data: recent support issues, key players from recent calls, current usage trends.
The agent reads the knowledge base, traverses the knowledge graph, checks the glossary, and drafts a brief where every claim is traceable to its source. The rep reviews it. “Too long — give me one page of bullets.” That preference gets written to memory. Next time, she gets one page of bullets without asking.
“Don’t make agents rediscover the business from raw records on every run,” Brickman summarized. “Let agents spend the runtime reasoning on what changed and what to do.”
“Don’t make agents rediscover the business from raw records on every run.”
Each interaction makes the next one better. At Atlan, Sankar reported seeing up to 75% improvement in token costs and “a dramatic impact on accuracy” after implementing a context layer across customers. At a large energy company working with apiphani’s* Principal Director James Kendrick described the compounding effect of building context infrastructure up front: products launched 90% faster at 30% of the cost, and the AI program advanced at least three months by enriching their data catalog with structured context. “It clearly advanced our program at least 3 months in terms of moving us into the AI world,” he explained.
Where to start with building a context layer
The experts are aligned on sequencing: don’t wait until the architecture is perfect. The context layer is a progression, not a prerequisite.
Insight’s team frames it as a three-stage maturity journey, each with a realistic time to value:
Stage 1: Knowledge base (days to value). Start with what you’d hand a new hire on day one: company identity, product overview, ICP definition, competitive cards. Write it in structured markdown. Organize by scope: org-level, function-level, user-level. Give every file a named owner and a review cadence with clear triggers. That single artifact, loaded into any AI tool you use, will improve output relevance and accuracy. Early on, this may be the entire context layer, and that’s a fine place to start.
Stage 2: Database-backed retrieval (weeks to months). Add a real retrieval layer: a vector database that lets your AI search across your data estate rather than rely on static files. A product docs and codebase index is a strong starting point: accurate product knowledge becomes queryable across the org and stays up to date as you ship. This is the step that makes the knowledge base dynamic.
Stage 3: Context operating system (months). The fully mature layer, with a versioned knowledge graph, governed memory, real-time data feeds, and autonomous write-back. Every interaction makes the system smarter. Every new use case builds on the last.
The risk, as Marsala learned, was starting with the wrong things. “Data isn’t empowering; data is entropy,” he said. “AI makes everyone an enthusiastic developer engaging in shadow IT.” His hard-won lessons from rolling out Tricentis’s AI Business Studio: get your discovery right before you start building, and treat fine-grained customizations as a stage-three problem, not a stage-one ambition.
*Editor’s note: Insight Partners has invested in apiphani, Tricentis, WRITER, and Atlan.
