The AI Scaffolding Layer Is Collapsing — Here's What Survives

The AI industry built an elaborate scaffolding of tools to make large language models useful. Retrieval-augmented generation pipelines, vector databases, orchestration frameworks, chunking libraries — an entire ecosystem of middleware emerged to bridge the gap between raw model capabilities and production applications.

That scaffolding is now collapsing.

LlamaIndex CEO Jerry Liu laid out the case bluntly this week: as foundation models get better at reasoning, tool use, and context handling, the layers of glue code between the model and the application are being absorbed. What took thousands of lines of custom pipeline code in 2024 can now be handled by a single model call with the right prompt and a few tools.

The Great Flattening

The pattern isn't new — it's what happened in every platform shift. When relational databases got SQL, the custom query parsers disappeared. When browsers got JavaScript engines, Flash died. When cloud providers added managed services, entire categories of infrastructure startups evaporated.

AI is hitting the same inflection point. Models like GPT-4.1, Claude 4, and Gemini 2.5 can now handle multi-step reasoning, long context windows, and structured tool use natively. The orchestration layer that was essential six months ago is becoming redundant.

"The models are eating the scaffolding," Liu explained. "If your product's main value proposition was making AI easier to use by wrapping it in abstractions, those abstractions are now table stakes inside the model itself."

What Dies

The casualties are already visible. RAG-specific startups that built their moats on custom retrieval pipelines are seeing their core functionality replicated by models with native context handling. Vector database companies are pivoting to broader data infrastructure as the need for specialized embedding storage diminishes. Orchestration frameworks that required extensive configuration are being replaced by agent architectures where the model decides its own workflow.

The specific categories under pressure:

• Custom RAG pipelines — Models with 1M+ token contexts and native retrieval can handle most use cases without hand-tuned chunking and embedding workflows
• Prompt engineering platforms — As models become more instruction-following, the art of prompt crafting is commoditizing
• Model routing services — When one strong model handles 95% of tasks well, the value of intelligent routing diminishes
• Observability wrappers — Model providers are building tracing and evaluation directly into their APIs

What Survives

Not everything in the scaffolding layer disappears. The companies and tools that survive share a common trait: they solve problems that models can't absorb because the value lives outside the model's context window.

Data connectivity remains essential. Models can reason about data they can access, but they still need secure, reliable pipelines to enterprise databases, document stores, and APIs. The connection layer — not the orchestration layer — has durable value.

Evaluation and testing infrastructure grows more important, not less. As models handle more complex tasks autonomously, the need to verify their outputs against business rules and safety constraints increases. Companies building robust evaluation frameworks are positioned to thrive.

Domain-specific knowledge graphs resist commoditization because they encode information that isn't in any training corpus. A medical knowledge graph, a legal precedent database, or a financial regulation ontology represents years of curated expertise that models can reference but can't replace.

Agent infrastructure — the tools for managing long-running autonomous processes, handling human-in-the-loop workflows, and maintaining state across complex operations — is actually growing, not shrinking. As AI moves from single-shot queries to multi-step agents, the infrastructure needs shift from orchestration to execution management.

The Enterprise Reality Check

For enterprise AI teams, the collapsing scaffolding layer creates both opportunity and risk. The opportunity: simpler architectures, faster time to production, lower maintenance burden. The risk: over-indexing on any single model provider whose capabilities could shift overnight.

"The smartest teams I talk to are building thin abstraction layers, not thick ones," Liu noted. "They want just enough insulation to swap models when the landscape shifts, without rebuilding their entire application. The days of six-month integration projects for a single AI feature are over."

The message for the industry is clear. The AI stack is flattening. The winners in the next phase won't be the companies that build the most sophisticated scaffolding — they'll be the ones that build the least, while solving real problems that exist outside the model's context window.