When an agent boundary stopped paying for itself

Summary

I worked on a multi-tenant AI platform that had grown from a retrieval system into a full agent runtime. Knowledge retrieval, SQL, external tools, delegated agents, tracing, long-running tasks, and user-facing streaming all ran through the same backend. One boundary remained from an earlier version: after the primary agent finished executing, a second LLM-based component generated the final answer for the user.

I wrote an RFC to remove the redundant synthesis boundary and run execution and synthesis in one loop. We kept specialist-agent delegation where it provided useful isolation or cheaper execution. Step-level tracing replaced the old "which agent failed?" debugging model. Removing the second mandatory LLM stage cut the total response time by more than half in some cases, from roughly 35 seconds to 13 seconds.

The architecture the team inherited

The original backend worked more like a sequential RAG pipeline. Each stage used a separate model call, and later stages waited for earlier ones to finish. A subsequent rewrite aimed to preserve the existing capabilities while moving toward agentic execution.

That architecture had two important components. A primary orchestration agent routed requests, called tools, ran retrieval and SQL, used connectors, and delegated work. A separate final-answer model received a reconstructed summary of the execution state and wrote the user-facing response.

The separation made sense at the time. It maintained compatibility with the previous backend and supported user-controlled response styles and personas. It also gave the system a clear presentation layer and a way to return raw execution artifacts without synthesis. Its assumptions simply stopped matching the system as the runtime evolved.

More capabilities moved into the primary runtime until it had everything required to produce the final answer itself. That included knowledge-base and SQL results, connector data, tool failures, retry state, delegated-agent results, sources, execution history, and conversation context. The final-answer component took no part in execution. The backend serialized state it already owned into another large prompt, then asked a second model to reconstruct enough context to answer the user.

The cost went beyond model latency because every execution capability needed two representations. When the system learned a new connector failure mode, the execution layer had to represent it correctly. The prompt-building layer then had to translate it for the second model. This duplication became more expensive and brittle as integrations and execution history grew.

What the second model was doing

I inspected the implementation instead of treating it as a generic "LLMs are slow" problem. The runtime already owned the chronological execution state, including connector routing, action results, failures, source payloads, retrieval results, SQL output, history, and execution summaries. The second model got a reconstructed version of that same state.

I wanted to know what new information or capability the model boundary introduced. Earlier in the product's lifecycle, it provided presentation separation and compatibility. By this stage, it contributed very little. The primary runtime already understood what had happened, while the second model mainly read that information again.

Failure attribution also got worse. The old distinction between an "execution-agent failure" and a "final-answer-agent failure" had stopped being useful once step-level traces existed. We needed to know whether planning, retrieval, a connector, a tool timeout, or synthesis had failed. Execution phase had become a more useful unit of observability than agent identity.

The RFC and what changed

I wrote an RFC proposing that the primary execution loop handle final synthesis. The primary runtime would route the request, execute tools directly, maintain live state, stream high-level progress, and write the final answer. It wouldn't need to reconstruct execution context for another mandatory LLM call.

The surrounding capabilities would stay. The proposal preserved user-facing progress events and final-response events, along with structured source metadata and execution tracing. It also kept custom presentation and persona rules, plus structured or raw output for downstream consumers that required it. The change was about where those capabilities lived.

I was concerned about mixing execution rules with presentation styles controlled by users. Sharing a model call shouldn't mean sharing a policy. The design kept execution policy conceptually separate from presentation policy. Execution policy covered available tools, execution flow, source constraints, retries, permissions, and completion criteria. Presentation policy covered response style, persona, formatting, and final-answer preferences. Both could operate in one execution flow while remaining separate concerns.

When a second agent earns its rent

We kept delegated specialist agents where they provided real isolation or cheaper execution. The RFC separated those useful boundaries from the final-answer model, which only reread state the primary runtime already owned.

A separate agent can justify its place through isolated context, specialized tools, cheaper models, permission boundaries, reusable capability, or parallel execution. For example, a specialist meeting-search agent could use a cheaper model to search a large external corpus, then return a compact result to a stronger model for final reasoning. The specialist owns information the caller doesn't have. It also uses different models and runs independently, so the boundary adds value.

The final-answer component lacked those properties. It consumed serialized state that the primary runtime already held, while adding another synchronous model call and another serialization layer.

Tracing steps instead of blaming agents

Removing the final-answer model took away a simple debugging statement: "the second agent failed." We already had better tools by then. The RFC proposed explicit tracing for planning, retrieval, execution, external integration, and synthesis. The system retained structured tool records and connector success and failure data. It also kept source metadata, status events, outcome markers, and end-to-end traces. Debugging could point to the step that failed, which matched the runtime's actual behavior more closely.

The raw-output question

The old system had a raw-query path that bypassed the final-answer model and returned execution data directly. That path was a natural fit for the two-model design because it exposed the execution artifacts. After final synthesis moved into the primary runtime, an entirely separate branch existed only because the old architecture had used two models.

Raw output is a product capability. It doesn't need to constrain the execution engine's design. I proposed keeping the capability but changing how it worked. The primary execution would produce either a final answer or structured artifacts. This separated the downstream contract from the internal architecture that had historically implemented it.

How we got there

I proposed an incremental migration rather than a rewrite. Phase one moved final response generation into the primary execution flow while preserving the existing streaming and result behavior. Phase two folded presentation rules into a final-synthesis layer but kept execution policy separate.

Phase three replaced the old raw-query branch with intentional structured output instead of keeping a branch shaped by the previous architecture. Phase four added a compact internal planning phase with the goal, sources to inspect, expected tool order, fallback branches, and completion criteria. It was an execution aid, not another agent that answered the user. Phase five deleted the obsolete answer-agent and prompt-building paths after the behavior had been consolidated.

The result

Removing the second model call cut the total response time by more than half in some cases, from roughly 35 seconds to 13 seconds. The runtime also had less context re-serialization and less duplicated prompt logic. New tools were easier to integrate, synthesis had direct access to live execution state, and step-level failure attribution was more useful.

Agent architectures collect model boundaries quickly because each one looks modular on its own. Before adding one, I check whether it owns new information, needs different permissions, enables cheaper or parallel execution, or exposes a reusable capability. If it does none of those things, I keep the work in the primary loop. This boundary survived two architectural rewrites because it had been around long enough to become part of the background.