Agent infrastructure
From a sequential RAG API to a production agent harness
I rebuilt a fixed LLM backend as an agent harness in two phases while keeping the API stable for the products already using it.
- Outcome
- RAG → Agent harnessa complete runtime rebuild behind the same product API
- Read time
- 7 min
- Agent runtimes
- RAG migration
- LLM orchestration
- Tool execution
- Observability
- Architecture evolution
Summary
I worked on an AI backend that started as a sequential RAG API. Retrieval, SQL, chart generation, answer generation, and the other capabilities ran as separate LLM stages in a fixed flow.
The product needed to become a complete agent harness. Models were getting much smarter and able to work long-horizon tasks. They were also getting cheaper and easier to use, while open-source models literally changed the world. The direction was clear, but getting there meant rebuilding most of the product underneath an API that other systems already depended on (yikes!).
I led that move in two phases. In phase one, we replaced the handwritten pipelines with an agent runtime while keeping the existing API and final-answer behavior intact. Once the runtime had live execution state, tracing, tool use, and specialist delegation, I initiated phase two. We removed the compatibility layer and let the primary runtime produce the final response itself. As expected, it really improved the product.
The result was a new product underneath the same interface. We went from a fixed RAG pipeline to a harness that could decide how to execute a request, call tools, delegate work, and keep track of the whole run. The phase two cleanup also reduced time and model cost. Some knowledge-query paths went from roughly 35 seconds to 13 seconds.
The system before agents
The original backend was a sequence of LLM-powered flows. The application decided which stage ran next, passed the output forward, and waited for the next model call.
user query
|
retrieval stage
|
LLM
|
SQL or another stage
|
LLM
|
answer generation
|
response
This was a reasonable fit for the earlier product. Models were less capable, more expensive, and harder to trust with a long chain of work, so keeping the flow explicit made sense.
That changed quickly. We could see models handling longer tasks reliably, prices were falling, and open-source models gave us more options. The old pipeline was becoming the thing that limited the product.
Why the RAG pipeline stopped scaling
The requests were never completely predictable. We knew the fixed pipeline would eventually run out of room, and for a while we handled that with workarounds.
Every new capability created another branch. Every branch needed a router to decide whether it should run, and that router often meant another synchronous model call. After two or three routers, latency became the limit. Adding one more capability made the whole request slower, even when most of the pipeline was irrelevant to what the user had asked.
We reached a point where we could not keep adding capabilities to the existing design. We had to rebuild the execution layer.
Before: the application decided the execution graph
After: the runtime exposed capabilities and the agent decided the execution graph
Phase 1: rebuild the internals and keep the API stable
We kept the external API the same and replaced the handwritten pipelines underneath it with a proper agent harness. SQL, chart generation, retrieval, connectors, and delegated work became tools the primary agent could choose when it needed them.
+-- knowledge-base search
+-- SQL
user -> Main Agent -+-- charts
+-- connectors
+-- specialist agents
+-- other tools
|
execution state
|
final response
Keeping the API stable mattered because several downstream products already depended on it. Rebuilding the runtime and changing every integration at the same time would have made the migration much harder to ship.
Keeping the Answer Agent during the rebuild
The first version of the harness still passed its execution state to the existing Answer Agent.
+-- knowledge base
+-- SQL
user -> Main Agent -+-- connectors
+-- tools
+-- delegated work
|
execution state
|
Answer Agent
|
user
A lot of downstream behavior depended on that final stage. It handled the response contract, presentation rules, personas, and raw-output behavior. Keeping it gave us room to replace the execution engine without forcing every dependent product to migrate alongside it.
The Answer Agent was a compatibility layer. We expected to remove it later, but it made phase one possible.
What the new runtime could do
The runtime could now decide where a request should go instead of following a route written in advance. A single request could involve several capabilities at once. RAG was only one of them. We also had SQL generation, chart generation, external tools, actions, etc. The agent could choose the relevant ones, decide the order, and skip everything it did not need.
The harness supported:
- tool execution
- external connectors
- streaming status updates
- specialist-agent delegation
- work across knowledge domains with access controls
- memory and long conversations with compaction
- scheduled and long-running agents
- execution logs, traces, and artifacts
The primary runtime also held the full state of a request: tool calls, retrieval results, SQL output, connector state, delegated-agent results, failures, retries, sources, and conversation history.
Phase 2: remove the scaffolding
Once the harness had matured, the Answer Agent had to go. The primary runtime already knew the path it had taken and held every result needed to answer the user. Passing all of that state into another model meant paying for a second model to reconstruct work that had already happened.
Main Agent
|
live execution state
|
serialize state
|
Answer Agent reconstructs it
|
final response
I initiated phase two and led the execution. It was easier than I had expected. The hard work had already happened in phase one: the main runtime owned the execution path, and models were now capable enough to synthesize the answer directly from that state.
+-- retrieval
+-- SQL
+-- connectors
user -> Main Agent -+-- tools
+-- specialists
+-- execution state
|
final synthesis
We removed the mandatory handoff but kept the product behavior around it. Streaming events, source metadata, presentation rules, structured output, and raw output still worked through the same API.
Why the specialist agents stayed
I did not collapse the specialist agents into the primary runtime. They maintained their own context and tools, which made them reusable and kept their work out of the main context window.
That separation was still useful. A specialist could search a large external corpus or handle one domain, then return a smaller result to the main agent. The old Answer Agent did not own a capability like that. It only reread state the main runtime already had.
Observability was ready by phase two
By phase two, observability was no longer a reason to keep the Answer Agent as a debugging boundary. We looked at Langfuse and PostHog for monitoring. Both were useful, and we chose Langfuse because it gave us the logs and traces we needed for model calls, tool execution, and the full request path.
We no longer had to ask which agent failed. The trace showed the exact step: planning, retrieval, a tool call, an external integration, or final synthesis.
That made removing the Answer Agent much safer. We lost a coarse boundary and gained a much more useful view of the run.
Result
The backend became an agent harness that could plan and execute different combinations of tools across the products connected to it. We kept the existing API working while the execution model underneath it changed almost completely.
Removing the Answer Agent cut one mandatory model call from the response path. That reduced model cost and helped bring some knowledge-query paths from roughly 35 seconds to 13 seconds total response time.
Before
fixed RAG and LLM pipeline
-> routers and branches
-> compatibility synthesis
After
primary agent harness
-> tools, specialists, and connectors
-> live execution state
-> direct synthesis
What I learned
This was the first time I helped rewrite so much of a live product while keeping it available to the systems already using it. I learned how much a stable API matters during a migration. The internals could change completely as long as the products around them did not have to change at the same time.
I also learned when a compatibility layer is worth keeping and when it starts getting in the way. The Answer Agent gave us a safe path through phase one. Removing it in phase two gave the new runtime room to work the way it was designed to.