Be ambitious: give it a complete goal

Think back to a few recent requests. Could the agent have handled the work between them too?

We increasingly delegate a complete outcome: where to investigate, what to deliver, how to check it, and when to involve us. That gives the agent room to explore, plan, implement, and verify.

For a feature, an illustrative brief might be:

Make notification preferences work end to end. Inspect the code and propose a plan. After our approval, implement it, run the relevant checks, and test the flow in the development environment using the browser. Bring us a reviewable PR with the results. Ask us about missing product decisions; leave merging and production deployment for review.

We have used this approach for changes spanning several services. Once we approve the plan, the agent can keep going through implementation and testing. We return when there is something worth reviewing or a decision it needs us to make.

Even substantial requests can go further: include the initial investigation, or the verification that usually happens afterward.

Give it the outcome, the definition of done, and clear review points.

Be a whisperer: think out loud to the agent

Askomity push.

That is what the transcription produced when one of us said, in Spanish, haz commit y push: roughly, “do a commit and push.” The agent understood the intention and responded about committing and pushing the changes.

When preparing a presentation, we used voice to work through the structure, reshape slides, and rehearse the delivery. The agent turned that rambling into presenter notes and reminders of what to demonstrate.

We still write or edit exact wording when needed. For exploration, the conversation can start before the idea is fully organized.

Try a minute of thinking out loud, then ask the agent to clarify the goal and any open questions.

Start talking while thinking. Let the agent help find the structure.

Be lazy: let it find the context

A request arrives in Slack. Before work starts, there is a related issue to find, a document to read, and last week’s discussion to recover.

That preparation costs attention. Copying and pasting is work.

Connect the sources the work needs, with access appropriate to the task. We can then point to a conversation and let the agent investigate related information, including what we might have forgotten.

Distinguish what the agent can access from what it needs for this task. Keep the working context focused with concise tool results, especially under tighter model or token budgets.

Point to the sources. Let the agent do the gathering and connecting.

Turn those habits into a daily loop

Here is one way we put them together:

Email review. A scheduled task brings us the important messages at the start of the day. Initially, we checked the inbox too and corrected what it missed or included unnecessarily. Those corrections refined the criteria; we now rely much less on manual scanning.

These reviews now do more than filter relevant mail: they sometimes catch important messages we would have missed. One example was a certificate-expiry notice that could have left several of our internal services unavailable.

Priorities. A separate ongoing workflow reviews the connected sources and updates our priorities document in Notion. It can connect a chat request with a meeting tomorrow and surface what needs preparation today.

Parallel work. From those priorities, we launch independent goals: investigate this, prepare that proposal, start this implementation. We review the results, and completed work feeds back into the priorities.

Parallel work also means more conversations in an agent workspace (Codex, Claude Code, or another tool). Keeping active work visible takes some organization. One arrangement we use is:

• Permanent: recurring routines and scheduled tasks, such as the daily email review.
• Pinned: the work we are actively following.
• Key projects: projects we return to frequently.

Organization is personal; project folders may be enough. The useful test is whether work needing attention is easy to see and find.

Keep the priorities, delegated work, and completed results connected.

Make each correction useful next time

Useful knowledge can disappear into old chats. A knowledge workspace with an index, current work, project folders, and reusable instructions gives later sessions something to pick up.

Make upkeep part of the workflow, rather than another thing to remember. A skill or instructions in an agent’s file, such as AGENTS.md or CLAUDE.md, can make recording decisions, corrections, and remaining work part of the task.

Some of us use a scheduled task to review sessions that are largely finished, extract the useful lessons, and update our personal or project knowledge base. PR review feedback can feed the same workspace.

A manual session review is still useful at the end of a task:

What did we have to explain repeatedly? Propose updates to the reusable instructions, and save the decisions and remaining work so we can resume later.

Keep task-specific details in the relevant project notes, and review changes to reusable rules so a one-off correction does not become a blanket instruction. The lesson lives in files and examples that later runs actually read.

Save the lesson where future runs can read it—and make saving it routine.

Make the result easier to understand

Getting an agent to finish is only part of the job. A finished result is only useful if it is easy to understand and review.

Ask for a format that makes the next decision easier. A diagram can clarify service relationships. An HTML report can make a proposal easier to navigate. An interactive artifact can turn an explanation into something to explore.

We can save the explanation styles that work well as reusable instructions, including the diagrams and examples that make a result easier to follow.

Then question the result:

Why this approach? Which assumption does it depend on? Show us how you checked that it works.

For the billing example, an illustrative answer could look like this:

Now there is something specific to challenge: does that assumption hold, and do the checks support the decision?

Delegate some of the work of making the result reviewable.

Protect attention as parallel work grows

More agents can advance independently. Our capacity to switch between them still has a limit.

Prepare each goal so it can advance until there is a reviewable result, a blocker, or a decision that needs human input. Workstreams touching the same services may need coordination rather than simultaneous execution.

As coordination grows, a director agent with clear roles, session names, and documented state is one pattern to explore.

The environment matters too. Scripts that return check results and relevant errors can keep huge build logs out of the conversation. Different models may suit planning and execution, depending on the budget.

Keep asking: is this reducing the attention each task needs, or giving us more things to monitor?

Pick one practice and try it this week

Pick one new practice: think out loud, delegate a complete goal, connect a source, automate a daily review, or ask for a visual explanation.

Use it on a real task. Notice how often it needs intervention, what has to be repeated, and how much effort review takes. Adjust and try again; if it does not fit, choose another practice.

These habits reinforce one another. Connected context makes a spoken brief easier to act on. Clear goals let work advance further. Saved feedback reduces repeated corrections. Readable results make it easier to pick work up again.

One practice. One real task. Give it a proper try.

Keep building on what works. There is a title to work toward:

SWAT: Supreme Whisperer of All Time.

Topic on tamHub: Agentic engineering →

An agent sets the day’s priorities

One of us works with Codex and Claude Code side by side, with a lot of things running in parallel, and starts every day with two agent sessions. One reviews all the email. The other is connected to Slack, email, Notion, practically everything, and its job is to say what the priorities are.

There is a Notion document with all of them. The agent goes through it: this one was due today, this one has already moved forward. An urgent Slack message about something that needs doing becomes a new priority and a new task.

The agent manages those priorities now.

Our attention caps how many tasks run in parallel

With the list ready, everything that can run in parallel gets launched, which is usually most of it. The first wave is easy: go and research this, look at what is happening here, find out why this is not working. That is four or five tasks, more or less.

Then come the implementations that have already been running for several days. Review what was done the day before, decide where to go next, read what results came back.

Altogether, five, six, seven tasks is what one person can sustain in this setup. More could be launched. But as they finish, each one needs attention, and someone has to get involved and see where it went. How many agents you can run depends on how much your own mind can process.

An example: the agents built the filter, we supplied the constraint

On AIFindr, people were typing their personal data into the chat, and we needed to clean the PII out of each message before it reached the main model. Our first attempt put a third-party model in front of the main one. It worked, but it was very slow: the user waited too long before seeing an answer. And the raw PII still had to leave our infrastructure, something a major Latin American bank we work with would not accept.

Optimization is always a rabbit hole, one you can go into and not come out of for a month. So we sent coding agents down it instead. They researched open-weight models that could run inside our own infrastructure, a setup complicated enough that the engineer who led this would have spent two months learning it. The agents did it in an hour or less.

They tried several models and found one that filtered the personal data out of users’ queries in 100 to 200 milliseconds. It matched the remote model on the evaluation set we had built first. From idea to a working service took practically no time.

The first version ran on a GPU. It was fast and private, and it came to about EUR 1,340 a month for that one feature.

That is where the engineer has to step in: you have this constraint, that is too expensive, look for alternatives. We gave an agent a stricter goal: keep the privacy, the quality and the speed, and make it sensible to run. It looped through hypotheses, testing each one against our evaluations and benchmarks, until it found an answer we had barely considered: no GPU at all. Before deploying, we used the previous months’ usage data, requests per second included, to load-test exactly the setup we would run. It now runs on small ARM CPU tasks for about EUR 120 a month across all our environments, 91% less. The full story is in our post on AIFindr Privacy.

The agents did the research, the setup, the load tests and even the search for a cheaper architecture. Their first working answer was too expensive to run, and the constraint that ruled it out was ours. So was the evaluation set that defined what good enough meant. The AI can do practically anything, but not everything is actually right for your problem. That judgment, and knowing how to guide the AI, is our job now.

The bottleneck moved to the ticket

When a ticket does not land in one shot, we see a bit of both: sometimes the miss is in the code, sometimes in the ticket. But getting the code right on the first try keeps getting easier. Getting all the context into the ticket does not, especially when part of it has not been figured out yet. That is why the engineer still reviews the result and decides whether it actually does what we need.

With Facility we automated the path from ticket to production. Now we are working on the step before it: taking what is in people’s heads, meetings and chats, and turning it into tickets with enough context for more PRs to land in one shot.

The problems got bigger

So what is AI? Right now it writes very good code. We would call it a tool, except it is obviously much more than that. It is one more thing we have available, and as engineers what we can do now is understand it, learn to use it well and know it well enough to solve problems with it.

We are not going to spend our time painting a button blue anymore. We are going to be working on problems closer to curing a disease. The problems have grown in complexity, and what we have to do is take a tool that powerful and, with an engineer’s mind, apply it correctly to the problem and find the solution.

The next time an optimization looks like a month-long rabbit hole, send the agents down it first. Keep the constraint for yourself.

This piece is one of a pair. The other, Engineers finally have time for the problems nobody could prioritize, is about what to invest in: skills, our own tooling and how we learn. This one is about how the work gets done.

Topic on tamHub: Agentic engineering →

Writing and reviewing code are losing value

The clearest skill to lose importance is writing code, in every sense, including the ability to review it. Once a system can hand you the output, the environment, the artifact, whatever it is, and you can check it directly from a functional point of view, code review stops being where the value is.

The same goes for defining an architecture graphically. Claude is going to do that for you too.

New skills appear, starting with the obvious one: these AI systems have to be learned, and learning them matters. But the bigger shift is in skills that were always important and rarely got the time.

The time code used to take now goes to engineering problems

Before, some things could not be prioritized. Latency problems, or PII problems, could take a long time, and prioritizing them was a serious problem. So they waited.

Now they get prioritized. They ask for time, attention and a share of problem solving as an engineer that they rarely got before. One of those PII problems, and what solving it took, is in Our job was never translating requests into code.

That has a consequence for engineers who were never very fluent at that kind of problem solving: now they have to be. If you want to stand out, or just be useful, those are the situations where an engineer can be far more useful than before.

The internal tooling that used to be a time sink now pays off

Our team has a lot of people who treat development almost as a hobby. For years the same thing kept happening: everyone wanted the perfect development environment for their own team. People would spend time on “I’ll build this library, this tool”, and it was always a time sink. It never really sped anyone up, and the day-to-day work ate it.

AI gives us a huge capacity to build, and we have not managed to exploit it fully yet. Seeing both at once changes the calculation. What used to be a time sink, building a tool like that, now makes a lot of sense for a company like ours. It is how we ended up building Facility, large and complex tooling for working with AI that works very well, and reached effectiveness in a short time.

We think any company should start considering what used to be the time sink nobody touched: spending time on its own software development lifecycle. That is what will actually speed it up, and quickly.

There is one condition. You can only do it if you have the tool and you understand the rules of the game.

The rules are learned from people who have hit the problems

There is too much hype. You try to stay optimistic, but a lot of it is easy to question unless it comes from very reliable sources, and if you let everything you see online carry you, you end up overwhelmed. It happens to all of us.

Consuming content is an important part of learning. But the people we end up learning from are the ones who have actually faced the problems. What they keep running into is that things are not that easy. That is good news. It means there is a lot to learn, and it means you do not have to try everything, because not everything is useful.

That is why being in the same room with people who are really doing the work matters so much to us. It is what separates being overwhelmed from being grounded, focused on what actually works for you.

Bigger systems carry more responsibility

We are going to build more, and bigger. Complexity will grow, and so will the responsibility and the impact software has on people.

That is where engineers stay necessary: engineers who can keep control of the AI, make it evolve within the domains they work in, and deliver value with it. Someone has to maintain and improve those systems so they keep serving the people who depend on them.

The job will be more complex and will need different skills. There is an opportunity now to adapt and do bigger, better things with what we have.

Take the list of problems your team knows about and never schedules. Pick one, and give it the time the code used to take.

Topic on tamHub: Agentic engineering →

The problem: Forty Banks, Zero Patience

Here’s the catch. Every bank writes its statement differently and calls the same number three different things. On top of that, most people don’t hand over a clean PDF. They upload a photo taken on the kitchen table, or a screenshot from their banking app, and both come out messier than any clean document would.

When the reading failed, the app fell back to asking people to type the numbers in by hand, right in the middle of signing up for something they hadn’t even tried yet. A lot of them just stopped there.

There’s no soft failure mode here, either. Get a number wrong, and you’re not looking at a typo. You’re handing someone bad financial advice.

Our solution: Split the Job, Trust the Answer

We redesigned nau!’s reading pipeline on Claude and rebuilt it around two simple steps.

The first step answers a single question: which bank issued the statement. The second step reads the actual numbers, using a template built for that specific bank. Each template spells out what shows up on that bank’s statement, what each field is called, and roughly where it sits on the page. There are about forty of these, one per lender, plus a generic fallback for anything that doesn’t match.

We split the work this way because asking one pass to do both jobs is asking too much of it. It would have to figure out which bank’s statement it’s looking at while also hunting for numbers whose names depend on that answer. Once the bank is known, the second step already knows exactly what it’s looking for.

The reason there are forty separate templates, instead of one that covers every bank, comes down to precision. A template built for one bank can be exact: this field is called this, and it sits here. A template meant to cover every bank can only say “look for something that means outstanding balance,” and that vagueness is exactly where most misreadings came from. Before the redesign, 29% of statements were falling through to that generic template, and those were the readings the system trusted the least.

Both steps also require the model to answer in a fixed format: a form with named fields it has to fill in, nothing else. An empty field is something the code can handle. A paragraph of explanation sitting where a number should be is what used to break things downstream.

Three checks stand between a reading and the database. Is this even a mortgage statement? (Over sixty days, the system turned away twenty-one files that had nothing to do with a mortgage.) Does the bank the model guessed match what the IBAN itself says? (If not, the reading gets thrown out and redone.) And then there’s the user: every reading is saved next to whatever they later correct by hand, so the team always knows exactly what to fix next. Nothing irreversible happens without the user confirming it first.

The real impact: Reading With a Little Bit of Judgment

Here’s the part that actually makes this different, and it’s not really about reading at all. It’s parsing. Parsing itself isn’t new, it’s been around for years. What’s new is how fast and how easy it’s gotten.

The old way meant a couple of developers and a few weeks to hardcode every field for every bank format. Once that code existed, it was rigid. A bank tweaks its layout, moves a field, changes a label, and the whole thing breaks until someone goes back in and rewrites it.

This version was built in under a week by someone with no technical background, and it kept getting better with input directly from nau!’s own team, no dev cycle required.

The real difference is that the model behind it can make small decisions on its own. It doesn’t need every single field hardcoded to work. If a bank makes a small change, a slightly different layout, a relabeled field, the agent can still adapt and read it correctly. Traditional code doesn’t have that option. If the statement didn’t match exactly, that was it: a developer had to go back in and rewrite it. Here, a small variation just doesn’t break anything.

There’s also a second agent running quietly every night, going through every statement the system has seen and getting a little sharper each time. Fifteen nights in, it had read 210 mortgage statements on its own and built a fingerprint catalog covering 21 different banks, with no one checking its work and not a single mistake.

We designed three levels of autonomy for this agent and deliberately started it at the most cautious one: observe and accumulate. It doesn’t write anything a client will ever see. Since these statements carry someone’s name, ID number, and IBAN, the catalog it builds is limited to a fixed set of fields, verified against the database, not just whatever it decided to ask for in a prompt. We’d rather an error cost us an empty table than have someone find it in their own mortgage.

That’s the real shift here. We used to save the documents themselves, because that was the only way for the system to learn. Now we save what it learned, and we can delete the documents.

The Numbers Don’t Lie

The biggest gain showed up on the hardest input: a photo taken on a phone. Before the redesign, the system picked the wrong bank template for roughly half of those. Right after we shipped the fix, that number dropped to under a third, and it keeps shrinking as the system teaches itself a little more with every statement it reads. Outright failures, where the reading breaks and nothing comes back at all, now sit at 1.2%, twelve out of every thousand statements. Before we rebuilt it, that number couldn’t even be measured, because those failures weren’t being logged in the first place. Today, the system has read 1,873 statements for 1,008 users.

In our client’s own words

Automating this one step didn’t just cut down on manual data entry, it took away the single biggest point where people were dropping out of signup, and gave the nau! team the confidence to keep building faster on top of it.

“This agent has been working alone for fifteen nights now: it’s read 210 mortgage statements and built a fingerprint catalog of 21 banks, with no one checking its work and not a single mistake. (…) The hardest part wasn’t deciding what it could do. It was deciding what we wouldn’t let it do.(…) The real shift is we used to save documents because that was the only way to learn. Now we save what we learned, and we can delete the documents, increasing our customer’s data safety”

Sergio Carbajal, CEO of nau!

There’s more to build here, and we’re glad to be building it with the team at nau!.

A privacy layer that still crossed the boundary

Our initial PII provider used GPT-5.4 Nano with low reasoning. Functionally, it worked well. The workflow and our downstream analytics received redacted text instead of the original message.

That was useful containment, but the original PII still had to leave our infrastructure: AIFindr (The Agile Monkeys’ platform for building reliable enterprise AI agents) had to send it to a third-party model for redaction.

A major Latin American bank made the requirement explicit: original PII should not leave our infrastructure. At the same time, the remote redaction call was adding roughly 0.8-1.2 seconds before the agent workflow could start. The client felt that delay directly: it degraded the experience before the first useful word could even appear.

Why not just use regular expressions? They work well for predictable formats, but PII is often ambiguous and context-dependent. Formats also vary across countries and user input, so adding more rules quickly becomes brittle and risks creating false positives.

So we decided to investigate open-weight models that could run fully inside our infrastructure. To compare them responsibly, we first needed a strong evaluation dataset: standard PII examples plus difficult cases with ambiguous values, malformed formats, and false-positive traps. It had to tell us whether sensitive data remained visible, whether harmless content disappeared, and how long redaction took. Quality came before speed or cost.

Breakthrough one: an open-weight model on our GPU

We created AIFindr Privacy as a separate internal service that the backend could call for redaction. The service ran the open-weight model on a GPU inside our infrastructure.

The first engine used OpenAI’s open-weight openai/privacy-filter. It is a purpose-built classifier, not a generative LLM. We combined its detections with deterministic structural safeguards and span-resolution logic, then exposed redacted text and non-sensitive detection telemetry.

On an NVIDIA T4 inside our network, the result felt like a genuine step change.

The result was a 4.9x speed-up in median PII-stage latency, while both approaches reached a 96.6% evaluation pass rate. Most importantly, it was a win on both fronts: raw PII stayed inside our infrastructure, and the workflow began much sooner without giving up quality.

There were serving lessons hiding inside the win. A GPU does not automatically make concurrent HTTP requests parallel. We had to instrument queue time, configure microbatching explicitly, add backpressure, and distinguish model time from end-to-end latency.

Conclusion: we had achieved both goals. Redaction was much faster, and the original message stayed entirely inside our infrastructure. Everything looked solved.

Or so we thought.

Then we calculated the bill

The original deployment design kept one g4dn.xlarge GPU instance active in development, one in staging, and two in production for availability and deployments.

That was approximately EUR 1,343.76 per month, or EUR 16,125 per year, in compute for AIFindr Privacy alone. That bill covered only PII redaction, not the rest of AIFindr’s infrastructure.

We had an approach that solved privacy and latency exceptionally well, but its cost made the architecture economically unsustainable.

This was not a model failure. It was a system-design failure. We could have chosen a slightly cheaper GPU and reduced the bill a little, but not enough: the architecture would still have been far too expensive.

So we went back to the drawing board.

We gave an agent the goal. Then we let the loop run

The new objective was stricter than “find cheaper hardware”:

Preserve privacy, evaluation quality, and a clearly faster experience, while making the service economically sensible to operate.

We decomposed the GPU prototype into variables we could test independently: model artifact, numeric representation, runtime, CPU architecture, task size, batching, traffic shape, and deployment topology.

We used coding agents to make that breadth practical. We gave an agent a clear goal and ran it in an autonomous loop with scoped access to the repository and AWS. For each hypothesis, it researched the official artifacts, prepared a temporary deployment, ran evaluations and benchmarks, saved the evidence, removed the temporary infrastructure, and returned to the loop with the next hypothesis. It stopped when an approach met the goal.

That loop began pointing toward an answer we had barely considered. What if the GPU was not needed at all?

Breakthrough two: the surprising answer was CPU

It sounded wrong at first. Running the model on CPU felt like the route back to slow inference. The measurements told a different story.

That was possible because we were serving a small, task-specific classifier rather than a generative LLM, making an optimized CPU runtime a realistic option.

OpenAI had already published several ONNX variants of the same model. We tested the official artifacts without changing the weights ourselves. The smaller Q4 variants introduced new quality regressions, so we rejected them. The official INT8 artifact, model_quantized, passed.

INT8 preserved every baseline redaction we checked and reduced runtime memory to about 1.6 GiB, which meant a 4 GiB task was enough.

ARM64 changed the performance result too. On direct inference, a 2-vCPU ARM task reached a median of 115 ms, compared with 144 ms for the tested T4 path. Quality held, and AIFindr Privacy even outperformed GPT-5.4 Nano on the harder evaluation set (76% vs. 69% normalized pass rate, with half as many false negatives).

Could a CPU service really handle production traffic?

We used several months of AIFindr production analytics to reconstruct real peaks across all projects, then load-tested the exact topology we intended to deploy:

Two ARM64 Fargate tasks, each with 1 vCPU and 4 GiB, behind an internal load balancer.

The answer was yes. The pair comfortably handled loads above the peaks we had observed while keeping latency within our target and producing no HTTP errors. CPU was not merely viable in a benchmark; it was enough for our real production demand. Two tasks also provided availability during failures and deployments, with a clear horizontal scaling path as traffic grows.

From EUR 1,344 to EUR 119 a month

The production pair costs approximately EUR 59.55 per month. DEV and STG add approximately EUR 29.77 each, bringing the complete three-environment topology to EUR 119.09 per month. Compared with the original EUR 1,343.76 GPU design, that is a 91.1 percent reduction.

There was still one inconvenient fact to acknowledge: at today’s AIFindr traffic, GPT-5.4 Nano remains cheaper on direct inference spend. Being honest about that matters. We had dramatically reduced the cost of private inference, but had not yet crossed the pure-cost line against a usage-priced API.

The curves cross at roughly 4.1 times our July 2026 redaction baseline. That month’s projected traffic was approximately 65 percent above June, making the crossing point a plausible horizon rather than a distant thought.

But cost was never the only ledger. Even before that crossing point, raw PII stays inside our infrastructure and redaction is much faster. Those were the two outcomes our clients actually asked for. Lower cost at scale would simply become the third win.

What we actually gained

AIFindr Privacy ended with three concrete results: raw PII stays inside our infrastructure, redaction is substantially faster, and the selected design is 91.1 percent cheaper than the original GPU topology.

The model at its center is a classifier, not an LLM. That is part of the lesson: small, task-specific open-weight models can remove latency, cost, and data boundary problems from narrow parts of a larger agent workflow.

More importantly, this is no longer an experiment. It is an open-weight model running in production and a reusable way to investigate the next improvement: set the goal, give an agent bounded autonomy, evaluate every hypothesis, and follow the evidence.

The stack is becoming the surface

The leading AI SDLC products are converging around the same pattern: understand the task, inspect the codebase, plan, modify files, run tools, test, review, and submit work.

Claude Code is already positioned as a command center for multi-agent coding. Codex follows the same direction from OpenAI’s side. GitHub Copilot, Cursor, Atlassian Rovo, and others are moving along the same curve. The interfaces vary. The strategic pattern is becoming clear.

The real bottleneck is the enterprise context

Most regulated enterprises have enormous amounts of knowledge. They rarely have that knowledge in a form an agent can use reliably. The relevant context is scattered across repositories, Jira tickets, architecture documents, Confluence pages, Microsoft Teams threads, policy PDFs, risk registers, incident reviews, security exceptions, vendor assessments, and the heads of senior engineers.

Agents cannot reliably use knowledge that is scattered, stale, contradictory, or unowned. That creates the practical failure mode we see most often: the agent is intelligent, yet behaves like a newcomer.

It does not know which payment flow is sensitive. It does not know which customer field is regulated. It does not know which dependency is banned. It does not know which architecture decision came from a past incident. It does not know which tests matter, which reviewer owns the domain, or which evidence an auditor will expect.

A serious AI SDLC environment needs a managed domain context:

• System ownership.
• Data classes and regulated flows.
• Architecture rules.
• Mandatory controls.
• Approved patterns.
• Trusted tests.
• Human approval thresholds.
• Policy exceptions and expiry dates.
• Evidence required before merge or release.

This is engineering infrastructure.

Unlocking the data requires data engineering effort

Agent performance does not depend only on model quality. The harness around the agent shapes the outcome: which tools it can call, how data is retrieved, how results are ranked, and which sources are treated as authoritative.

Naively connecting Codex, Claude Code, or any other agent to every internal data source will not create a strong AI SDLC system. It will create a noisy context. The work is to build the data layer agents need: classify, normalize, index, filter, rank, deduplicate, resolve ownership, preserve lineage, and expose the right source at the right moment.

Regulated companies also need to separate working context from validated context.

Meeting transcripts, chat messages, draft documents, and ticket comments can help an agent understand current intent. They should not carry the same authority as ERP records, production systems of record, approved policies, signed contracts, validated requirements, or released architecture decisions.

A human-in-the-loop process can promote working context into validated context when needed. That promotion should leave evidence: who approved it, when, for which scope, and when it should expire.

For precision and cost, the long-term answer is usually some form of indexed knowledge layer: part structured data, part unstructured knowledge, part temporal memory. Call it a knowledge graph, a context graph, or an enterprise memory layer. The name matters less than the behavior. It should help agents answer the most frequent questions quickly, cite the source of truth, avoid stale or low-authority information, and keep LLM context manageable when the enterprise has too much data to fit into a prompt. Agents can always go back to the original source when the task requires it. The context layer decides what they should see first.

Skills are becoming the new runbooks

Skills make this direction concrete. They package instructions, resources, and optional scripts so an agent can follow a workflow consistently instead of relying on whatever an engineer remembers to say in chat.

For regulated AI SDLC, that matters. A good skill can capture the sources to inspect, the tools that are allowed, the actions that are forbidden, the tests that must run, the evidence that must be attached, and the risk threshold that requires escalation. It turns fragile, repeated guidance into a versioned operating procedure.

The best organizations will turn their senior engineering judgment into reusable agent runbooks: small enough to maintain, specific enough to guide real work, and explicit enough to audit.

Governance has to live inside the workflow

In regulated industries, governance cannot remain a policy document or a final review step. Agents can read files, run commands, call tools, open PRs, inspect logs, and propose changes. Governance has to live inside that workflow. As it happens with human employees, agents must be properly isolated by design, with a strong permissions system that makes it physically impossible for an agent to access the wrong data. This maps naturally to HIPAA safeguards, FDA Computer Software Assurance, and the NIST AI Risk Management Framework.

For AI SDLC, that means:

• Agent identity and access control.
• Tool permissions by task and risk level.
• Data boundaries and secrets protection.
• Branch protections and merge policies.
• Mandatory tests and security scans.
• Human approval gates.
• Audit logs for prompts, tool calls, models, and decisions.
• Evidence capture for regulated workflows.
• Incident review when agent-generated work fails.

Instructions guide behavior. Controls enforce behavior.

Measure ROI where it matters

Activity metrics are dangerous. More generated code can mean more review burden, more rework, and more risk.

AI SDLC redefines how you measure ROI. We’ve seen other metrics being more useful on this context:

• First-pass PR acceptance rate.
• Reviewer minutes per accepted change.
• Issue-to-merge lead time.
• Defect escape rate after agent-authored changes.
• Security findings by severity.
• Policy violations caught before human review.
• Percentage of agent changes backed by required evidence.
• Time to produce audit or release documentation.
• Repeated review comments converted into reusable skills or rules.

And those are just examples. The way you measure depends on your own environment. AI can help to measure, too.

The best AI SDLC program should feel faster to engineering and safer to risk, compliance, and security.

What questions CTOs should answer

Before scaling AI SDLC, leadership should answer a small set of hard questions:

1. Where does our domain knowledge live, and who owns it?
2. Which restrictions should agents learn, and which restrictions should agents be unable to bypass?
3. What does an acceptable AI-generated PR look like in our environment?
4. Which workflows can safely run with high autonomy?
5. How will we measure accepted work, rework, defects, policy violations, and audit readiness?
6. How will failed agent runs improve our skills, tests, rules, and context?

Topic on tamHub: AI SDLC →

Where the brand lives

Your brand probably lives in good shape. A Figma library with Variables and components. A brand portal on the web. A Notion page the team bookmarks. A senior designer who reviews the final pass. None of that is broken. None of it is what the AI reads.

Brand portals are optimized for human consumption: visual, browseable, narrative. The AI needs the same source in a different shape. AI didn’t create that gap. It made it visible. And now it produces against the gap at scale.

Move the brand to where the AI reads

If the AI produces, the brand has to live where the AI reads. Plain text, structured, in a single source, accessible to any tool your team opens tomorrow. Not as a replacement for your Figma library or your brand portal. As the layer the AI can actually read.

In April 2026 Google published the DESIGN.md spec under Apache 2.0: an open format that combines YAML for exact tokens with markdown for rationale, intent, and use cases. Any LLM reads it natively. So do agentic builders that follow the spec. So does any code editor with file access. It’s the brand turned into something machines can parse and humans can keep editing with a commit.

That’s the foundation we built The Agile Monkeys Brand Hub on.

The system

A single source: the DESIGN.md in a private repo. A hub that distributes it alongside SVG logos and ready-to-paste prompts. The tools your team uses for generative work read from the hub: agentic builders like Lovable, v0 or Bolt, AI code editors like Cursor, CLI agents like Codex or Claude Code. They produce a first pass. Humans iterate on it according to what the piece needs. Evaluators check that the iteration didn’t pull it off-brand. The loop closes.

Two kinds of prompts live in the hub: generators that produce, evaluators that judge. They never combine. It’s the same rule that keeps author and reviewer apart in any healthy process. Both are loadable, so any tool that reads markdown picks up the right one when it starts a project. That’s what turns a brand portal into something machines can operate.

The loop that matters: apply, correct, learn

Most teams stop at produce, validate, publish. The interesting loop runs one step further: produce, validate, learn.

The evaluator doesn’t block publication. It informs. Each gap comes with cause and consequence, and three open paths for the user: correct, publish with a notification to the brand guardian, or reevaluate whether the piece is in the right lane. When the user picks correct, the evaluator hands over a ready-to-paste prompt with the fixes. It never edits the piece itself. That keeps producer and judge separate by mechanism, not just by principle. The system’s authority lives in what it offers. Forbidding is not how it works.

If the evaluator keeps flagging the same gap across different pieces, that’s the system asking for adjustment. The pieces are doing their job. They’re surfacing where the source has gone stale. Every use becomes signal. Each piece that passes validates the system. Each piece that breaks refines it.

Updating the system means editing a file

A new decision (a component, a rule, a tone) opens a PR to the DESIGN.md from GitHub’s web UI. No terminal, no Git CLI, accessible to non-technical people. Automation handles what’s technical. Human guardians judge whether the change makes sense for the brand.

Once merged, the change is live. New pieces consume the new version automatically. Nothing else needs rebuilding. The source changed, and everything that reads it keeps reading it.

The window

The first draft is already on-brand, and AI speed stops fighting brand quality because the brand is already inside the prompt. The system also gets smarter with use: evaluations don’t only improve pieces, they move the system. The brand doesn’t petrify. It grows because coherence lives in the file, and the file is editable.

A brand only humans can read doesn’t survive AI. Your team already uses generative tools to produce. The open question is whether your brand can survive that production without you watching every piece.

We built this for ourselves, and it’s already producing. Brands hold their shape over the next two years if their guidelines are where the AI can read them. Otherwise the AI writes the brand.

The real failure mode wasn’t “the model got confused”

Our first lesson was that classic single-pass RAG was too weak for a corpus with heavy overlap and weak boundaries.

Different entities shared similar names while content overlapped semantically across contexts that had nothing to do with each other. Freshness made things worse: old chunks could stick around and remain plausible long after the underlying source had changed. A single retrieval pass would return something that sounded right while actually being wrong.

That last problem is the dangerous one. Retrieval failures in this kind of corpus tend to look plausible. The system finds a chunk adjacent to the truth, close enough to generate a confident-sounding answer but far enough to mislead anyone who trusts it.

In practice, that pushed us toward the worst possible coping strategy: one-off prompt tweaks for specific failure cases. A well-placed system instruction can rescue a demo, but it doesn’t make the retrieval layer any better at separating lookalike content. Over time, those patches multiply and become their own maintenance burden.

Once we framed the problem that way, the work that actually mattered turned out to be lower in the stack than we’d expected: metadata boundaries, chunk navigation, retrieval strategy, and verification loops.

If the knowledge changes at ingestion time, evaluate it there

The key shift was simple: run retrieval checks when content enters the system, rather than waiting for a human to notice a bad answer downstream.

That means treating ingestion as more than a data plumbing step. Ingestion is where retrieval quality can regress, because that’s where you decide how files get parsed, how content gets chunked, which metadata survives the process, and whether canonical identifiers and freshness signals make it through intact. If any of those decisions change, answer quality can change with them.

So we started embedding an evaluation step directly into the automated ingestion scripts, with the goal of verifying retrieval quality and raising alerts when something degrades.

The flow looks roughly like this:

graph TD
  A["Source Fetch"] --> B["Parse / Normalize"]
  B --> C["Chunk"]
  C --> D["Enrich with Canonical Metadata"]
  D --> E["Index"]
  E --> F["Run Retrieval Eval Set"]
  F --> G["Compare Against Baseline"]
  G --> H["Alert or Block on Regression"]

This does two useful things:

1. It moves quality detection closer to the change that caused it. You’re no longer waiting for a user to stumble into a regression three hours later.
2. It turns retrieval quality into something that can participate in release logic: if this ingestion run makes retrieval materially worse on known cases, it shouldn’t pass quietly.

We increasingly think this is the missing piece in a lot of RAG systems. Teams evaluate prompts, inspect traces after incidents, and review final answers. But they rarely evaluate the exact moment where the knowledge base changed shape.

Why we stopped trusting one retrieval pass

Moving evaluation earlier only helps if the retrieval stack itself surfaces ambiguity instead of hiding it.

For us, that meant replacing the single retrieval call with a more agentic pattern. We added hybrid search to avoid depending on one similarity mechanism, pagination so the system could look beyond the first page of results, and document-level drill-down so the agent could zoom into a promising source rather than guessing from a fragment. On the data side, we invested in strong metadata boundaries to prevent overlapping content from bleeding across contexts, and we preserved canonical IDs from the source whenever possible so disambiguation could rely on something sturdier than naming luck.

The contrast is instructive. A one-shot RAG call asks “what seems relevant right now?” and commits to whatever comes back. A search-plus-drill-down workflow lets the agent ask a follow-up: “what evidence belongs to this specific context, and what should I inspect next?” In domains where content overlaps heavily and naming is messy, that second question is the one that keeps the system from acting like a very confident intern with three browser tabs open.

Keeping the safety path and the experiment path separate

The other lesson was organizational rather than technical.

Once we started taking ingestion-time evaluation seriously, we realized we needed two distinct loops. The first is a safety loop for production ingestion: did content ingest cleanly, did retrieval regress on the evaluation set, and should we alert or block promotion? The second is an experiment loop for retrieval hypotheses: would different chunking work better, does a new file processing approach improve things, and which ideas are worth graduating into production?

These loops are related, but they shouldn’t be the same system. We started treating the experiment loop as its own harness rather than letting every idea leak directly into ingestion. That makes the work less glamorous but far more usable. You get a place to test retrieval hypotheses systematically instead of rediscovering the same failure mode document by document, demo by demo.

It also forces a healthy question that’s easy to avoid: what evaluation set and alert thresholds are strong enough that newly ingested content is safe to trust? That lives squarely in the boring-but-critical territory where production reliability actually gets built.

What we’d do differently from day one

If we were starting this system over, we’d be more opinionated about three things. We’d treat ingestion as part of retrieval quality from the start, rather than wiring in evaluation after the first few regressions forced our hand. Metadata boundaries and canonical identifiers would come before prompt-level fixes, because those fixes multiply fast and age poorly. And the agent would get real search and drill-down tools from day one instead of hoping a single retrieval pass could hold up against a corpus full of overlapping entities.

Key takeaway

If your retrieval QA starts when a user opens the chat, most regressions have already happened upstream during ingestion. Put evaluation there, add alerting there, and treat metadata quality as part of the product rather than cleanup work you’ll get to later.

The platform-first trap is still a trap

The first useful agent immediately exposes realities that architecture diagrams politely ignore.

For us, the bar was concrete: produce publishable content on a weekly cadence. Something a human reviewer could actually ship, not just find “surprisingly good.”

That bar made the real blockers obvious. Source access was uneven, and retrieval quality varied enough between runs that we couldn’t predict whether a given draft would be usable. Feedback loops depended on manual effort that didn’t scale. Review, which we’d initially assumed was temporary scaffolding, turned out to be a permanent part of the system. And the hardest problem wasn’t generation at all — it was earning enough operational trust that people would actually use the output.

A control plane on top of those problems would have been premature. A very organized way to scale fragile behavior.

One real agent gives you a better roadmap than a month of platform speculation, because it forces contact with failure modes instead of abstractions.

Cost dashboards aren’t ROI

Once we started looking at agent platforms more closely, a pattern jumped out.

In the products we reviewed — including Salesforce Agentforce, Microsoft Copilot Studio, Notion Custom Agents, Dust, and CrewAI — cost tracking was common. Model spend is visible, scary, and easy to graph, so it makes sense as a starting metric.

The interesting question for a real agent, though, is whether it reduced work, increased throughput, or improved a decision enough to justify existing. Those questions require mission-specific metrics.

For a content workflow agent, the dashboard we actually wanted looked like this:

graph TD
  M["Mission: Produce publishable technical articles"]

  M --> C["Cost"]
  M --> O["Outcomes"]
  M --> L["Learning"]
  M --> R["Risk"]

  C --> C1["Model/tool spend per run"]

  O --> O1["Drafts generated"]
  O --> O2["Drafts approved for publication"]
  O --> O3["Time to publish"]
  O --> O4["Edit rate / reviewer effort"]

  L --> L1["Feedback captured"]
  L --> L2["Lessons learned written"]
  L --> L3["Repeated mistakes retired"]

  R --> R1["Messages/posts sent"]
  R --> R2["Actions awaiting approval"]
  R --> R3["Write operations by channel"]

That’s a very different platform surface from “token usage over time.”

A platform that only tracks spend optimizes for cheaper wrong answers. Useful for finance, but nowhere near enough for operations. We increasingly think every serious agent needs a mission-specific ROI dashboard rather than a generic activity panel. Otherwise teams end up arguing about vibes, with a cost chart on the side pretending to be evidence.

Feedback can’t live in one person’s settings page

The second gap was how platforms handle improvement.

A lot of current tooling assumes a workflow where one person creates the agent, opens settings, rewrites the prompt, and tests again. That works for demos. It breaks down the moment a second person has an opinion.

In real work, feedback is distributed. One reviewer catches tone problems, another flags a missing source, and a third keeps noticing the same factual error the agent was already corrected for. If all of that learning stays trapped in the creator’s prompt window, the team isn’t improving the agent — they’re generating a queue of editorial cleanup.

We hit that directly. Written feedback wasn’t reliably changing later behavior, so the agent could repeat the same mistake with fresh confidence. One of the least charming traits an agent can have.

Our response was to move toward a simpler, more operational loop: let reviewers give feedback in the flow of work, let the agent produce a better revision from that feedback, and when corrections repeat, turn them into durable lessons learned that persist across runs.

Even our first implementation made the real requirement clear. Improvement has to work across the whole team and compound over time, because individual prompt tweaks just don’t accumulate. The early version also surfaced its own limitations quickly — feedback handling only worked on newly generated pieces, which is exactly the kind of boundary you only notice once the workflow is real.

That’s why we now treat feedback capture as platform-level work.

Governance starts with action visibility

The third lesson was even more operational.

During testing of a Slack-connected workflow, we noticed messages had been sent without the user realizing they’d already gone out. In an internal setting that’s merely annoying, but in a customer-facing workflow it’s the kind of thing that ends pilot programs.

This is where a lot of “agent governance” talk becomes too abstract. Before you design exotic policy engines, there’s a simpler question worth answering: can the user see exactly what the agent did, where it did it, and whether it acted with explicit approval? Without that visibility, you have permissions plus optimism — which isn’t the same thing as governance.

The minimum useful layer is boring in the best possible way: a clear action log, explicit approval points for risky writes, visible traceability for messages sent on a user’s behalf, and easy ways to reduce permissions when trust hasn’t been earned yet. If using the agent makes people want to revoke write access, that’s worth treating as product feedback rather than user hesitation.

We now think action visibility is one of the first platform primitives worth building, especially for anything that writes into shared systems.

What one real agent taught us to build next

After all of this, the platform shape that feels justified is much smaller and much sharper than the original control-plane fantasy.

We’d start by making it clear what the agent can read and which missing source would make it obviously wrong. Then we’d add mission-specific outcome tracking, because measuring useful work is the only way to get past “it seems good” as an evaluation strategy. From there, the priorities are a durable review loop where feedback actually improves future runs rather than evaporating after one correction, and action logs with approval controls for anything writing into Slack, docs, tickets, or external systems. Only after those pieces were earning their keep would we invest in broader abstractions like registry, orchestration, and generalized runtime.

The sequence looks like this:

graph TD
  A["Pick one real agent"] --> B["Define operational success metric"]
  B --> C["Connect the sources it actually needs"]
  C --> D["Instrument outcomes and review effort"]
  D --> E["Make feedback durable"]
  E --> F["Log risky actions and approvals"]
  F --> G["Abstract common pieces into a platform"]

Less glamorous than “let’s build the agent platform.” Also much harder to fake.

Key takeaway

If you’re building agents, don’t stop at source connectors and spend dashboards. One real workflow will quickly show you what the platform is actually missing: outcome tracking, durable team feedback, and action visibility. Build those before you fall in love with orchestration diagrams.

Most “implementation estimates” are really estimates of hidden system behavior

On greenfield work, scope is usually visible. On mature systems, it often isn’t.

A request that sounds small can sit on top of a surprising amount of invisible machinery. Routing rules exist but aren’t documented. Background jobs only become obvious once you trace them end to end. Authorization assumptions hide inside service boundaries. Operational controls live outside the application layer entirely. Downstream notifications, retries, and failure handling never came up in the original request.

This is where teams get trapped. Somebody hears “we need feature X,” imagines a clean standalone implementation, and estimates accordingly. Two days later, the real work shows up: “Actually it has to fit the existing intake path, reuse the current workflows, preserve existing security controls, and behave consistently with the rest of the platform.”

At that point, you’re no longer estimating implementation. You’re paying interest on unknowns.

We’ve found LLMs are unusually good at compressing that discovery phase.

Start by asking for a flow map

The first thing we want from the model is a map of what exists, not a recommendation for what to build.

A good prompt is closer to technical archaeology than brainstorming:

Read this codebase and map the end-to-end flow for capability X. Cite the files, routes, handlers, jobs, and config involved. Separate clearly between: 1) behavior directly supported by source code 2) likely behavior inferred from naming or structure 3) important unknowns that require human confirmation

💡 A note on prompts: These are starting points, not recipes. Every codebase has its own idioms, and you’ll need to adjust wording, scope, and follow-up questions to get useful output. If you copy these verbatim and get mediocre results, iterate on them before writing off the approach.

That three-way distinction matters. Without it, the model blends observation and invention into one very confident paragraph. You get a map you can’t trust.

What we want instead is a reviewable pipeline:

graph LR
  A["New request"] --> B["Entry point"]
  B --> C["Routing / orchestration"]
  C --> D["Validation & authorization"]
  D --> E["Storage / persistence"]
  E --> F["Downstream processing"]
  F --> G["Notifications / side effects"]
  G --> H["Error handling / retries"]

That artifact makes the existing system discussable. Vague “I think it works like this” turns into something the team can actually review. And it immediately exposes where the request assumes something exists that doesn’t, or ignores something that does.

We’ve had good results cross-checking the same mapping task with more than one model, then comparing outputs. Disagreement between models is informative: if two produce different maps from the same repo, humans should slow down and look closer.

Use the flow map to generate the questions that change the estimate

Once we have a draft map, the next step is structured questioning.

This is where the LLM earns its keep as a question generator for missing work. We ask something like:

Given this observed flow, what security, operational, and scope questions must be answered before deciding whether to integrate, extend, or rebuild? Group them by risk area and keep each question specific.

The checklist that comes back is rarely publishable as-is, but it’s a strong starting point. We rewrite it in our own words and send it for confirm-or-correct review.

The categories that repeatedly change the estimate are consistent enough to be worth listing:

• Authentication and trust boundaries. Where does this input come from? Where is identity established? What parts of the trust chain live in the application versus external infrastructure?
• Authorization and tenancy. Who can trigger this flow, how is access mapped to the right account or tenant, and what happens when identity is known but permission isn’t?
• Validation and content safety. Are we validating structurally, semantically, or both? Are there hidden rules around formats, payload sizes, or attachment types? Where do malformed inputs go?
• Abuse prevention and backpressure. Where is throttling enforced? What happens under spikes, replayed requests, or noisy neighbors? Are retries idempotent, bounded, and observable?
• Downstream side effects. Which jobs, notifications, webhooks, or derived artifacts depend on this path? What must happen synchronously versus eventually? What constitutes partial success?

This is the point where a supposedly small feature reveals itself as either a straightforward extension, an integration into a partially built capability, or a genuine architectural mismatch that justifies a rebuild.

Without the checklist, teams discover that late. With it, they discover it while decisions are still cheap.

LLMs are surprisingly good at spotting scope drift

One of the more valuable patterns we’ve found is using the model as a contradiction engine.

Once the existing flow is mapped, we give the model the observed architecture (supported by code and config) alongside the newly requested behavior (as described by stakeholders). Then we ask it to highlight tension:

Compare the requested change with the current implementation. What assumptions would break if we implement it the proposed way? Which parts look like an extension of the current design, and which parts imply a different architecture?

This works well because scope drift enters projects in harmless-looking language. “Can we make it work through this other path too?” “Can we move this responsibility to another component?” “Can we simplify by bypassing the existing workflow?” Those sentences sound small. A model that has already built a repo-level map can surface the consequences quickly: where trust boundaries shift, where orchestration gets duplicated, where isolation breaks, where failure handling becomes inconsistent, or where the operational burden exceeds anything the original estimate accounted for.

We treat that output as a sharp draft of the conversation humans need to have, not as a conclusion in itself.

The guardrails that keep this useful

This approach only works if the team is disciplined about how the output is used. Our operating rules:

• Every claim should trace back to source: files, handlers, config, jobs, tests, or logs. If the model can’t point to evidence, the claim stays provisional.
• Use confirm-or-correct review. The goal is helping maintainers validate or falsify the map quickly, not producing a document that looks authoritative on its own.
• Separate observed behavior from desired behavior. Mature systems are full of “it should do this” stories. That’s not the same as “it does this today.”
• Treat infrastructure as a first-class unknown. A repo rarely contains the whole truth. Critical behavior may live in gateways, queues, policies, or provider configuration.
• Pull security and operations questions earlier, not later. The model shouldn’t replace security review. It should make sure it happens before anyone commits to a plan.

This makes the LLM less magical and more useful. It becomes a force multiplier for engineering judgment.

Key takeaway

The best use we’ve found for LLMs in “integrate vs rebuild” decisions is reducing ambiguity before planning begins. If the model can help you produce a reviewable end-to-end flow map, a concrete checklist of hidden security and operational assumptions, and a clear view of where the new request contradicts the current design, you’ve already shortened the riskiest part of the project.

Before your team commits to the next “small” change in a mature system, try letting the LLM read the codebase first. Force it to cite its work. Make humans confirm the map. Then decide whether to integrate, extend, or rebuild.

From Flexible Integrations to Production Constraints

As a SaaS product, we are not integrating one clean product catalog. We need to support many different businesses, each with its own search API, payload shape, and data quality quirks. Unless those differences are normalized early, integration flexibility turns into token overhead, latency, and frontend fragility.

Where Latency Was Really Coming From

The bottleneck was not just the model. It was everything we let flow through it.

In some tenant flows, the LLM was getting too much product data, output schemas that were too broad, and response structures that were too close to UI rendering.

The model was doing too much. More context in, more structure out, more latency and cost.

The Four Optimizations

<PS>[id1,id2,id3]</PS>

A Simplified View of the Flow

After those four changes, the architecture becomes much simpler: the model only sees a compact contract, and the frontend renders from deterministic product memory.

Business Impact and Trade-Offs

The business impact was straightforward: faster answers, lower cost per conversation, and a more consistent product experience. Once the model stopped carrying oversized payloads and rendering-heavy structures, the UX became more responsive and the unit economics became easier to control.

The trade-off is that more responsibility moves into application logic. You need stable product IDs and clear hydration rules across stores. In practice, that is usually a good trade, because deterministic application logic is easier to debug and scale than pushing more structure through the model.

If you only measure three things, measure input tokens per turn, end-to-end latency (especially median and P95), and cost per conversation.

Before vs After: Same Query

Using the same query in both versions of the flow:

This is a single-query reference point, but it clearly shows the direction and magnitude of impact.

What This Looks Like at 300,000 Queries

If this query were representative of 300,000 queries, the operational impact would be:

And using GPT-5.1 pricing as of March 4, 2026 ($1.25 / 1M input, $0.125 / 1M cached input, $10 / 1M output), the cost impact would be:

This is a directional estimate, not a measured production average.

Final Takeaway

The less context an agent has to process, the faster and cheaper it will be. That sounds obvious, but it is easy to lose that discipline as a product grows.

More tenants, more integrations, and more product requirements usually mean more fields, broader outputs, and more rendering logic creeping into the model path. Unless those boundaries stay tight, inefficiency becomes the default.

In practice, the biggest gains often come from better contracts, not bigger models. Optimize what goes around the model, not just the prompt inside it.

Start by auditing one high-traffic tool-output path and measuring token, latency, and cost deltas before expanding.

Why You Can’t Just “Update Everything”

The instinctive response to a vulnerability report is to update everything to the latest version. Simple, right?

In practice, this is rarely simple — and for a mature enterprise application, it can be nearly impossible without effectively rewriting the system.

Our application had already been partially modernised over the years. It was running Spring 5 and Spring Security 5, with a commercial support contract covering the harder-to-maintain legacy dependencies. That commercial arrangement had kept the system compliant, but it was not a permanent solution.

The application’s entire request-handling architecture, page navigation logic, and form processing were built around Struts. Spring was the backbone of dependency injection, database access, and transaction management — essentially the skeleton of the entire system.

The modern, secure versions of both ecosystems had moved on dramatically:

• Spring 6 requires the Jakarta EE namespace — an industry-wide rename of the javax.* package family to jakarta.* — and had dropped support for Java versions below 17
• Spring 6 had completely removed its built-in Struts integration module (spring-struts), which existed precisely to wire Spring beans into Struts actions
• Struts 1.2.9 was written against the old javax.* namespace and was never updated for the jakarta.* world

In plain terms: the two frameworks we depended on had evolved in ways that made them fundamentally incompatible with each other. You could upgrade one or the other, but not both — and you needed both.

This is the kind of problem that, in the past, would have meant a multi-year, multi-million dollar “big bang” rewrite. Write everything from scratch in modern technology. Hope nothing gets missed. Pray the business logic gets reproduced faithfully.

We took a different approach.

Bringing AI Into the Room

We engaged AI as an active participant in the modernisation process — not just as an autocomplete tool, but as a technical collaborator that could hold the entire codebase in context, reason about complex dependency chains, and propose solutions to problems that had no obvious off-the-shelf answer.

The first task we gave it was deceptively simple: understand the problem and tell us what is real.

The Art of the False Positive

Here is something that security reports almost never tell you clearly: not every vulnerability listed is actually a vulnerability in your application.

Automated tools work by scanning the names and version numbers of the libraries you use, then matching them against a global database of known vulnerabilities (the NVD — National Vulnerability Database). This matching process is imprecise. The tools are designed to over-report, because missing a real vulnerability is far worse than flagging a false one.

But for a team trying to act on those findings, false positives are a serious problem. They dilute attention, waste engineering time, and — most dangerously — can lead to a kind of learned helplessness where the team starts dismissing reports entirely.

AI was extraordinarily effective at distinguishing real threats from noise. Here are the categories of false positives it identified:

A Structured Attack: Triage Before Tools

With the false positives cleared away, the team had a much cleaner picture of the real work ahead. But even after stripping out the noise, a meaningful backlog of genuine issues remained — and attempting to fix everything at once is a reliable path to paralysis.

The next step was to classify the remaining vulnerabilities into three tiers, based on the complexity of the fix required:

Easy: Simple version bumps. A newer release of the same library already contained the fix, the update was backwards-compatible, and changing a version number was all that was needed. These could be resolved quickly and in bulk.

Medium: Updates that required care. The fix existed, but the newer version involved API changes, configuration adjustments, or testing to confirm compatibility with the rest of the stack. These needed individual attention but had clear, documented upgrade paths.

Difficult: Cases where no straightforward fix existed — where upgrading one component would break another, or where the required version was fundamentally incompatible with a core framework already in the application.

This tiered approach transformed the work from a single overwhelming task into a structured campaign. The team started with the easy wins, clearing a large portion of the backlog quickly and building momentum. Medium-complexity fixes followed, each one narrowing the gap further. By the time the team arrived at the genuinely difficult cases, the list was short, the context was clear, and the focus was sharp.

The hardest case — the one that required the most creative engineering — was the incompatibility between Spring 6 and Struts 1.x.

The Incompatibility Problem: Building a Bridge

After eliminating the false positives, a core set of genuine modernisation challenges remained. The biggest: how do you upgrade to Spring 6 when your entire UI layer depends on a framework (Struts 1.x) that Spring 6 refuses to support?

This is where AI moved from analysis into invention.

The Result: 229+ Vulnerabilities → 0

After completing the modernisation:

The last row is the one that matters most to the business. The application behaves identically to users. No features were removed. No business logic was altered. The security posture was transformed without any of the risk that a rewrite would have introduced.

What This Tells Us About AI-Assisted Modernisation

This project illustrates something important about where AI adds genuine value in software engineering — and it isn’t just about writing code faster.

The hardest problems in legacy modernisation are not coding problems. They are reasoning problems:

• Is this vulnerability real, or is it a false match?
• Does this CVE apply to the way we actually use this library?
• Is there a way to satisfy two incompatible requirements simultaneously?
• What is the minimum change that achieves the security goal without breaking the application?

These questions require holding a large amount of context simultaneously, understanding the semantics of security advisories, and designing solutions that work within tight constraints. They are exactly the kind of problems where AI currently provides the most leverage.

The compat layer is a particularly good example. A human engineer could have designed it — but it would have required deep expertise in three different technology stacks (Struts internals, Spring internals, and the javax→jakarta migration) and the creative insight to see that a thin translation layer was the right architecture. AI could draw on all of that knowledge simultaneously and propose a coherent design quickly.

False positive analysis is another. Working through 229+ reported vulnerabilities one by one, understanding the semantics of each CVE, matching it against the actual usage patterns in the codebase, and writing precise suppression rules with documented justifications — this is tedious, high-skill work. AI can do it faster and more consistently than a human team under time pressure.

The Takeaway

Legacy modernisation projects have a reputation for being long, expensive, and risky. That reputation is largely deserved — when they are approached as “big bang” rewrites.

A better approach is surgical: understand precisely what needs to change, distinguish real risk from noise, build bridges where direct replacement is impossible, and preserve everything that works.

AI doesn’t replace the engineers who understand the system. But it amplifies what they can do — turning a project that might have taken quarters into one that takes weeks, and turning a vulnerability report that looked like a condemned building into a system that is genuinely, verifiably secure.

Zero vulnerabilities. Zero lines of business logic rewritten. One very large problem, solved.

Written February 2026.

The New Feature-Building Paradigm

In the agent-native mindset, every new feature begins with a different set of questions:

• What tools do I need to provide so that an agent can do this?
• What should the agent reason about vs. what must be deterministic and controlled via tools?
• What prompt (instructions + constraints) will consistently guide it to the right behavior?
• How do I evaluate it to verify it does what it is supposed to do?

This changes how you design software.

Instead of writing rigid flows for every edge case, you build capable agents with a set of tools, a prompt contract, an evaluation harness, and a loop for autonomy and improvement. That is the new stack.

1) Tools Are the Product Surface Area for Agents

If agents are going to run your features, they need two things: the ability to get the right information, and the ability to act on your system safely.

That is what tools are: controlled interfaces to your product.

2) Prompting Still Matters: Even More Than Before

Yes, prompting is still key. Arguably more key now.

As models get better at following instructions, the prompt becomes less “a suggestion” and more like a specification. It determines whether the behavior is mediocre and inconsistent or robust and repeatable.

3) Evaluation Is the Backbone, and It Must Be Integrated into the Product

This is the part most teams under-invest in, and it is the reason many “agent features” remain demos.

Evaluation is not a separate phase. It is part of the system. If agents are performing product work, then evaluations must be automated, continuously running, deeply integrated, and ideally improved with agents themselves.

Toward Autonomy: Less “Human in the Loop” by Design

The goal is not to guide agents constantly. The goal is to make them autonomous: self-validate what they do, self-correct when something goes wrong, self-test changes, and automatically deploy or publish when confidence is high.

In other words, keep humans in the loop only where it matters: reserve manual review for critical or high-risk actions, and let the rest run automatically. That is how agent-native products evolve faster, scale further, and deliver more value as autonomy increases.

Autonomy does not come from a bigger model. It comes from the triad:

Tools + prompting + integrated evaluation

That is the formula.

The Real Takeaway

If you are still thinking “How can AI help me build my product faster?” you are already behind.

The new question is:

How can I build my product with agents as first-class components, designed for autonomy, determinism, and continuous evaluation?

You will only really understand the leverage once you build at least one feature this way end-to-end.

And once you do, it becomes hard to go back.

What Static Re-indexing Can’t Fix 🎯

The typical search pipeline already reindexes every new or edited SKU overnight, so the catalog itself is always up to date. What isn’t fresh are the context signals, those short-lived moments when customers want a different flavour of “relevance” without any change in product data.

Re-training or re-ranking for every event is overkill, slow, GPU-hungry, and risk-heavy in prod.

We need a surgical dial that we can turn on/off per query, per segment, or per campaign without touching product information (description or embeddings).

Our Angle: the Embedding Lens 👓

How We Built It 🔧

{
  "query": "search text query",
  "query_embedding": "embedding representation of the query",
  "product_id": "unique identifier of the product",
  "product_embedding": "embedding representation of the product",
  "len_score": "adjustment parameter indicating the desired similarity level"
}

class EmbeddingTransformer(nn.Module):
    def __init__(self, dim, hidden_dim):
        super().__init__()
        self.transform = nn.Sequential(
            nn.Linear(dim, hidden_dim),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(hidden_dim, dim)
        )

    def forward(self, x):
        delta = self.transform(x)
        return delta

def _cosine_similarity_regression_loss(self, q, p, target):
    p_normalized = F.normalize(p, dim=-1)
    cos_sim = (F.cosine_similarity(q, p_normalized, dim=-1) + 1) / 2
    return F.mse_loss(cos_sim, target)

query_tensor = torch.tensor(query_embedding, dtype=torch.float32).reshape(1, -1)

delta = self.model(query_tensor)
adapted_query_embedding = F.normalize(query_tensor + delta, dim=-1)

alpha = 0.7 # Blending factor

weighted_query = F.normalize((1-alpha) * query_emb + alpha * adapted_query_emb, dim=-1)

from seasonal_lens.common.metrics import (
    load_search_results,        # jsonl → {query: [ids]}
    load_scores_from_jsonl,     # jsonl → {query: {id: score}}
    precision_at_k
)
import numpy as np
from pathlib import Path

# --- 1. Load artefacts ----------------------------------------------------
variant   = load_search_results(Path("results/lens.jsonl"))
relevance = load_scores_from_jsonl(Path("labels/relevance.jsonl")) # cosine
summer    = load_scores_from_jsonl(Path("labels/summer.jsonl"))    # 0-1 summer-score

# --- 2. Compute metrics ---------------------------------------------------
def mean_prec(results, labels, thr, k=5):
    return np.mean([
        precision_at_k(
            [labels[q].get(pid, 0.0) for pid in ranks],
            k=k,
            threshold=thr
        )
        for q, ranks in results.items()
    ])

prec5        = mean_prec(variant, relevance, thr=0.67, k=5)
summer_prec5 = mean_prec(variant, summer,    thr=0.70, k=5)

print(f"Precision@5        : {prec5:.3f}")
print(f"Summer-Precision@5 : {summer_prec5:.3f}")

def precision_at_k(scores, k, thr):
    hits = sum(s >= thr for s in scores[:k])
    return hits / k

The numbers: Decoding the Results 📊

After training and evaluation, the results confirmed the value of embedding adaptation through the Summer Lens. The best-performing setup used a 50/50 blend of the original and adapted query vectors, successfully capturing the seasonal relevance boost without sacrificing the baseline precision.

Even better, performance remained production-ready: added latency averaged just 1.04 ms per query, making the lens suitable for real-time search pipelines with no noticeable overhead.

What We Learned (Tips) 🎓

1. Cheap synthetic data is gold. OpenAI batch prompts gave us realistic, summer scores for all the catalog items (11k) in less than a day.
2. Screenshots trump spreadsheets. Putting results side‑by‑side makes the shift instantly clear to business stakeholders, far better than dropping a table of numbers, so every evaluation run now auto‑exports that comparison image.
3. Trust, but eyeball. When we build the len_score (relevance + summer vibe), the weightings can accidentally tip the scale, everything looks “very summer” or “hardly summer” at all. A quick histogram of those combined scores shows how much variety we really have on the summer axis, so we can rebalance triplets before training and avoid teaching the lens a one‑note view of the catalogue
4. Blending is a secret weapon. That extra knob α lets us decide how much lens power to apply, α = 0.5 hit the sweet spot and saved days of hyper‑param tinkering and code churn.
5. Version everything. We store triplet files and LLM scores on Hugging Face Datasets; rolling back a bad run is just changing a dataset tag.

Conclusions 🎯

The Embedding Lens proves that contextual search can be made both agile and production‑ready. By adding a lightweight, toggleable layer to our existing embedding pipeline, we can deliver seasonally or thematically tailored results without re‑indexing, retraining the base model, or adding operational complexity, while preserving production‑grade performance and simple deployments at scale.

Key takeaways:

• On‑demand adaptation: Activate or retire themed search experiences in seconds, no catalog reprocessing required.
• Faster data prep: LLM‑generated synthetic data cuts human labeling effort, enabling rapid rollout of new lenses aligned to business needs.
• Rapid, low‑cost iteration: The lens architecture trains economically, supporting quick prototype-test-refine cycles.
• Production‑safe latency: ~1.04 ms added per query, effectively negligible for user experience.
• Risk‑mitigated rollout: Straightforward A/B testing lets us trial multiple themed searches across segments without infrastructure changes.
• Built for extensibility: Supports lens composition (e.g., “summer+party”), multi‑aspect context (weather, location), and behavior‑adaptive lenses, without retraining the foundational embedding model.

Overall, the embedding lens approach advances our search architecture toward a more responsive, context‑aware system that scales operationally and economically, making continuous innovation practical in production.

At a glance

• Scope: 5 providers, 6 document types, images and PDFs
• Metrics: accuracy, average latency, and cost
• Runs: 64 experiments using a reproducible harness
• Default pick: GPT-5.1 for accuracy, cost, and latency balance
• When to specialize: AWS or Azure for passport-heavy ID workflows

Quick recommendations

• General pipelines: GPT-5.1 as the default; consider Claude Sonnet 4.5 only when a 1M context window is required
• IDs (passports): AWS Textract or Azure ID model; GPT-5.1 is the strongest LLM choice
• Defer: Google DocAI unless you are investing in custom processors
• QA: Plan human review on roughly 30 percent of outputs

Executive summary

• Top performers: GPT-5.1 leads with 73.2 percent accuracy, production-ready average latency (~12.3 seconds), and very low cost (~$0.00054 per document). Claude Sonnet 4.5 matches accuracy but is slower and 53x more expensive.
• OCR specialty: AWS Textract and Azure excel on passports and other government IDs thanks to specialized models, but fall short on general documents where LLMs dominate.
• Cost value: GPT-5.1 delivers near state-of-the-art accuracy at almost the lowest cost.
• Not production ready: Google DocAI underperformed and would require substantial custom development.
• Testing and reproducibility: All benchmarks were run with the open harness, document samples, and evaluation scripts in document-data-extraction-benchmark.

Why we ran this evaluation

Operational workflows depend on accurate extraction of structured data from IDs, vital records, and tax forms. Accuracy, speed, and reliability directly affect automation quality and cost. We tested five providers on eight documents to see which vendor delivers the best real-world performance today.

Setup overview

Key findings

1. LLM vision models outperform classic OCR on general documents. GPT-5.1 and Claude Sonnet 4.5 reach 70 to 75 percent accuracy on certificates and tax forms where OCR lags at 0 to 40 percent.
2. Specialized OCR wins for narrow ID use cases. AWS Textract and Azure hit around 90 percent accuracy on passports due to ID-trained models; their accuracy drops sharply on certificates and tax forms.
3. GPT-5.1 is the composite leader. It ties for highest accuracy (73.2 percent) with Claude Sonnet 4.5 while being faster on average (~12.3 seconds vs ~16.3 seconds) and far cheaper (~$0.00054 vs ~$0.0286 per document).
4. Providers with major gaps. Azure is strong on passports but inconsistent elsewhere without configuration. Google DocAI showed the lowest accuracy and needs custom processor development.

Provider snapshots

Performance by document type

• Passports: OCR tools (AWS Textract, Azure) dominate passports with MRZ-trained models at roughly 90 percent accuracy. GPT-5.1 is the best LLM here (~80 percent), closing most of the gap while staying far cheaper.
• Driver licenses: All providers struggle; layout variability is the main issue and no model clearly dominates.
• Birth and marriage certificates: GPT-5.1 and Claude Sonnet 4.5 reach 75 to 90 percent accuracy. OCR models often remain below 40 percent.
• Tax forms (W-2 and 1040): LLMs show strong semantic understanding, delivering around 80 to 100 percent accuracy on many fields. Claude Sonnet 4.5, GPT-5.1, and Textract cluster around ~91.7 percent on W-2 and ~80 percent on 1040.

When to choose each provider

• Choose GPT-5.1 if you want the best composite score (73.2 percent accuracy, ~12.3 second average latency, ~$0.00054 per document) across mixed document types with a single default.
• Choose Claude Sonnet 4.5 if you need a 1M token context window for long or complex documents and can accept slower latency and much higher cost for the same accuracy.
• Use AWS Textract or Azure only if you have ID-heavy workflows (especially passports) and need maximum ID accuracy at the expense of general-document performance and higher page-based pricing.
• Avoid Google DocAI for now; low accuracy outweighs its speed advantage unless you plan significant custom processor work.

Strategic recommendations

1. Adopt GPT-5.1 as the default provider for general-purpose document extraction across mixed document types.
2. Use AWS Textract or Azure ID models for passport-heavy or government ID workflows that demand near 90 percent ID accuracy.
3. Reserve Claude Sonnet 4.5 for scenarios where the 1M context window is a hard requirement.
4. Budget for human review on roughly 30 percent of outputs given the current accuracy ceiling of about 73 percent.

Phase 1: The Limitation of “Agent Mode” (ChatGPT)

My first approach was to use ChatGPT’s Agent Mode. The premise is seductive: describe a user flow in natural language and watch the AI navigate, click, and validate results.

The technology works, the results are surprising, and the agent’s capability borders on the magical. For prototyping quick flows, it is an exceptional tool. However, when attempting to translate this into an engineering environment, the limitations became evident:

• Lack of CI/CD Integration: The execution lives isolated within the chat. There is no API to trigger these tests automatically after a Pull Request.
• Collaboration and Versioning: Relying on manually copy-pasting prompts eliminates any real automation benefit. Without code, there is no version control (Git), no execution history, and team collaboration becomes chaotic.
• Human Dependency: In flows requiring authentication or credential management, the agent often halts, waiting for human intervention.
• Non-deterministic Environment: Since you don’t control the browser or the execution context, reproducing a specific error for troubleshooting is practically impossible.

Conclusion: Chat is ideal for assisting, iterating, and validating. Code is what allows us to scale, automate, and monitor.

Phase 2: Programmatic Architecture with Stagehand

To address the scalability and operational issues, I migrated to Stagehand. This tool abstracts away the complexity of “human navigation” but lives within our code (TypeScript).

The brilliance of Stagehand is that it doesn’t try to reinvent the wheel; instead, it introduces intention-based interaction primitives:

• act() (The Executor): Replaces the traditional click/type. You don’t tell it where to click (selector); you tell it what you want to achieve.
  • Ex: await stagehand.act("Log in with the saved credentials");
• extract() (The Analyst): Converts unstructured HTML into typed JSON objects.
  • Ex: “Extract the price and stock of this product.”
• eval() (The Judge): The evolution of assertions. It evaluates logical conditions based on visual and contextual information.
  • Ex: const isError = await stagehand.eval("Is there a wrong password error?");
• observe() (The Eyes): Allows the script to understand what interactive elements exist in the current state before making a decision.

With these primitives, we shift from “giving instructions to the DOM” to describing behaviors.

The Technical Architecture: Browserbase and Gemini

While Stagehand is versatile and can run locally or in headless mode on GitHub Actions, basic execution often falls short for a demanding production environment. Validating on a standard runner works but lacks the deep observability necessary for a modern QA pipeline.

That is where we consolidated the technical architecture:

Technical Comparison: Imperative vs. Declarative

The deepest change isn’t just the tools, but the paradigm. Until now, UI testing was written in imperative code, which is very brittle in the face of frontend changes. Stagehand, with its AI integration, allows us to define tests based on intentions.

Traditional Approach (CSS/XPath Selectors): Requires knowing the specific name of identifiers. It is fragile: if a developer changes a CSS class, the test breaks.

// If the ID changes or the button moves, the test fails
await page.type('#username-field', process.env.USER);
await page.click('.submit-btn-wrapper > button');

// Cryptic logs if it fails
await expect(page.locator('.dashboard-header')).toBeVisible();

Modern Approach (Stagehand + AI): More resilient and readable. The AI interprets the interface as a human would, regardless of changes in the underlying code.

// The instruction describes the intention, the AI solves the execution
await stagehand.act("Log in with standard user credentials");
await stagehand.act("Navigate to the configuration panel");

// Semantic validation
const isVisible = await stagehand.eval("Verify that the sales chart is visible");
if (!isVisible) throw new Error("The chart did not load");

Instead of tying ourselves to a selector name, we tie ourselves to the behavior expected by the user.

Use Cases and Conclusion

This architecture is not limited to QA. The ability to navigate the web intelligently opens doors to broader automations:

• RPA (Robotic Process Automation): Automating tedious administrative tasks, such as monthly invoice downloads or routine dashboard updates.
• Synthetic Monitoring: Periodic validation of critical flows in production from different geolocations and devices.
• Resilient Data Scraping: Extracting data from highly dynamic sites where traditional scrapers, based on rigid selectors, frequently fail.

The era of maintaining manual CSS selectors is coming to an end. The era of designing behaviors and letting AI execute has just begun.

What Really Matters in Evaluations

⚙️ Process data at scale.
We generate millions of events every month from our search and chat systems. The first challenge is building reliable data transformation pipelines — in our case, we lean on virtually unbounded cloud databases and lambdas to collect and process events efficiently, but there are many ways to do this. The key is ensuring the data can flow, transform, and aggregate without friction.

📏 Define the right metrics.
Decide what you want to measure and why. Sometimes it’s simple metrics (like response times or number of results returned), and sometimes it’s more complex (like LLM-based verdicts or semantic accuracy scores). These definitions often come more from product and business needs than from engineering, but they determine whether you have your product under control or not.

📊 Visualize results clearly.
Each metric should be visible and easy to interpret — whether in a table, a chart, or a trend graph. The goal is clarity, not complexity.

🧩 Evals are a source of very nice reference datasets.
Evaluations are an efficient way to label data at scale, and making a human-curated selection of this data produces high-quality datasets that you can use to benchmark new versions of your system before they go live, and to fine-tune or train better models over time (increasing quality and performance and reducing cost).

Borrowing the Mindset from Product Analytics

In many ways, building AI evals isn’t so different from tracking product KPIs. The main difference is that much of our data is unstructured — often text — and requires LLMs to interpret. But the principles are the same: focus on data, metrics, and insights, not on reinventing analytics tooling.

That’s why we now rely on existing analytics platforms and standard cloud services to process and visualize events and evaluation results. That way we spend most of our time where it truly matters — defining metrics, building robust pipelines, and making decisions based on real data.

By fostering a culture of continuous evaluation — one built on clear metrics and real datasets — we’re not just measuring quality. We’re creating the foundations to improve it every single day.

And that’s the real goal of AI evals: not building another tool, but building understanding.

The “Cognitive Core” and Externalized Memory

Karpathy’s central idea is to separate a model’s reasoning from its memory. Instead of storing vast knowledge in weights, he suggests keeping a compact “cognitive core” focused on reasoning and problem-solving, while retrieving facts and context from external systems. This underscores the growing importance of information retrieval and echoes our belief that GenAI should be viewed as an enabler for processing and understanding unstructured data—not as a database. The real opportunity lies in building powerful retrieval systems that amplify smarter reasoning models, especially when working with private or proprietary data that isn’t publicly available.

The Decade of Agents — Not the Year

Karpathy predicts that this will be the decade of agents, not the year. Building robust autonomous systems still requires advances in continual learning, multimodal reasoning, and reliable computer use. Our team agreed: it’s more realistic to focus on assistive agents with bounded scope and autonomy, capable of executing tasks under human supervision, than to chase the mirage of full autonomy.

Why Software Is AI’s Natural Beachhead

As discussed in our session, software development is uniquely suited for AI integration because outcomes are measurable and reversible. Code either compiles or fails, tests can verify correctness, and metrics quantify performance. This quantifiability — as Andrej noted during the discussion — explains why AI-assisted coding has advanced faster than creative or legal applications. Software is the perfect playground for measurable, iterative improvement. What other domains share these measurable and reversible qualities? Use cases where outputs can be deterministically validated, tested, and audited may represent the next wave of near‑term opportunities for applied AI.

Smaller Models, Bigger Leverage

We were struck by Karpathy’s emphasis on the potential of small, specialized models. Many tasks don’t require billion-parameter LLMs; a smaller model combined with retrieval and rules can achieve similar quality at a fraction of the cost. Our team highlighted how this approach can significantly reduce infrastructure costs and enhance performance—particularly in production environments where token usage scales fast and response speed directly shapes perceived product quality. We’ve been applying this principle by fine‑tuning smaller models to handle tasks once reserved for the largest ones. The key is data: start with a powerful model to generate high‑quality examples, curate real usage data, and iteratively refine the smaller model until it meets or surpasses an acceptable quality threshold.

The Value of Continuous Evaluation

Another recurring theme was the importance of evals as a strategic process, not just QA. Teams that continuously transform user data and incidents into automated evaluations will build faster, safer, and more reliable systems. Static, dataset‑driven evaluations remain invaluable—they serve as the foundational complement to unit tests before deploying an AI‑first application. At the same time, our team is heavily investing in live evaluations that continuously benchmark real‑world usage. These systems help us respond to unexpected usage patterns and serve as a rich source of new, high‑quality samples to refine and expand our static evaluation datasets over time.

Where the Debate Remains Open

We discussed several open questions:

• How small can a cognitive core realistically be before quality collapses?
• When does it make sense to fine-tune or build custom models instead of relying on well-prompted general ones?
• How do we design memory layers that combine persistent user context, transient session data, and structured retrieval systems?

Closing Thoughts

Karpathy’s conversation was a refreshing reminder that progress in AI is as much about engineering discipline as about model scale. As a team, we found inspiration in his pragmatism: simplify architectures, externalize memory, prioritize measurable outcomes, and view agents as tools for augmentation, not autonomy. The future of AI is exciting, and this kind of discussions remind us that we’re just starting this journey.

These reflections were compiled from an internal discussion by The Agile Monkeys team following Andrej Karpathy’s interview with Dwarkesh Patel (October 2025). The recording and full transcript are available at dwarkesh.com/p/andrej-karpathy.

Introduction

When OpenAI releases updates to its foundational models, engineering teams face a critical decision: when and how should we upgrade our production systems? While the GPT-4.1 model promises improvements in language understanding, inference speed, latency, and cost, switching models without proper evaluation can introduce regressions in business-critical tasks.

To address this challenge, we designed a multi-layer evaluation protocol that would give us confidence in our migration decision while minimizing risks to our production e-commerce application.

In our specific use case, we use LLMs to extract intent and business-specific features from user queries made in an e-commerce site search engine.

Our Evaluation Strategy

We structured our migration process around two complementary evaluation tracks:

What We Tested

Our evaluation included:

Prompt Versions:

• v25: Our current production prompt used with GPT-4o
• v26: A new prompt designed specifically for GPT-4.1, featuring significantly reduced input tokens by removing tasks that could be handled reliably with code instead of the LLM
• v27: A variation of v26 that reintroduces our domain-specific glossary to help the LLM better understand our e-commerce terminology

Models: GPT-4o and GPT-4.1

Evaluation Methods:

• Prompt-level: Raw model responses analyzed through a dedicated REST endpoint, with outputs compared against expected classification and translation results
• Application-level: Complete system testing using our golden dataset, measuring accuracy on a 0-1 scale by comparing returned product IDs against expected results

Results and Discussion

Conclusion

Our comprehensive evaluation demonstrated that GPT-4.1 with prompt v27 outperforms our current GPT-4o with v25 setup, both in isolated testing and real-world application scenarios:

• Application accuracy improved from ~80.6% (v25) to ~87.3% (v27) on our golden dataset
• GPT-4.1 achieved an impressive 91.72% pass rate on our evaluation dataset

This evaluation gave us the confidence to proceed with the migration, knowing we’re not just maintaining performance but actually improving it while benefiting from GPT-4.1’s enhanced speed and cost efficiency.

Experimental Section

Acknowledgments

Special thanks to the engineering and data teams at The Agile Monkeys for their meticulous work on evaluation design and implementation. Their rigorous approach made this confident migration possible.

1 Motivation

A shopper types “oversized off‑white tee with crew neck”.

The catalogue, however, says:

• Colour: stone — not off‑white
• Style notes: loose fit, boyfriend cut
• Image: crew neck visible, but not part of the product information.

Keyword search collapses; a single embedding model still misses half the picture.

The pragmatic question:

How far can we get by blending two complementary off‑the‑shelf encoders before investing in task‑specific training?

2 Experimental Set‑up

3 Fusion Strategies

import numpy as np
import faiss
import clip, torch, openai

# --- 1. encode ---------------------------------------------------------------
device = "cuda" if torch.cuda.is_available() else "cpu"

# Image embedding
clip_model, clip_preprocess = clip.load("ViT-bigG-14", device=device)
def encode_image(path):
    img = clip_preprocess(PIL.Image.open(path)).unsqueeze(0).to(device)
    with torch.no_grad():
        vec = clip_model.encode_image(img).squeeze().cpu().numpy()
    return vec / np.linalg.norm(vec)

# Text embedding (OpenAI)
def encode_text(text):
    resp = openai.embeddings.create(
        model="text-embedding-3-large",
        input=text,
    )
    vec = np.array(response.data[0].embedding, dtype=np.float32)
    return vec / np.linalg.norm(vec)

# --- 2. early fusion ---------------------------------------------------------

def concat_fusion(img_vec, txt_vec, alpha=1.0, beta=1.0):
    return np.concatenate([alpha * img_vec, beta * txt_vec]).astype("float32")
# Example
img_vec = encode_image("sku_123.jpg")
txt_fields = [name, description, product_type, primary_color, secondary_color]
txt_vec = encode_text(" ".join(txt_fields))
z = concat_fusion(img_vec, txt_vec)

# cosine because vectors are L2‑normalised
index = faiss.IndexFlatIP(z.size)
# add all catalogue vectors
index.add(np.stack([z, ...]))
# retrieve top‑50
D, I = index.search(query_vecs, k=50)

def weighted_sum(img_vec, txt_vec, lam=0.5):
    return (lam * img_vec + (1 - lam) * txt_vec).astype("float32")

# simple λ grid‑search on a held‑out set
lams = np.linspace(0.1, 0.9, 9)
best_lam, best_score = None, -np.inf

for lam in lams:
    fused = np.stack([weighted_sum(iv, tv, lam) for iv, tv in zip(img_vecs, txt_vecs)])
    # user‑provided evaluation routine
    score = evaluate_nDCG(fused) 
    if score > best_score:
        best_lam, best_score = lam, score
print(f"best λ = {best_lam:.2f}")

4 Results

Concatenation delivers the highest lift at the lowest implementation complexity.

5 Next Steps

• Domain-specific fine-tuning: Gather a modest set of (query, positive, negative) triplets from your production logs or a small annotation sprint. Apply LoRA adapters (or similar lightweight methods) to the CLIP image tower while freezing the text encoder. Even three epochs on ≈ 1–2 % of the catalogue often yield an extra single-digit nDCG lift, enough to justify the few GPU-hours required.
• Adaptive weighting: Replace the fixed α, β scaling factors with a two-layer MLP that predicts modality weights from the query text alone (e.g., longer colour-heavy queries lean toward the image slice). Train the MLP with pairwise ranking loss against your triplets. Early sandbox tests show +1–2 % nDCG over constant weights with negligible runtime cost.
• End-to-end dual-tower: Train the image and text encoders together from scratch (or from CLIP-like checkpoints) with a shared projection head so that their concatenation is directly optimised for your ranking loss. The result stays ANN-friendly while baking domain semantics into both towers. Expect the biggest payoff once you control tens of thousands of labelled pairs.
• Memory optimisations: At 3 840-D (CLIP + text-embeddings-3-large) you’re storing ~15 kB per item uncompressed. Orthogonal Product Quantisation or a 95 %-variance PCA cut reduces storage by an order of magnitude and slashes RAM/IO, typically at < 1 % quality loss. Combine with HNSW-PQ for sub-30 ms latency on million-scale catalogues without a GPU.

6 Conclusion

Early fusion by simple vector concatenation yields a +15 % retrieval boost in fashion e‑commerce with almost no engineering effort. Late‑fusion weighted sums help, but less so; supervised fusion and fine‑tuning remain promising avenues once the “trivial concatenation” gain is banked.

References

1. Sivic, J. & Zisserman, A. Video Google: A Text Retrieval Approach to Object Matching. ICCV 2003.
2. Lanckriet, G. et al. Learning the Kernel Matrix with Semidefinite Programming. JMLR 2004.
3. Gönen, M. & Alpaydın, E. Multiple Kernel Learning Algorithms. JMLR 2011.
4. Radford, A. et al. CLIP: Contrastive Language–Image Pre‑training. 2021.
5. OpenAI. text‑embedding‑3 models. 2024
6. Zhu, X., Huang, S.-W., Ding, H., Yang, J., Chen, K., Zhou, T., et al. (2024). Bringing Multimodality to Amazon Visual Search System. arXiv preprint arXiv:2412.13364.
7. Superlinked Research Team. (2024, February 21). Retrieval from Image and Text Modalities. VectorHub by Superlinked.