PII redaction has an awkward property: it has to happen before almost everything else.
Before an agent retrieves a document, calls a tool, or generates its first word, the user’s message must be checked for names, email addresses, account numbers, and other sensitive data. If that check is slow, the user waits before the useful work even begins. If it sends the original text elsewhere, the privacy boundary is already compromised.
For us, solving those two problems led to two very different breakthroughs. The first used an open-weight model on a GPU to make redaction private and much faster. The second began when we priced that architecture and realized we could not justify running it.
A privacy layer that still crossed the boundary
Our initial PII provider used GPT-5.4 Nano with low reasoning. Functionally, it worked well. The workflow and our downstream analytics received redacted text instead of the original message.
That was useful containment, but the original PII still had to leave our infrastructure: AIFindr (The Agile Monkeys’ platform for building reliable enterprise AI agents) had to send it to a third-party model for redaction.
A major Latin American bank made the requirement explicit: original PII should not leave our infrastructure. At the same time, the remote redaction call was adding roughly 0.8-1.2 seconds before the agent workflow could start. The client felt that delay directly: it degraded the experience before the first useful word could even appear.
Why not just use regular expressions? They work well for predictable formats, but PII is often ambiguous and context-dependent. Formats also vary across countries and user input, so adding more rules quickly becomes brittle and risks creating false positives.
So we decided to investigate open-weight models that could run fully inside our infrastructure. To compare them responsibly, we first needed a strong evaluation dataset: standard PII examples plus difficult cases with ambiguous values, malformed formats, and false-positive traps. It had to tell us whether sensitive data remained visible, whether harmless content disappeared, and how long redaction took. Quality came before speed or cost.
Breakthrough one: an open-weight model on our GPU
We created AIFindr Privacy as a separate internal service that the backend could call for redaction. The service ran the open-weight model on a GPU inside our infrastructure.
The first engine used OpenAI’s open-weight openai/privacy-filter. It is a purpose-built classifier, not a generative LLM. We combined its detections with deterministic structural safeguards and span-resolution logic, then exposed redacted text and non-sensitive detection telemetry.
On an NVIDIA T4 inside our network, the result felt like a genuine step change.
The result was a 4.9x speed-up in median PII-stage latency, while both approaches reached a 96.6% evaluation pass rate. Most importantly, it was a win on both fronts: raw PII stayed inside our infrastructure, and the workflow began much sooner without giving up quality.
There were serving lessons hiding inside the win. A GPU does not automatically make concurrent HTTP requests parallel. We had to instrument queue time, configure microbatching explicitly, add backpressure, and distinguish model time from end-to-end latency.
Conclusion: we had achieved both goals. Redaction was much faster, and the original message stayed entirely inside our infrastructure. Everything looked solved.
Or so we thought.
Then we calculated the bill
The original deployment design kept one g4dn.xlarge GPU instance active in development, one in staging, and two in production for availability and deployments.
That was approximately EUR 1,343.76 per month, or EUR 16,125 per year, in compute for AIFindr Privacy alone. That bill covered only PII redaction, not the rest of AIFindr’s infrastructure.
We had an approach that solved privacy and latency exceptionally well, but its cost made the architecture economically unsustainable.
This was not a model failure. It was a system-design failure. We could have chosen a slightly cheaper GPU and reduced the bill a little, but not enough: the architecture would still have been far too expensive.
So we went back to the drawing board.
We gave an agent the goal. Then we let the loop run
The new objective was stricter than “find cheaper hardware”:
Preserve privacy, evaluation quality, and a clearly faster experience, while making the service economically sensible to operate.
We decomposed the GPU prototype into variables we could test independently: model artifact, numeric representation, runtime, CPU architecture, task size, batching, traffic shape, and deployment topology.
We used coding agents to make that breadth practical. We gave an agent a clear goal and ran it in an autonomous loop with scoped access to the repository and AWS. For each hypothesis, it researched the official artifacts, prepared a temporary deployment, ran evaluations and benchmarks, saved the evidence, removed the temporary infrastructure, and returned to the loop with the next hypothesis. It stopped when an approach met the goal.
That loop began pointing toward an answer we had barely considered. What if the GPU was not needed at all?
Breakthrough two: the surprising answer was CPU
It sounded wrong at first. Running the model on CPU felt like the route back to slow inference. The measurements told a different story.
That was possible because we were serving a small, task-specific classifier rather than a generative LLM, making an optimized CPU runtime a realistic option.
OpenAI had already published several ONNX variants of the same model. We tested the official artifacts without changing the weights ourselves. The smaller Q4 variants introduced new quality regressions, so we rejected them. The official INT8 artifact, model_quantized, passed.
INT8 preserved every baseline redaction we checked and reduced runtime memory to about 1.6 GiB, which meant a 4 GiB task was enough.
ARM64 changed the performance result too. On direct inference, a 2-vCPU ARM task reached a median of 115 ms, compared with 144 ms for the tested T4 path. Quality held, and AIFindr Privacy even outperformed GPT-5.4 Nano on the harder evaluation set (76% vs. 69% normalized pass rate, with half as many false negatives).
Could a CPU service really handle production traffic?
We used several months of AIFindr production analytics to reconstruct real peaks across all projects, then load-tested the exact topology we intended to deploy:
Two ARM64 Fargate tasks, each with 1 vCPU and 4 GiB, behind an internal load balancer.
The answer was yes. The pair comfortably handled loads above the peaks we had observed while keeping latency within our target and producing no HTTP errors. CPU was not merely viable in a benchmark; it was enough for our real production demand. Two tasks also provided availability during failures and deployments, with a clear horizontal scaling path as traffic grows.
From EUR 1,344 to EUR 119 a month
The production pair costs approximately EUR 59.55 per month. DEV and STG add approximately EUR 29.77 each, bringing the complete three-environment topology to EUR 119.09 per month. Compared with the original EUR 1,343.76 GPU design, that is a 91.1 percent reduction.
There was still one inconvenient fact to acknowledge: at today’s AIFindr traffic, GPT-5.4 Nano remains cheaper on direct inference spend. Being honest about that matters. We had dramatically reduced the cost of private inference, but had not yet crossed the pure-cost line against a usage-priced API.
The curves cross at roughly 4.1 times our July 2026 redaction baseline. That month’s projected traffic was approximately 65 percent above June, making the crossing point a plausible horizon rather than a distant thought.
But cost was never the only ledger. Even before that crossing point, raw PII stays inside our infrastructure and redaction is much faster. Those were the two outcomes our clients actually asked for. Lower cost at scale would simply become the third win.
What we actually gained
AIFindr Privacy ended with three concrete results: raw PII stays inside our infrastructure, redaction is substantially faster, and the selected design is 91.1 percent cheaper than the original GPU topology.
The model at its center is a classifier, not an LLM. That is part of the lesson: small, task-specific open-weight models can remove latency, cost, and data boundary problems from narrow parts of a larger agent workflow.
More importantly, this is no longer an experiment. It is an open-weight model running in production and a reusable way to investigate the next improvement: set the goal, give an agent bounded autonomy, evaluate every hypothesis, and follow the evidence.