graph TD M["Mission: Produce publishable technical articles"] M --> C["Cost"] M --> O["Outcomes"] M --> L["Learning"] M --> R["Risk"] C --> C1["Model/tool spend per run"] O --> O1["Drafts generated"] O --> O2["Drafts approved for publication"] O --> O3["Time to publish"] O --> O4["Edit rate / reviewer effort"] L --> L1["Feedback captured"] L --> L2["Lessons learned written"] L --> L3["Repeated mistakes retired"] R --> R1["Messages/posts sent"] R --> R2["Actions awaiting approval"] R --> R3["Write operations by channel"]
A control plane isn’t your starting point. It’s what remains after one real agent has forced you to solve retrieval, permissions, review, action visibility, and outcome tracking.
Most teams building agents still start in the wrong place.
They sketch the platform first: agent registry, orchestration layer, eval dashboards, memory, governance, runtime, beautiful control plane. It feels strategic, but it hides the question that actually matters: can one agent do useful work against real company data, with a failure rate people will tolerate, and a review cost the team can actually live with?
We took a different path. We skipped the platform, shipped one real agent, and let that agent surface the stack we actually needed.
Something awkward happens fast when you do that. The missing pieces usually aren’t the ones the platform demos advertise.
In our case, one production-oriented internal content workflow forced a sharper realization. A platform that can tell you token spend is still incomplete if it can’t tell you whether the agent created any value, whether reviewer feedback improved the next run, or what the agent actually did in a user’s name.
That changed our view of what the second layer of an agent platform should be.
The platform-first trap is still a trap
The first useful agent immediately exposes realities that architecture diagrams politely ignore.
For us, the bar was concrete: produce publishable content on a weekly cadence. Something a human reviewer could actually ship, not just find “surprisingly good.”
That bar made the real blockers obvious. Source access was uneven, and retrieval quality varied enough between runs that we couldn’t predict whether a given draft would be usable. Feedback loops depended on manual effort that didn’t scale. Review, which we’d initially assumed was temporary scaffolding, turned out to be a permanent part of the system. And the hardest problem wasn’t generation at all — it was earning enough operational trust that people would actually use the output.
A control plane on top of those problems would have been premature. A very organized way to scale fragile behavior.
One real agent gives you a better roadmap than a month of platform speculation, because it forces contact with failure modes instead of abstractions.
Cost dashboards aren’t ROI
Once we started looking at agent platforms more closely, a pattern jumped out.
In the products we reviewed — including Salesforce Agentforce, Microsoft Copilot Studio, Notion Custom Agents, Dust, and CrewAI — cost tracking was common. Model spend is visible, scary, and easy to graph, so it makes sense as a starting metric.
The interesting question for a real agent, though, is whether it reduced work, increased throughput, or improved a decision enough to justify existing. Those questions require mission-specific metrics.
For a content workflow agent, the dashboard we actually wanted looked like this:
That’s a very different platform surface from “token usage over time.”
A platform that only tracks spend optimizes for cheaper wrong answers. Useful for finance, but nowhere near enough for operations. We increasingly think every serious agent needs a mission-specific ROI dashboard rather than a generic activity panel. Otherwise teams end up arguing about vibes, with a cost chart on the side pretending to be evidence.
Feedback can’t live in one person’s settings page
The second gap was how platforms handle improvement.
A lot of current tooling assumes a workflow where one person creates the agent, opens settings, rewrites the prompt, and tests again. That works for demos. It breaks down the moment a second person has an opinion.
In real work, feedback is distributed. One reviewer catches tone problems, another flags a missing source, and a third keeps noticing the same factual error the agent was already corrected for. If all of that learning stays trapped in the creator’s prompt window, the team isn’t improving the agent — they’re generating a queue of editorial cleanup.
We hit that directly. Written feedback wasn’t reliably changing later behavior, so the agent could repeat the same mistake with fresh confidence. One of the least charming traits an agent can have.
Our response was to move toward a simpler, more operational loop: let reviewers give feedback in the flow of work, let the agent produce a better revision from that feedback, and when corrections repeat, turn them into durable lessons learned that persist across runs.
Even our first implementation made the real requirement clear. Improvement has to work across the whole team and compound over time, because individual prompt tweaks just don’t accumulate. The early version also surfaced its own limitations quickly — feedback handling only worked on newly generated pieces, which is exactly the kind of boundary you only notice once the workflow is real.
That’s why we now treat feedback capture as platform-level work.
Governance starts with action visibility
The third lesson was even more operational.
During testing of a Slack-connected workflow, we noticed messages had been sent without the user realizing they’d already gone out. In an internal setting that’s merely annoying, but in a customer-facing workflow it’s the kind of thing that ends pilot programs.
This is where a lot of “agent governance” talk becomes too abstract. Before you design exotic policy engines, there’s a simpler question worth answering: can the user see exactly what the agent did, where it did it, and whether it acted with explicit approval? Without that visibility, you have permissions plus optimism — which isn’t the same thing as governance.
The minimum useful layer is boring in the best possible way: a clear action log, explicit approval points for risky writes, visible traceability for messages sent on a user’s behalf, and easy ways to reduce permissions when trust hasn’t been earned yet. If using the agent makes people want to revoke write access, that’s worth treating as product feedback rather than user hesitation.
We now think action visibility is one of the first platform primitives worth building, especially for anything that writes into shared systems.
What one real agent taught us to build next
After all of this, the platform shape that feels justified is much smaller and much sharper than the original control-plane fantasy.
We’d start by making it clear what the agent can read and which missing source would make it obviously wrong. Then we’d add mission-specific outcome tracking, because measuring useful work is the only way to get past “it seems good” as an evaluation strategy. From there, the priorities are a durable review loop where feedback actually improves future runs rather than evaporating after one correction, and action logs with approval controls for anything writing into Slack, docs, tickets, or external systems. Only after those pieces were earning their keep would we invest in broader abstractions like registry, orchestration, and generalized runtime.
The sequence looks like this:
graph TD A["Pick one real agent"] --> B["Define operational success metric"] B --> C["Connect the sources it actually needs"] C --> D["Instrument outcomes and review effort"] D --> E["Make feedback durable"] E --> F["Log risky actions and approvals"] F --> G["Abstract common pieces into a platform"]
Less glamorous than “let’s build the agent platform.” Also much harder to fake.
Key takeaway
If you’re building agents, don’t stop at source connectors and spend dashboards. One real workflow will quickly show you what the platform is actually missing: outcome tracking, durable team feedback, and action visibility. Build those before you fall in love with orchestration diagrams.