ROLE
Lead Product Designer
Lead Researcher
INDUSTRY / PRODUCT
Enterprise B2B AI Platform (NDA)
USERS IMPACTED
200+ developers
TIMELINE
2025, 4 months
OVERVIEW
Enterprise teams were building autonomous AI agents for high-stakes systems like HR and finance but had no real place to test them. Developers built each agent in whatever coding tools they preferred, deployed it into chat tools like Teams or Slack, and tested it by talking to it like a user. When it broke, debugging meant bouncing between tools to reconstruct what the agent was even thinking, about 18 minutes per error, and rarely just one. I owned the testing-and-debugging experience, and designed one place where developers could watch an agent's reasoning unfold, see exactly where it broke, and fix it without leaving the conversation.
Due to NDA, product names and visuals are abstracted. Contact me to view the presentation deck.
PROBLEM
An agent doesn't fail in one place. To answer a single request it makes a chain of decisions like pick a tool, call it, read the result, decide the next step, and all of it happens out of sight.
So when the final answer was wrong, there was no way to tell where in that chain it went wrong. Was the tool never called? Called with the wrong inputs? Called fine, but the agent misread what came back?
Each of those is a different fix, and nothing on screen told developers which one they were looking at. So they did the only thing they could: opened every log and tool they had and rebuilt the run by hand.
RESEARCH
I ran two-day sessions with five developers building agents for real clients, sat in on live sales demos, and interviewed the sales engineers who fed those developers their requirements. Mostly I watched.
Testing one app took an hour or more. And every time it broke, the same loop ran: leave the test, open a different tool to read the error, guess at a fix, redeploy, try again. I timed it — roughly 18 minutes per error, and there was rarely just one. The pattern showed up everywhere I looked. In live client demos, agents failed mid-conversation even though the sales engineers scripted carefully around the unreliability. Requirements passed through so many hands — analyst to sales to developer — that apps took months to ship, by which point the client had often moved on. And when I surveyed the developers on how confident they felt an agent would work on the first deploy, the number was low. The work got done, but slowly, anxiously, and at real cost to the business.
INSIGHT
Every instinct in design says simplify and abstract the mess, show the clean surface. Watching these developers, the opposite was true.
Debugging wasn't slow because there was too much on screen; it was slow because the one thing they needed which was the agent's actual reasoning, and the exact step it broke lived nowhere they could see it. They tested by talking to the agent, so the answer had to live inside that conversation.
That became our north star: when I'm building a mission-critical agent, I want to see the full chain of reasoning and where it failed, so I can trust what I'm putting into production. From there, my developer, my PM and I pinned every decision against that one sentence.
DECISION
The cleaner, faster option would have shipped sooner and looked simpler. I told my PM it was the wrong call. That was the argument I had to win. I made the case to my PM that it solved for our build time, not the developer's trust. The timeline meant more screens and harder calls about what mattered most on each one. We chose it anyway: transparency over efficiency.
SOLUTION
Developers test by chatting with the agent, so its reasoning now streams right alongside the conversation — a high-level view first, then a path down to the exact step that broke. A few deliberate choices made it readable at a glance and recoverable in one move. (The how — and the timeline component I built from scratch — is the part I love walking through in person.)
IMPACT
We retested in the same QA environment with the same five developers, this time with a short manual to orient them first. The loop that used to eat 18 minutes an error now took 8 measured by task analysis. A follow-up survey caught a 36% jump in their confidence deploying on the first try.
The slow, anxious build-test-break cycle had become something they could move through fast enough to put agents in front of clients.
REFLECTION
The instinct as a designer is always to simplify, hide the mess, and smooth the surface. This project taught me that when an agent is making decisions about someone's paycheck or their health coverage, the simple choice and the trustworthy choice aren't the same one and my job was to argue for the harder one, even when it cost us build time. Designing for five specific developers I'd watched struggle, instead of for a tidy screen, is the instinct I'm carrying into whatever I build next.
The messy middle, the dead-ends, the iterations it took to get here, lives in the deck.Contact mefor the walkthrough.