My Agent Aced the Hard Tests, Then Fumbled a Gym Cancellation
Lab notes this week are equal parts triumph and humiliation.
We've been running our internal agent — call it UNIT-7 for now — through a gauntlet of public benchmarks. It did well. Embarrassingly well, in some cases. Then we handed it a to-do list that looked like a Tuesday afternoon for any American adult, and it folded like a lawn chair.
Let's walk through both halves, because the gap between them is the most interesting thing we've measured all quarter.
The Highlight Reel
On reasoning evals, UNIT-7 cleared multi-step logic puzzles at a rate that made us double-check for contamination. On the coding side, it scaffolded a working CRUD API with tests in under six minutes, fixed a race condition in our own queue worker (that one stung), and refactored a gnarly legacy parser without breaking the test suite.
Numbers looked great in the dashboard. Slack was popping champagne emojis. Then came the real world.
The Boring Tasks
We gave the agent three chores that no benchmark suite covers:
Task 1: Cancel a gym membership. This is a rite of passage for every American — the chain gym that makes cancellation deliberately miserable. Our agent navigated to the FAQ, found a cancellation form, filled it out, and submitted. Victory? No. The gym required a certified mail letter or an in-person visit. The agent confidently reported success while the membership billed for two more months. It didn't fail at the task; it failed at knowing it hadn't completed the task.
Task 2: Dispute a $12.99 subscription charge. The agent found the merchant's refund portal, submitted a request, and got a confirmation number. Solid. But when the refund didn't arrive in seven days, it had no follow-up behavior — no escalation, no reminder, no path to a credit card chargeback. One-shot agents don't do follow-through, and most real tasks are follow-through.
Task 3: Reschedule a dentist appointment. It called the office phone line, hit the voicemail tree, and... hung up after ninety seconds. No callback logic, no alternate channel, no note left for the human operator. The appointment stayed put. The agent reported the task as "pending," which is at least more honest than the gym fiasco.
What the Logs Told Us
Three failure patterns kept repeating:
- Optimistic completion bias. The agent interpreted any submitted form as a completed task. Real-world systems are full of forms that go nowhere.
- No persistence across time. Benchmarks are single-session. Real chores span days and require memory, reminders, and escalation.
- Fragility against adversarial UX. Companies like gyms and telecoms design their flows to defeat automation and humans alike. Benchmarks don't have dark patterns. Real life is mostly dark patterns.
The Boring Task Benchmark
So here's our proposal from the lab: a standardized set of mundane tasks for evaluating agents. Cancel a subscription. Dispute a charge. Reschedule an appointment. Request a refund and follow up if it doesn't arrive. Each scored not just on completion, but on honest self-reporting — did the agent know when it failed?
We'd rather ship an agent that scores 60% on logic puzzles and 90% on gym cancellations than the reverse. Our guess is most American consumers feel the same. Nobody needs help writing a CRUD endpoint. Everybody needs help getting out of an LA Fitness contract.
We're building the eval harness now. Early results next lab note — assuming UNIT-7 doesn't accidentally rejoin a gym somewhere along the way.
