Built with Devin and Braintrust

We built a self-improving eval suite for our sales agent that grows with production. Every week, Devin, the coding agent from Cognition, reads real conversations on Rox from our production traces in Braintrust, turns them into evals, and opens a pull request that we review. Instead of hand-writing evals, we spend our time improving the skill that writes them.

Functionality in Rox has grown considerably in just the past few months. Our agent went from answering questions to reading dozens of data sources, editing CRM records, orchestrating outbound, and building decks. Evals we wrote by hand started to pass consistently even while bug reports kept coming. Here's an example:

A user asked for a follow-up from the last call. Only the earlier call had a transcript, so Rox quietly wrote the email from that one. Devin rebuilds the same situation with synthetic data, replays it until Rox makes the same mistake, and commits it as an eval.

Every conversation with Rox is traced in Braintrust, our observability platform, so we can see exactly what the agent read, which tools it called, and what it sent back. Each week, Devin pulls traces like this one, and together we judge whether Rox passed the bar. The core problem we tackled is turning that trace into a faithful offline eval.

From trace to eval

Our traces come from Rox's own sales team, who use Rox for their daily work, and live in Braintrust. A trace is what makes a faithful replay possible: it records the user's exact request and every record and tool our agent touched along the way. To turn one trace into an eval, Devin:

• Pulls the trace from Braintrust to see the user's request, the data it touched, and what our agent did with it.

• Writes synthetic data to emulate exactly what the request touched: the account, the meetings, the emails, all with the names swapped out.

• Launches a preview environment that contains a full copy of Rox with its own backend, public URLs, and an empty organization.

• Replays the user's message with our real agent and compares the outcome with the production trace in Braintrust to ensure the bug is faithfully reproduced.

• Adjusts the synthetic data and replays if the two don't match.

• Freezes the data and commits the eval once they do.

Devin runs this whole loop on its own. It works in its own environment, so it can stand up a preview environment, use Rox there the way a salesperson would, and open a pull request with the finished eval. The procedure lives in a single skill file that Devin follows. When something goes wrong, we fix the skill instead of patching evals by hand.

Preview environments replace our earlier approach, which replayed frozen snapshots through a mocked data layer. Mocks can hide bugs in the very code they replace, and that risk grows as Rox takes on more work by itself: editing records, building decks, drafting emails. Now every eval runs on a real copy of Rox that is isolated from production, holds only the data one trace needs, and loads identically every time.

When a request changes data, like sending an email or updating a record, code checks the result. Otherwise, an LLM judge grades the reply against a written criterion.

The suite is built to catch behavioral regressions: a tool path that stops working, or a supported request Rox no longer handles. It isn't a taste-based rating, so every failure should point to something we can fix. Agents aren't fully deterministic, though. Even with no code changes, about 1 in 8 evals passes some runs and fails others, so we read a failure before we trust it.

One of our real evals, trimmed for length. The request is copied word for word from production, and the seeded data leaves the newest meeting without a transcript. The weekly sweep caught this bug, and the eval passes now that the fix has shipped.

Grouping conversations into behaviors

Running this on every conversation would be wasteful, since most requests repeat the same few shapes. Ideally, two conversations share an eval when an eval built from one would faithfully replay the other. Checking that directly would mean building an eval for every conversation, so Devin approximates it with four labels that together define a behavior:

• Goal: what the user wants (recap a meeting, draft an email, prep for a call)

• Output: what comes back (chat answer, email draft, document, deck, website)

• Mutation: what it changes (nothing, create, edit, send)

• Target: whether the user named the object or left Rox to figure it out ("the call we just had")

Each week, Devin samples about 350 conversations from Braintrust and labels each one. Conversations with the same four labels form one behavior. Every behavior starts with at least one eval and gains more whenever production fails in a new way.

Coverage holds week to week

Labels only help if behaviors stay stable, so we tested whether one week's suite covers the next. Coverage is the share of a week's conversations whose behavior already had an eval. The suite from our first sweep covered 72% of conversations it had never seen, in each of the next two weeks:

Coverage is the share of a week’s conversations whose behavior (goal, output, mutation and target) already had at least one eval. Both weeks came after the first sweep, so the suite had never seen these conversations.

Before the first sweep, we had 142 hand-written evals. We labeled them the same way and measured them against the same two weeks:

One automated sweep covered twice as many behaviors as everything we had written by hand.

Devin improves the skill too

When the skill gets something wrong, Devin proposes a fix to the skill itself, and we review it like any other change. Grouping is the clearest example. The first version grouped conversations by their exact request, output, and state changes. Across 1,337 conversations, that produced 1,214 groups: nearly one per conversation, and barely better than no grouping at all. Devin noticed that new conversations rarely matched existing groups and proposed the four-label definition above. We approved it, the registry converged, and today it tracks 132 behaviors.

What the suite gives us

In practice, it changes four things:

• Bugs surface before anyone reports them. When Rox gets something wrong in production, it shows up as a failing eval in that week's pull request.

• Fixes have to prove themselves. A fix ships only once its eval goes from failing to reliably passing. We reverted three candidate fixes that didn't move their eval.

• Failures are mostly real. We reviewed 40 failing LLM-judge verdicts. The judge applied its criterion correctly in 36: 25 caught real bugs, and 11 exposed criteria that were too strict.

• Regressions get caught overnight. We run the full suite thre

  e times every night and investigate any eval that fails all three runs.

It also gives us a live map of how people actually use Rox:

In how we build agents at Rox, we wrote that agents don't come with a verification loop and that we would have to build one. Living Evals is one piece of it. We don't write the evals; we maintain the procedure Devin follows to write them, and our week is spent reviewing its pull requests, including its changes to the skill. Braintrust shows us what happened in production, Devin turns it into evals, and the suite evolves with it. Its coverage is measured on traffic it has never seen, and for the first time, our evals keep up with our agent.

The eval pipeline in this post was built and is run by Rox's engineering team, using Devin, the coding agent from Cognition, and Braintrust for production tracing and observability. Published in partnership with Cognition and Braintrust.