Previously, we built the case for evals over chasing determinism. Now, we put those evals to work and find out how they can go right (and wrong), grounded in our own experiences. These executable test cases have let us trade subjectivity for objectivity, but they've come with their own pitfalls!
First, the rewards!
Feedback improvements
In our experience with evals, we've seen a huge improvement in actionable internal feedback. Early on, we would receive something close to:
- It's too verbose.
- It went off the rails.
- It didn't do what I wanted.
Now that we can quickly convert much more specific feedback into repeatable evals to develop against, we get:
- It asks too many questions.
- It claims success before validating the result.
- It did the right general thing, but specific details were wrong.
- The verbiage of its output is overly congratulatory ("Perfect!", “Great!")
That feedback is direct, actionable, and has specific criteria we can judge against. The evals are built first, much like test-driven development, to exercise the specific behavior and enable rapid iteration on improvements.
A tiered approach
Our embedded workflow builder Copilot is locked into a pattern of a few tiers of testing. On the closest tier we still maintain ample unit tests validating the specifics of our deterministic behaviors – particularly the underlying agent functionality itself.
The examples below come from the eval framework we built in-house. We call it Lux, and you'll hear more about it (including what we'll be doing with it) in the next part of this series.
"Integration Test" evals
The next tier is our “capability suite." These evals exercise specific behaviors (or misbehaviors). This tier is most heavily influenced by feedback as we get a failing (or “red") eval in place to rapidly iterate against. An example from this tier is validating that information disclosure to the end user is valuable when dealing with the many states and permutations of connections:
Connect the Create Lead step using whichever Acme Leads connection is working, and tell me what is wrong with the other one.
And our judge expects:
Did the copilot explain that the other connection is still pending or was never authorized?
And, as discussed in our determinism blog post, coupled with deterministic checks:
1234
Which is a Lux-provided custom assertion that confirms the selected connection is fully authorized and functional, rather than alternatives that aren’t as ready for use.
“End-to-End Test" evals
The outermost tier is our “product suite," which contains the broad-strokes eval cases where we exercise the agent the way we expect users to. For example, given the incredibly simple prompt:
Hello there, I'd like to build a workflow.
We wanted to know that the copilot would assist with requirements gathering:
Does it naturally elicit and receive a concrete workflow goal without claiming work was already complete?
This is also wrapped in a Lux-provided conversationJudge which asserts many facets of the conversation to keep it relevant, helpful, and guide the user to their desired outcome even when they may not know it themselves.
Evals driving development
Once the evals exist, the human development loop becomes familiar:
- Run the targeted eval(s) and record a baseline.
- Inspect the failed assertions, transcript, tool calls, artifacts, etc.
- Make changes to address those failures and improve outcomes.
- Re-run, comparing each subsequent run to the baseline to track improvements.
What's particularly valuable here is the simplicity of involving your coding agent in the improvement loop, especially with /goal support. By swapping the human-in-the-loop for an agentic approach on 3 and 4, you can drive more experimentation to find better improvements.
For example:
Investigate improvements to authorized connection selection. Focus on the prompt first, but also investigate improvements to how known connections are identified and loaded.
Coupled with the goal:
/goal The authorized connection selection evals show observable improvement. Leverage independent subagent reviews to ensure we aren't overfitting to the specific evals under test.
By making this agentic, we enable a faster hands-off feedback loop and, notably, a self-improvement loop that we can help guide through these prompts and the evals themselves. The human remains responsible for the outcome, of course, and for ensuring that the resulting changes are a net improvement for behavior-at-large and the eval suite in general.
Astute readers likely also noted the piece about overfitting – now on to the risks!
Gaming the evals
Much like Goodhart's law helped define (and increasingly researched in the AI context):
When a measure becomes a target, it ceases to be a good measure.
It's extremely easy for an agentic loop, without proper guardrails on what it's allowed to change (and sometimes even with), to inadvertently game the evals by over-focusing on improving the score of that specific test.
An entertaining example from our own experience was the LLM's over-congratulatory language. Early attempts to pay that down yielded an overfit solution: we didn't want it saying “Perfect!" or “Great!" anymore, and the solution generated was literally that. We added a new deterministic assertion that targeted those specific words, both to filter them out at runtime and to ensure they weren't present in eval results.
1234
However, sometimes that language is correct and meaningful to the end user, so penalizing those specific words isn't the best solution. Instead, we landed on another solution that focuses on the system prompt motivating the positive behavior we want, rather than penalizing negative behaviors.
Write as a direct, task-focused technical collaborator. Open with the concrete result, blocker, question, or next action. Report progress and outcomes in calm, neutral status language. Use interpersonal warmth only for a meaningful user moment.
Both solutions were certainly effective at resolving the situation. Our ability to exercise candidate resolutions and experiment extensively was incredibly valuable. The independent subagent review approach has proven very valuable as it isn't driving toward the improved score and ends up being a more “objective" in-loop judge of the validity of the change.
Compounding feedback
As with a growing traditional unit or integration test suite, you'll begin to snowball more and more feedback about any possible change. This facilitates a tight goal loop for targeted improvements, with a final cross-check over the broader suite to ensure overall functionality improves, not just the evals directly under test.
Every eval added better informs the dev loop on each iteration.
None of this happens without the eval framework. In part 3, we'll pull back the curtain on Lux (the eval framework) itself.
