Agent Evals

You can now test your agents before shipping them, directly inside the agent editor.

Agent Evals lets you define test sets with real prompts, set expectations for what a good answer looks like, and run them against your agent on demand. No more manual spot-checking after every update.

What's new

Test sets & test cases

Create one or more test sets per agent. Each test set holds a collection of test cases - a prompt paired with the criteria the agent's response must meet. You can add cases manually, edit them inline in a table view, or import them in bulk via CSV (a sample template is available to download).

Three ways to grade a response

  • AI judge: Provide a reference answer or a rubric, and a model evaluates whether the agent's response passes. The judge returns pass, unsure, or fail.
  • Tool use check: Specify which tools the agent should call. Pick from the agent's connected tools and actions using a built-in tool picker.
  • Keyword check: Define words or phrases the response must mention or must not mention.

Choose your judge model
Each test set lets you pick which model grades the responses, independent of the model the agent itself uses.

Run and watch results live
Kick off a run from the Tests tab. Results stream in as cases complete. A cross-set timeline gives you a history of all past runs. Click any case in the results table to open a full detail view: the conversation, tool inputs and outputs, and the expected vs. actual answer side by side.

Run individual cases
You don't have to run an entire test set every time. Select one or more cases and run only those - useful when iterating on a specific prompt. Unselected cases are skipped and stay in the set.

Download results as CSV
Export any run's results as a CSV for analysis or sharing with stakeholders.

Safe tool execution

When running tests, tool calls that could have side effects (like writing data or sending messages) are executed in a safe mode by default, so you can test freely without worrying about unintended actions.

Note: Test runs count against your personal usage limits.

Where to find it

Open any agent in the editor and go to the Evals tab. The tab was previously visible but disabled, it's now fully available.

We'd love to hear how this fits into your workflow of building agents!