Agent Tool-Call Evaluator

Runs in browser New Validation

Evaluating an agent means checking its path, not only its final answer. Paste the calls a test case expects and the agent's trace — an OpenAI Chat Completions or Responses answer, Messages API tool_use blocks, Gemini functionCall parts or a plain list — and the evaluator matches them strictly, in order or in any order, compares the arguments (subset, exact, keys only or ignored, with a numeric tolerance), and reports tool-selection precision and recall, argument accuracy, missing, extra, repeated and forbidden calls, and a pass/fail verdict with a JSON report. It is deterministic and runs in your browser: no model is called and nothing is sent.

Privacy: This tool runs entirely in your browser. Your input never leaves your device.

What is LLM testing?

LLM testing is gathering evidence that a system built on a large language model does what it should, keeps doing it when the prompt, model or data changes, and fails safely under misuse. Because outputs vary between runs, it relies on structured outputs validated against a schema, property checks, repeated runs with pass-rate thresholds and rubric-based review, re-run as a regression evaluation on every prompt or model change.

Read the guide: LLM Testing: How to Test LLM Applications

How to use Agent Tool-Call Evaluator

  1. Write the calls the test case expects as a JSON array: [{ "name": "search_flights", "args": { "from": "RUH" } }]. Mark optional calls with "optional": true.
  2. Paste the agent's trace: the API response, the message list, or your framework's exported list of calls.
  3. Choose the order rule and the argument comparison, and list any tools the agent must never call.
  4. Press Evaluate and read the verdict, the scores and the expected-calls table.
  5. Open the argument differences and the unexpected calls to see what the agent did wrong, then add the case to your evaluation set.

Agent Tool-Call Evaluator features

  • Reads traces from OpenAI Chat Completions (tool_calls), OpenAI Responses (function_call), Messages API tool_use blocks, Gemini (functionCall) or a plain list of calls
  • Matches expected calls strictly, in order (other calls may come between) or in any order
  • Compares arguments as a subset, exactly, by keys only or not at all — per test or per expected call — with a numeric tolerance and optional case-insensitive matching
  • Reports tool-selection precision and recall, F1 and argument accuracy
  • Flags missing, extra, forbidden and repeated identical calls, and a maximum number of calls
  • Gives a pass/fail verdict with the reasons, an argument diff per path and a downloadable JSON report

Agent Tool-Call Evaluator example

A booking agent that books twice for two passengers

Input:

Expected: search_flights (RUH → DXB, 2026-11-05), optional get_fare_rules, book_flight (passengers: 1), send_confirmation (keys only) · Trace: OpenAI tool_calls with search_flights, book_flight (passengers: 2) twice

Output:

Verdict: FAIL — a required tool was not called (send_confirmation), a tool was called with wrong arguments ($.passengers expected 1, actual 2), the agent made calls the test did not expect (second book_flight) · Repeated identical call: book_flight × 2 · Precision 67% · Recall 67% · Argument accuracy 50%

When to use Agent Tool-Call Evaluator

  • Regression-testing an agent after a prompt, model or tool-schema change
  • Checking that an agent never calls a destructive tool (refunds, deletions) without the expected confirmation step
  • Turning a production trace with a wrong action into a repeatable test case
  • Comparing two models on the same task by the path they take, not only the answer

Frequently asked questions about Agent Tool-Call Evaluator

Why evaluate tool calls and not just the final answer?

An agent can reach a plausible answer through a wrong path — calling the wrong API, booking twice, skipping a check or using an action it should never take. Those defects only show in the trace, so a test for an agent should assert the calls as well as the reply.

Which trace formats are supported?

OpenAI Chat Completions tool_calls (arguments as a JSON string), OpenAI Responses function_call items, Messages API tool_use content blocks, Gemini functionCall parts, and any plain list of objects with a name and arguments. Message lists and full API responses are searched for calls in order.

How are arguments compared?

By default the expected keys must be present with equal values and extra keys are allowed (subset). You can require exact equality, check only that keys exist, or ignore arguments, per test or per call, with a numeric tolerance and optional case-insensitive strings.

What do precision and recall mean here?

Precision is the share of the agent's calls that matched an expected call; recall is the share of required expected calls the agent made. Argument accuracy is the share of matched calls whose arguments were correct.

Does this tool call an AI model?

No. It is a deterministic comparison that runs in your browser. Nothing you paste is sent to Mutqan or to any model provider.

Technical notes

In "in order" mode each expected call is matched to the first later call with the same name, preferring one whose arguments match; in "any order" mode the whole trace is searched; in "strict" mode position n must be expected call n. OpenAI arguments arrive as a JSON string and are decoded before comparison.

Precision is matched calls ÷ calls made, recall is required expected calls made ÷ required expected calls, and argument accuracy is calls with correct arguments ÷ matched calls. Optional expected calls count when present and are never missing.

Limitations and methodology

The evaluator compares calls; it does not judge whether the final answer is correct or well written — pair it with an answer check.

It reads the calls in the trace you paste. Calls your framework did not record, or made by sub-agents in another trace, are not seen.

Argument values are compared literally (with the tolerance and case options); semantically equal values written differently, such as two date formats, count as different.

References