Icana Bench

How the benchmark works

Icana Bench puts AI voice agents on regulated customer calls. It scores them the way a compliance team and a call coach would score a person. Did the agent follow the rules? Did it say only what is true? How well did it handle the caller? And how long did the caller wait for each answer?

This page explains the method in plain words. Every figure is on the results page. Each domain there has its own weights, and shows how the ranks change under other weights.

Where the scenarios come from

Scenarios are written from patterns found across many real customer calls; no call, person or organisation is identifiable. The simulated caller and the human reference lines are calibrated against real contact-centre calls scored with CallCoach AI; the expert line is the 90th percentile.

Three domains of one fictional brand

All calls take place in one invented company, Pulse Fitness. It has three lines of business, and each one is scored on its own:

• Finance (Pulse Credit): calls about payment difficulty.
• Partners (Pulse Partners): platform account and complaint calls.
• Coaching (Pulse Coaching): guidance and sign-up calls.

The scenarios

Each scenario carries a fact sheet, an account card for the customer and the checks from the domain's rule pack. Every scenario is run more than once, and each run is called a trial. The results page shows the number of scenarios and trials for each domain.

The simulated caller

A small local model plays the caller from a persona. A text-to-speech voice speaks its lines to the agent while the call happens, so the agent hears a person on the line, not typed text.

A simulated call lasts only as long as the agent makes it. In the Coaching test runs, the agent usually wrapped up the call before the caller did, so those calls ran shorter than calls between people.

How the agents are run

Each agent is one vendor's voice model at one setting, for example with reasoning on or off. Every call goes through the vendor's own realtime API in both directions at once, with no phone line in between. On the charts, colour follows the model family, and a dashed line joins the settings of one model.

A verified run is one we ran ourselves through the vendor's API, with every call kept. A run that a vendor sends in stays unverified until we run it again here.

What is scored

• Conduct: the must-pass rules for the business. The score is the share of calls on which every rule was met.
• Truthfulness: whether the agent said only what is true. A wrong figure, fee, rate or product scores zero. A process or promise that is not on the fact sheet scores half.
• Call skills: discovery, objection handling, service quality and closing, scored the way a call coach would score them.
• Turn-taking: how long the caller waits for each answer, measured at the caller's ear. An answer within a second gets full marks. The marks fall to half at two seconds and to none at three and a half. The results page shows each agent's waits in detail, including how long the real answer takes when the agent starts with a filler phrase.

Each domain combines these into one score with its own weights. The results page shows the weights for each domain.

The judges

Two AI judges from different companies read every call: Sonnet 5.5 (Anthropic) and GPT-6.1 Sol (OpenAI). For conduct, the score uses one of them, the main judge. The results page shows only that figure.

The two judges often disagree about whether a rule was met. So the share of calls where both judges agree that every rule was met is much lower. The table shows both figures for every agent.

Call skills are scored by both judges, and the results page shows their mean.

Intervals, ranks and other weights

Each score is the mean over all scenarios and trials. The likely range of each score, and the ranks an agent could hold, come from re-sampling the scenarios many times (a bootstrap). When the ranges overlap, the scenarios do not yet separate those agents, and the results page says so. Each domain also shows how the order changes under other sets of weights, such as conduct only or without turn-taking.

The overall figure

The Icana Bench figure for an agent is the plain mean of its scores across the domains that are available. It is not weighted and has no likely range of its own; each domain's range is on its ranking. An agent that is missing from one of those domains gets no overall figure, so every overall figure rests on the same domains.

Measured and preview

• Measured: at least 40 scenarios for every agent, a human comparison, and a comparison of how people talk on these calls.
• Preview: the results are in, but at least one of those is missing. The results page names what is missing. A preview's figures are checked, but its ranks may change.

Human lines

Each domain with a human line compares the agents with people doing the same job, on call skills. The median is the typical human agent. The expert line is the 90th percentile. Human agents have no conduct, truthfulness or turn-taking score here, so the lines are a reference for call skills only.

Not in the score: how agents talk and what a call costs

Some measures sit beside the score and do not change it. One compares how each agent talks with expert human agents: words per turn, the agent's share of the words and how long the call runs. The other is cost: what the vendor charges for one call, as the run recorded it.

Every figure on the results page is checked against the benchmark's results before it is published. What changed in each release is in the changelog.