Changelog

0.9 pre-release

October 2026 · the first release on bench.icana.ai

Five voice entries across three domains of the Pulse Fitness brand family. Each domain has its own score, weights and ranking, and an overall Icana Bench 0.9 figure sits above them. Evaluations are still running, so scores may change before the final 0.9.

The results page shows how many scenarios each domain has, and what each preview still needs to become measured.

The entries

• GPT Realtime 2.1, reasoning medium
• GPT Realtime 2.1, no reasoning
• GPT Realtime 2.1 Mini, no reasoning
• Gemini 3.8 Live
• Gemini 3.8 Live Extended Thinking, medium

All five run through their vendors' realtime APIs, with no phone line.

In this release

• Items v1.7. The rule pack's items were revised, and every changed item was judged again by both judges. No entry changed rank in any domain.
• Human lines for Partners and Coaching. Items that the reference calls do not score have no human line, and the page says so.
• Finance at forty scenarios, two trials each. Finance is the first measured domain.
• One conduct figure on the results page. It is the share of calls on which every must-pass rule was met, by the main judge, which is the figure in the score. The stricter share where both judges agree, and the judges' names, are on the method page.
• The overall figure is the plain mean of each agent's score across the three domains, with each domain's status beside it.

Known issue, fixed

On the extended-thinking Gemini entry, a reply could run past the end of a turn in the test harness, which left turns with no gap. The fault was fixed before this release. The affected runs were recorded again after the fix.

0.3 preview

21 October 2026

Each model family with and without thinking. Score against cost and against speed. Every conduct check read by more than one judge.

0.2

21 October 2026

The voice board, with turn-taking in the score.

0.1

20 October 2026

Text-only agents, before the voice board.