Methodology

Audio A/B Test Methodology

By IABT Team Published

Audio A/B Test Methodology

The first tests on IABT compare AI voice systems — text-to-speech (TTS) and voice-conversion models. This page explains how a voice comparison is built and judged.

What’s being compared

Each comparison pair holds the same script rendered by two different voice systems. Because the words are identical, the only thing you’re judging is how it sounds — naturalness, clarity, expressiveness, and the absence of artifacts (robotic timbre, clipping, mispronunciation).

System names are replaced with neutral labels (A and B) at the source, so a model’s reputation can’t sway the verdict. The mapping is revealed to you only after you submit.

How a test is built

  1. An experiment owner designs the comparison with IABT’s test builder — choosing the scripts, the systems to compare, and the rubric.
  2. The candidate clips are scored offline with objective metrics (intelligibility, speaker similarity, signal quality) to catch broken outputs before humans ever hear them.
  3. The workflow is deployed to IABT, which presents the pairs to the verified crowd and collects preferences.

How you judge

For each pair:

  • Listen to A and B in full — replay freely.
  • Pick the one that sounds better overall.
  • Optionally rate your confidence and flag what was wrong with the loser (e.g., “metallic,” “mispronounced a word”).

Position is randomized per pair, so “A” is not always the same system and being first confers no advantage.

From votes to a ranking

Individual pairwise choices are aggregated into a ranking: which system is preferred, over which, and by how much. Because every vote comes from a verified unique human, the counts aren’t inflated by bots or repeat voters — so a 60/40 split means 60/40 among real people, not among scripts.

Aggregate results (never individual identities) can be fed back to the team that built the voice systems, closing the loop between “we think this model is better” and “real listeners agree.”

What this is not

  • It is not a measure of factual accuracy — only of perceived quality and preference.
  • It is not a single objective score — human preference is the ground truth here, with objective metrics used only as a pre-filter.

Ready to lend your ears? Take the audio test.