Guides

How Blind A/B Testing Works on IABT

By IABT Team Published

How Blind A/B Testing Works on IABT

IABT runs blind A/B tests — head-to-head comparisons where you judge two options, A and B, without knowing which system produced either one. You pick the one that sounds (or looks, or reads) better. That’s it.

Blindness is the whole point. If you knew that A came from a famous model and B came from an unknown one, your expectations would color the verdict. By hiding the labels until after you choose, IABT measures what people actually prefer, not what they expect to prefer.

The flow

  1. Open a test. A test is a set of comparison pairs — for example, the same sentence spoken by two different text-to-speech systems. See how to participate.
  2. Experience A, then B. Play both clips (or view both items) as many times as you like.
  3. Choose. Pick A or B, and optionally say how confident you are and why.
  4. Move to the next pair. Each pair is independent; the order of A and B is randomized per pair so position can’t bias the result.
  5. See the reveal. After you submit, the labels are unmasked so you can see which system you preferred.

Why pairs, not stars

Asking “rate this from 1 to 5” sounds precise but isn’t: everyone anchors the scale differently, and a lone clip gives you nothing to compare against. A forced choice between two options is easier to make, harder to game, and statistically cleaner. Aggregate enough pairwise choices and you get a reliable ranking — which system wins, and by how much.

Where the comparisons come from

Each test is designed by an experiment owner with IABT’s test builder — it defines the prompts, the systems being compared, and the rubric. The first tests on IABT compare AI voice (text-to-speech) systems — but the same machinery works for video, images, or text. Read the audio test methodology.

What makes the results trustworthy

Two safeguards:

  • Blindness + randomized order remove expectation and position bias.
  • A verified human crowd removes bots and duplicate voters. Every verdict on IABT comes from a person who has passed a proof-of-personhood check, so one human counts once. Why that matters.

The result is a preference signal you can actually build on: not “which model has the best marketing,” but “which output real, unique people choose when they can’t see the label.”