In zero-shot text-to-speech (TTS), Best-of-$N$ (BoN) inference enhances content consistency by selecting from $N$ candidates using an automatic speech recognition (ASR) verifier. However, we identify an underexplored evaluation confound: the apparent quality of a verifier strongly depends on the ASR family judging it.
In the LibriSpeech-PC test-clean dataset with F5-TTS, verifier rankings reverse across Whisper, wav2vec~2.0, and HuBERT evaluators, with same-family verifier-evaluator pairs recovering 2-3 times more oracle headroom than cross-family pairs, despite nearly identical representations (linear CKA of $0.978$). This pattern suggests identity- or lineage-level coupling rather than representational overlap.
We propose two cross-family rank ensembles (rank-averaging and conjunctive max-rank) that achieve the lowest mean word error rate (WER) across three independent evaluators—$1.61\%$ at $N{=}10$ (a $12\%$ relative reduction compared to F5-TTS)—with no measurable degradation in automatic SIM-o/UTMOS metrics; the best single verifier reduces WER from $2.06\%$ to $1.72\%$ (a $16.5\%$ decrease) under the official F5-TTS evaluator. We recommend cross-evaluator triangulation as the default reporting practice.
Blogger's Review: This paper reveals the impact of ASR family on TTS evaluation, highlighting the importance of matching verifiers to evaluators. The proposed cross-family rank ensembles significantly reduce error rates, showcasing the potential of multi-evaluator strategies in enhancing TTS system performance. For researchers, employing cross-evaluators will help provide more reliable results.