Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Overview§

This paper argues that existing automated TTS evaluators (MOS predictors and Audio-LLM judges) are evaluated against a single scalar "naturalness" score, obscuring which specific speech characteristics they capture. The authors deconstruct naturalness into 10 linguistic dimensions (e.g., stress, intonation, rhythm, articulation, and syntactic prosody) and build the first dimension-level meta-evaluation benchmark for TTS, consisting of 860 utterances rated by trained linguists.

Annotation Schema§

The schema includes 10 perceptual dimensions: lexical stress, sentential accent, phrasing, boundary tones, pitch range, speech rate, rhythm regularity, segmental articulation, coarticulation, and overall naturalness. Each dimension is rated on a Likert scale, providing fine-grained labels that go beyond a single MOS.

Benchmark Construction§

• 860 English utterances spanning diverse speakers, speaking styles, and TTS systems (including neural and concatenative).
• Each utterance is annotated by 2–3 trained linguist raters with high inter-rater reliability (intraclass correlation > 0.7).
• Utterances are deliberately selected to contain linguistically structured errors (e.g., misplaced stress, flat intonation, unnatural pauses).

Evaluated Models§

• MOS predictors: Four publicly available MOS prediction systems (e.g., MOS-SSL, MOS-BCE) that map audio features to a scalar score.
• Audio-LLM judges: Four audio-language models (e.g., SALMONN, Qwen-Audio) prompted with either generic ("rate naturalness") or dimension-specific ("rate stress accuracy") instructions.

Evaluation Protocol§

To measure how well each evaluator captures the 10 dimensions, the authors compute Spearman's rank correlation between model scores and human dimension-level ratings:

$$\rho = 1 - \frac{6\sum d_i^2}{n(n^2-1)}$$

Additionally, they define a dimension alignment score (DAS) as the average correlation across all dimensions, and a selectivity index that measures whether a model excels on only a few dimensions or generalizes broadly.

The prompt for Audio-LLM judges is varied: a generic prompt and a dimension-specific prompt (e.g., "Does the speaker place stress on the correct syllable?").

Results§

• MOS predictors collapse onto a single acoustic signal-quality axis. They correlate strongly with segmental articulation and naturalness but show near-zero or negative correlations with prosodic dimensions (stress, intonation, rhythm). They fail to distinguish distinct linguistic errors.
• Audio-LLM judges show selective, prompt-dependent detection. When prompted generically, they capture only overall naturalness. When given dimension-specific prompts, they can detect some dimensions (e.g., pitch range, speech rate) but not others (e.g., stress patterns, phrasal boundaries). Their performance does not reliably generalize across dimensions, and they are sensitive to prompt wording.
• Neither class of evaluators matches human-level discrimination across all 10 dimensions, highlighting the need for dimension-aware evaluation.

Implementation Insight§

The benchmark and annotation schema are publicly released. A typical workflow for using the benchmark:

import pandas as pd
from scipy.stats import spearmanr

# Load benchmark annotations
benchmark = pd.read_csv("tts_dimension_benchmark.csv")

# Suppose we have a MOS predictor that outputs a scalar score
def mos_predictor(wav_path):
    # Placeholder: in practice, a neural MOS model returns a value in [1,5]
    return 3.7

# Evaluate a dimension, e.g., 'lexical_stress'
human_scores = benchmark["human_lexical_stress"]
predicted_scores = benchmark["audio_path"].apply(mos_predictor)
rho, p = spearmanr(predicted_scores, human_scores)
print(f"Spearman rho for lexical stress: {rho:.3f}")

For Audio-LLM judges, the dimension-specific prompt is fed along with the audio, and the model's output is parsed into a numeric score.

Conclusion§

The paper provides the first fine-grained meta-evaluation benchmark for TTS, exposing systematic blind spots in both MOS predictors and Audio-LLM judges. It advocates for evaluating TTS systems on multiple linguistically motivated dimensions rather than a single naturalness score, and offers a public resource to drive more interpretable TTS evaluation.