How to Build a Pronunciation Test Suite for Local TTS

Direct answer: build a pronunciation suite from the words that can damage a real project: customer names, product names, places, acronyms, currencies, dates, units, technical terms, homographs, and language switches. Put every target into natural sentence context and record the expected reading, locale, importance, allowed variants, and source of truth. Generate with pinned models and settings, then combine automatic transcript checks with human listening. Save approved audio for regression comparison, but re-review when the model, normalization rules, voice, speed, or chunking changes. A pronunciation suite should stop a release on critical names and numbers, not produce a decorative average score.

Test-case fields

Collect failures from actual work

Start with support reports, regeneration history, pronunciation overrides, and domain glossaries. Add high-risk synthetic cases only after the real list. Group cases by normalization, grapheme-to-phoneme, accent, language detection, chunk boundary, or voice-specific behavior. This classification routes the fix to the right layer. If the normalized text is wrong, changing the acoustic model may hide one case and break another.

Write a clear oracle

Use IPA, a simple respelling, a consented reference recording, or a named dictionary source according to reviewer skill. State the intended accent and accepted variants. Do not label a regional pronunciation wrong when the project permits it. For multilingual text, mark the language of each span and the expected transition. Keep the source text unchanged and link any lexicon entry or substitution that should produce the result.

Combine machines and listeners

An ASR transcript can flag missing or substituted words but may agree with the same common error as the TTS system. Acoustic similarity can find large changes without judging correctness. Human reviewers remain necessary for stress, vowels, consonants, pauses, and language authenticity. Blind model identity where possible. Record pass, fail, uncertainty, and reason. Escalate uncertain critical items to a qualified speaker rather than averaging them into a pass.

Run the suite as a release gate

Pin the engine, checkpoint, precision, voice, language, speed, seed, normalization, and chunking. Generate the critical subset on every engine update and the full suite before a release. Store files by stable case ID, not by sentence text. Review diffs and regenerate affected production clips. Pair this suite with the completion benchmark, because correct pronunciation cannot compensate for missing sentences.

Production acceptance record

Before acting on this local TTS pronunciation test recommendation, create an acceptance record for the exact feature and release. Name the owner, reviewer, user task, minimum Mac, operating system, application version, runtime, model revision, precision, input fixture, expected artifact, and pass threshold. Attach one hundred contextual cases, expected reading and accepted variants, critical versus advisory severity. Keep upstream documentation in one section and measurements from your own machine in another. Record every unsupported path, timeout, crash, incorrect output, and manual repair. A result counts as a pass only when the artifact is usable for the stated job. Fast but incomplete output, an unrecoverable cancellation, or a file that cannot be reopened remains a failure.

Run the record once on a clean launch, once after warmup, once with realistic companion applications open, and once after deliberately interrupting the workflow. Exercise the operating steps that matter most here: Start with real project failures. Put targets in sentence context. Define accepted variants and severity. Save raw timings and checks instead of copying only a winning screenshot. Include rejected options and the reason each lost, so a later framework or model update can reopen the decision without starting from memory. Assign an update trigger such as a new runtime release, model revision, minimum operating system, material support issue, or change in licensing. Review the record before changing defaults, removing a fallback, or publishing a broad performance claim.

Evidence to collect before publication

Publish a starter CSV or JSON schema with 100 cases across names, acronyms, numbers, dates, units, homographs, and two-language transitions. Include expected readings and severity but use only licensed reference audio. Run two pinned local models, show anonymized failure categories, and provide the reviewer instructions.

• One hundred contextual cases
• Expected reading and accepted variants
• Critical versus advisory severity
• Pinned model and frontend settings
• Human-review rubric and regression files

Reader worksheet

Take one current project and list every name, number, acronym, foreign phrase, and domain term. Assign a stable ID, expected reading, context sentence, locale, severity, and source. Mark whether the fix belongs in text normalization, a lexicon, phoneme control, or model selection.

Practical checklist

1. Start with real project failures.
2. Put targets in sentence context.
3. Define accepted variants and severity.
4. Use human review for acoustic correctness.
5. Pin every generation setting.
6. Gate releases on the critical subset.

Frequently asked questions