AuK: A Practical Guide to the Open-Source Speech Generation and Editing Model

The interesting part of AuK is not simply that it can synthesize speech. It treats speech creation and speech editing as variations of the same problem: give the model an instruction, optionally give it audio context, and ask it to generate the desired audio. That makes the project closer to an instruction-following audio model than a conventional text-to-speech engine.

The authors report roughly 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision spanning five families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. The project releases two variants: the full AuK model for quality-focused generation and AuK-Flash, a distilled version built for fast four-step inference.

AuK and AuK-Flash at a glance

The paper says AuK-Flash is produced in two stages: trajectory-level consistency initialization first creates a few-step student, then a task-routed Decoupled DMD objective further distills it. Separation also receives clean-prediction regression. The result is a four-step model that does not need classifier-free guidance during inference.

How the model works

AuK combines three major pieces. A multimodal large language model handles semantic conditioning from the instruction and optional reference audio. A VAE trained jointly on speech, general audio, and music represents acoustic information at 50 Hz. A hybrid rectified-flow Transformer then generates the target audio using dual-stream MMDiT blocks followed by unified single-stream DiT blocks.

• Semantic conditioning: a multimodal language model interprets what you want the audio to become.
• Acoustic conditioning: a 50 Hz audio VAE carries information about speech and broader audio structure.
• Generation backbone: a hybrid rectified-flow Transformer first processes semantic and acoustic streams separately, then fuses them for final generation.
• Training sequence: generation-only warm-up comes first, followed by joint generation-and-editing pre-training.
• Post-training: human-feedback preference optimization is used for open-ended editing, while reward-based reinforcement learning is applied to speech generation.

That shared architecture is why AuK can accept instructions that would normally belong to very different tools. A command can ask it to speak new text, replace a phrase in an existing recording, remove reverb, keep a particular speaker, make a voice happier, raise pitch by semitones, or turn normal speech into a whisper.

Every task AuK currently exposes

The official repository groups AuK into five task families and lists sixteen concrete tasks. The table below mirrors that released task set so you can see the full surface area before we walk through each example.

1. Instruct TTS: describe the voice you want

Instruct TTS generates speech from a text description of the desired voice. You do not need a reference recording. The model can condition on details such as apparent age, vocal tone, pace, energy, emotional delivery, clarity, and speaking style, then synthesize the requested sentence.

The official ComfyUI workflow ships with essentially this kind of text-only example. The repository also shows a richer Chinese prompt describing a woman in her twenties speaking gently and affectionately to a partner who has just come home, then asks the model to say a specific welcome-home line. That illustrates the point of Instruct TTS: the voice description can be much more specific than a simple preset such as warm or female.

The official English Instruct TTS demo asks for a theatrically charged Shakespearean villain voice: a rich baritone, deliberate pacing, crisp staged diction, and a tone that shifts from false sweetness to menace. This is the generated result:

2. Zero-shot TTS: clone a voice from a reference clip

Zero-shot TTS adds reference audio. Instead of describing a voice from scratch, you ask AuK to say new text using the reference speaker's voice. The project page demonstrates this with a short speaker reference and a new Chinese sentence; the README shows the same pattern in English.

The reference clip supplies speaker identity and acoustic cues, while the instruction supplies the target content. In the CLI, the authors recommend either giving an explicit generation duration or providing text/transcript hints so AuK can estimate the target length.

In the official English demo, AuK receives a reference speaker and generates a new line: “If not even light can escape the event horizon, how do you think you’ll escape me?” Compare the reference with the synthesized result:

3. Speech content editing: replace, insert, or delete words

This is one of AuK's most useful editing modes. Give it an existing recording and describe exactly how the spoken content should change. The goal is to modify the words while keeping the surrounding speaker identity, prosody, and acoustic context consistent.

The task supports replacement, insertion, and deletion. In practical production work, this is the kind of operation you would use to repair a narration line without asking a speaker to re-record an entire paragraph.

4. Lyric editing: change sung words without rebuilding the song

AuK extends content editing to vocals. Lyric editing can replace, insert, or remove words in a singing recording while attempting to preserve the original melody and vocal identity. That is a much harder constraint than ordinary speech replacement because timing, pitch contour, musical phrasing, and the singer's tone all need to stay coherent.

5. Pitch editing

Pitch editing changes the perceived pitch of the speech by a specified number of semitones. Because this lives in the same instruction interface, the request can be expressed directly instead of selecting a dedicated pitch-shifter tool.

6. Speed editing

Speed editing changes speaking rate. The repository notes that output duration scales with the requested speed factor, so this is a structural timing change rather than an instruction to merely sound more energetic.

7. Volume editing

Volume editing asks the model to increase or decrease level by a requested decibel amount. It is the simplest of the acoustic edits conceptually, but its inclusion matters because AuK tries to make even signal-level operations available through the same language-driven interface.

8. Emotion editing

Emotion editing changes how a line is delivered while keeping its words and speaker identity. This is useful when the content is already correct but the performance needs a different emotional reading.

9. Timbre editing

Timbre editing changes vocal color from a natural-language description. The intended content stays the same, but the model can be asked for a brighter, darker, softer, raspier, or otherwise differently characterized voice quality.

10. De-accent editing

De-accent editing aims to reduce a regional accent while preserving the speaker's identity and the spoken content. This is a particularly demanding example of disentangling who is speaking from how pronunciation is realized.

11. Nonverbal editing

Nonverbal editing targets sounds that accompany speech rather than the words themselves. The project explicitly lists breaths, laughs, and coughs as examples. The instruction can ask AuK either to remove an unwanted event or add one where it makes sense.

12. Whisper conversion

Whisper conversion works in both directions: normal speech can be turned into whisper, and whispered speech can be converted back toward normal voiced speech. The task is designed to preserve speaker identity and linguistic content while changing phonation style.

13. Speech enhancement

Speech enhancement covers denoising, dereverberation, and general restoration. The official CLI example is written in Chinese and asks AuK to keep all original speakers, remove noise and reverberation, and output clean speech with the same duration.

The authors position restoration as one area where AuK remains competitive with specialized systems, while the strongest claims in the paper are around zero-shot and instruction-controlled generation plus general instruction-guided editing.

14. Speech separation

Speech separation lets an instruction identify which talker to retain by speaking order. Instead of choosing an abstract source index, you can ask for the first, second, or otherwise described speaker and remove the others.

15. Music separation

Music separation extends the model beyond spoken recordings. The released task list describes extracting a singing voice from a mix or keeping human vocals while removing the rest of the accompaniment.

16. Target speaker extraction

Target speaker extraction identifies the person to keep from what that person says. This gives the instruction semantic grounding: rather than supplying a separate speaker embedding, you can describe the target using a phrase associated with that speaker in the recording.

The common interface is the real design idea

AuK's Python API uses a message structure that looks more like a multimodal chat request than a traditional audio DSP function. A user message contains text plus optional audio. The same generate call is used whether the instruction is TTS, content editing, enhancement, or another supported transformation.

That design lowers the conceptual cost of adding a new operation to a product. Instead of wiring a separate model and parameter panel for every task, an application can expose a common instruction surface and let the underlying model route the request. The tradeoff is that language-driven control can be less deterministic than a fixed DSP operator, so production tools still need careful previews, constraints, and undo/history around destructive edits.

How to run AuK

The official repository requires Python 3.10 and pins the tested PyTorch family to the 2.7 ABI line. The documented setup clones Tencent-Hunyuan/AuK, creates an isolated environment with either uv or Conda, and installs the package in editable mode. Optional extras add Gradio, ComfyUI integration, prompt enhancement, ASR dependencies, or fine-tuning support.

1. Clone the Tencent-Hunyuan/AuK repository.
2. Create and activate a Python 3.10 environment with uv or Conda.
3. Install the core package for CLI/Python inference, or add the Gradio/ComfyUI/train extras you need.
4. Download AuK or AuK-Flash weights plus the required Qwen2.5-Omni-3B component into the documented checkpoint layout.
5. Run auk-infer for command-line inference, auk-gradio for the web interface, the supplied ComfyUI workflow, or call AukInfer from Python.

Gradio, ComfyUI, CLI, and Python are all first-class paths

The repository includes an interactive Gradio app that can expose the base model, Flash, or both. It can place the two variants on separate GPUs or load them on one GPU. There is also an AuK Model Loader plus AuK Generate / Edit node for ComfyUI, and the included workflow starts with a three-second text-only example.

For code, the central class is AukInfer. You build a message containing instruction text and, for editing or reference-conditioned tasks, an audio item. Calling engine.generate(...) returns generated audio and a sample rate, which the helper can then save to disk. This same request shape also forms the basis of AuK's JSONL fine-tuning format.

What the benchmark claims actually say

The technical report evaluates AuK across generation, editing, enhancement, and separation rather than on one TTS benchmark. For generation it reports Seed-TTS-Eval metrics for intelligibility and speaker similarity plus InstructTTSEval for free-form timbre following. Editing evaluations include MMAE-Speech, SpeechEditBench, and Ming-Freeform-Audio-Edit. Restoration and separation use perceptual metrics including DNSMOS-OVRL and UTMOS across datasets such as DNS Challenge, CHiME-4, and Libri2Mix.

The authors' own summary is appropriately specific: AuK shows leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while staying competitive on signal-level restoration tasks. That is a more useful description than calling it universally best at every audio operation.

Where AuK is especially interesting

• Voice products that want TTS and editing in one model instead of separate generation and repair pipelines.
• Podcast and narration tools where a user may need to replace a phrase, change delivery, remove noise, or isolate a speaker after generation.
• Research on instruction-following audio models, especially systems that mix semantic and low-level acoustic edits.
• ComfyUI workflows that benefit from one node family handling generation and transformations.
• Developers who want inspectable code and weights under a permissive open-source license.

What to keep in mind before building around it

AuK is extremely new. The code and weights were released on September 9, 2026, so production experience, hardware guidance, third-party benchmarks, and integration patterns are still developing. The repository's documented examples are GPU-oriented, and the current implementation uses Qwen2.5-Omni-3B; its Gradio documentation explicitly says Qwen3-Omni is not currently supported through the corresponding path option.

Also separate model capability from product safety. Voice cloning, target-speaker extraction, and realistic speech editing can be useful creative tools, but an application still needs consent, provenance, access controls, and clear user expectations around how recordings may be transformed.

License and availability

Tencent releases the publicly available AuK training code, inference code, parameters, and weights under the MIT License. The project provides source code on GitHub and model links through Hugging Face and ModelScope. Because license terms can change around related assets or downstream components, check the current repository and any separately downloaded model dependencies before shipping a commercial product.

Why AuK matters for speech software

For years, voice software has mostly been assembled as a chain of specialists: TTS for generation, a separate editor for content fixes, another model for denoising, another separator for overlapping speakers, and DSP tools for pitch, speed, and gain. AuK asks whether a single instruction-conditioned model can cover enough of that chain to become the common editing layer.

The first release already spans sixteen named tasks, and the interface stays consistent across them. If that approach holds up under broader independent testing, the biggest contribution may be less about any one benchmark and more about collapsing a fragmented speech stack into one programmable model surface.