On text-based audio editing

Posted Apr 14, 2025 Updated May 6, 2026

By Rahul TR

16 min read

Overview and Workflow

Editing speech audio by modifying its transcript is an emerging workflow that treats audio like text. Instead of cutting waveforms, an editor transcribes the audio, lets the user edit the transcript (delete words, rephrase sentences, or insert new words), and then reconstructs the audio to reflect those changes. This approach is especially useful for spoken-word content like audiobooks, audiodramas, and podcasts, where precise word-level edits are needed. Key components of such a system include:

Speech Transcription (Automatic Speech Recognition): to generate an editable transcript of the audio.
Audio-Text Alignment: to align transcript words with timestamps in the original audio, so that text edits map to specific audio segments.
Voice Cloning / Neural TTS: to generate new audio for any inserted or modified text in the original speaker’s voice.
Seamless Audio Reconstruction: to splice the original and synthesized audio segments together smoothly, preserving natural flow.

Below, we delve into each of these components, highlighting leading tools/models and techniques. We also survey existing solutions and open-source projects that demonstrate this workflow in practice.

Speech Transcription Engines (ASR)

High-accuracy transcription is the foundation of text-based audio editing. The transcript will be the user’s editing interface. Key options include local ASR models (which run on your own hardware) and cloud-based transcription APIs:

Local models (Open Source): OpenAI Whisper is a state-of-the-art example. Whisper is an offline model that achieves near human-level accuracy on English and many other languages, available in various sizes. It can produce transcripts with timestamps for segments and, with some tooling, even word-level timing. As an open model, it’s free to run (no per-use cost) and keeps data on-premise, though it requires significant compute for long files. Alternatives include Kaldi-based models (e.g. Vosk), Coqui STT, and others, but Whisper’s accuracy on English is generally superior. Whisper is also available as a hosted API by OpenAI.
Cloud-based APIs (Software-as-a-Service): Numerous providers offer speech-to-text with high accuracy and convenience. Top English STT services in 2025 include Deepgram, OpenAI (Whisper API), Microsoft Azure Speech, Google Cloud Speech-to-Text, AssemblyAI, Rev AI, Speechmatics, Amazon Transcribe, and IBM Watson. These services handle the heavy computation on their servers and return transcripts, often with word timestamps and confidence scores. Many also support speaker diarization, punctuation and casing, and additional analytics (keywords, sentiment). For example, AssemblyAI’s API emphasises accuracy, speaker diarization, and a flexible pay-as-you-go model; Deepgram is known for real-time streaming transcription and competitive pricing (~$0.004/min); OpenAI’s Whisper API offers near-human accuracy at about $0.006/min.

To choose a transcription engine, consider accuracy on the relevant audio (e.g. audiobook narration vs. casual podcast speech), cost, whether data can be sent to the cloud or must stay local, and real-time needs. Whisper (local) provides an excellent free option if you have the hardware, while services like AssemblyAI or Deepgram simplify integration and may handle dialects or poor audio more robustly.

Popular transcription tools at a glance:

OpenAI Whisper · Open-source model (local or via API): Near-human accuracy on English; supports punctuation and multiple speakers; provides timestamps. Runs locally (no data leaves device) but requires GPU for speed. Also available as an OpenAI-hosted API (~$0.006/min).
AssemblyAI · Cloud API: High-accuracy English STT; automatic speaker diarization and filler-word detection; returns word-level timestamps and confidence. Pay-as-you-go pricing with no upfront costs.
Deepgram · Cloud API or On-Prem: Real-time streaming and batch transcription; offers custom model training. Advanced features like sentiment and topic detection. Competitive pricing (~$0.004/min). Can be deployed on-prem for privacy.
Google Cloud Speech-to-Text · Cloud API: Trusted accuracy on English (and many languages); automatic punctuation and casing; diarization and multi-channel recognition. Usage-based pricing via Google Cloud.
Microsoft Azure Speech · Cloud API: Enterprise-grade STT with high accuracy; diarization and batch or real-time modes. Offers a customisable Custom Speech service to adapt to domain-specific vocabulary.
Amazon Transcribe · Cloud API: Scalable transcription with custom vocabulary filtering, speaker identification, and timestamps. Provides specialised models (e.g. “Medical Transcribe” for clinical dictation).

Note: Cloud APIs simplify development but send audio off-site; local models keep audio private but require maintenance of ML models and infrastructure. In practice, some text-based editing apps combine approaches, offering fast local transcription for short clips and an API option for longer audio or higher accuracy.

Audio-Text Alignment (Forced Alignment)

Accurate audio-text alignment is what enables linking each word in the transcript to its exact timing in the audio. This is crucial for a text-based editor: when the user selects or deletes a word in the transcript, the software needs to know the corresponding audio time interval to cut or replace. While many ASR engines provide timing for recognised words, a separate alignment step is often needed after manual transcript corrections or to align an existing transcript (like a script) to the original audio.

Forced alignment is the process of taking an audio file and its transcript, and syncing them by finding where each word (or phoneme) occurs in the audio. Specialised tools can produce very precise word-boundary timestamps, even for long files like audiobooks.

Leading alignment tools for English include:

Montreal Forced Aligner (MFA): A widely used open-source aligner built on Kaldi. MFA uses acoustic models and a pronunciation dictionary to align at word and phoneme level. It supports English and dozens of other languages (with pretrained acoustic models available). Given an audio file and its transcript, MFA outputs timestamps (e.g. in Praat TextGrid format) for each word. It’s quite accurate when the transcript exactly matches the audio. MFA is command-line based and requires some processing time since it effectively runs a full ASR pass constrained by the transcript.
Gentle: An open-source “robust yet lenient” aligner for English, also based on Kaldi. Gentle was designed to tolerate slight discrepancies between the transcript and the spoken audio (e.g. skipped words, paraphrasing) by being more flexible in matching. It produces word-level alignments with confidence scores, indicating words that didn’t align perfectly. Gentle has been used in projects like transcribing podcasts and in early versions of Descript.
Aeneas: An open-source Python/C library geared toward aligning long-form narration (often used for audiobooks or subtitles). Aeneas can take a text (like a book chapter) and the corresponding audiobook recording and automatically produce a sync map (timestamps for each sentence or paragraph). Under the hood it uses MFCC analysis and heuristics to align, or can leverage ASR. It’s a handy tool for syncing e-book text with audiobook audio.

Many ASR services also output timestamps that can serve as alignment. For instance, Whisper can return time intervals for each text segment and per-word timing. Cloud APIs like Google’s or Deepgram also return per-word start/end times. In practice, an editing app might use those directly if the initial transcript is trusted. However, if the user corrects transcript errors or provides their own script, running a forced aligner ensures the corrected text matches the original audio timing.

Accuracy considerations: Alignment tools typically assume the transcript is verbatim. If the user performs significant edits (reordering or replacing phrases), those portions won’t align to the original audio; alignment is only used for unchanged parts, and new synthesised audio will cover the rest. For sections that are merely deleted or trimmed, having precise word timings allows surgical removal without cutting off neighbouring words.

Voice Cloning and Neural TTS

When the user edits the transcript to add or change words, the system must generate new audio for those edits in the same voice as the original speaker. This is where voice cloning comes in, using neural text-to-speech (TTS) techniques to synthesise speech that mimics the original speaker’s voice and speaking style.

State-of-the-art voice cloning techniques and models include:

Multi-Speaker TTS with Speaker Embeddings (SV2TTS): In this paradigm, a speaker encoder converts a few seconds of the target speaker’s audio into a fixed embedding; a synthesiser (like Tacotron 2 or FastSpeech) generates a spectrogram from text conditioned on that embedding; and a vocoder (like HiFi-GAN) converts the spectrogram to audio. This enables zero-shot voice cloning from as little as 5 seconds of audio. The open-source Real-Time Voice Cloning project by Corentin Jemine implements this approach. Quality improves with more audio, and output can be quite convincing for short phrases.
Voice Fine-tuning (Custom TTS models): Fine-tuning a pretrained TTS model on recordings of the target speaker. Services like Microsoft’s Custom Neural Voice (Azure Cognitive Services) use this approach: users upload training recordings and the service trains a custom TTS voice. This yields higher quality/cloning accuracy but requires more data (often 30+ minutes of speech) and processing time.
Zero-Shot and Few-Shot Voice Cloning: Recent research has pushed the minimum audio needed to clone a voice dramatically lower. In 2023, Microsoft unveiled VALL-E, a neural codec language model that can synthesise a person’s voice from only a 3-second audio sample, preserving not just timbre but also emotional tone and acoustic environment. While VALL-E and similar models remain research prototypes, they demonstrate the rapidly advancing state of voice cloning.
Neural TTS Models: Even without cloning a specific new voice, the underlying TTS technology is crucial. Modern systems are fully neural. Models like Tacotron 2 or Transformer TTS generate speech spectrograms from text (capturing natural prosody and pronunciation), which neural vocoders like WaveNet, HiFi-GAN, or WaveGlow then convert to audio. These neural voices sound highly natural. Open-source frameworks like Coqui TTS provide implementations of many of these models and support training or fine-tuning voices.
Voice Conversion: A related technique that takes audio from one speaker and transforms it to sound like another, without changing the linguistic content. This could be used if a different voice actor records the new lines, then converts them to match the original voice. In practice, TTS with cloning is the more direct route.

Notable Tools / Services for Voice Cloning:

Descript Overdub: A commercial tool that clones your own voice from ~10 minutes of speech. Typing a new word or sentence in the transcript causes Overdub to generate it in your voice, matching the acoustic environment so the edit blends seamlessly with the original.
ElevenLabs: A popular cloud-based AI voice platform known for very realistic voice cloning and synthesis from short samples. Supports cloning from a few minutes of audio with convincing intonation and emotion. Used for audiobooks and dubbing.
Resemble AI: A commercial voice cloning API that allows rapid cloning with as little as ~5 minutes of audio. Offers speech-to-speech voice conversion, emotion control, and style morphing.
Open-Source Projects: The Real-Time Voice Cloning project (Corentin Jemine) provides a local SV2TTS implementation using Tacotron 2 and WaveRNN. Coqui TTS has a growing collection of pre-trained multi-speaker models, including some that do zero-shot cloning. Other notable open models include YourTTS (multilingual zero-shot cloning with emotion transfer) and VITS (a VAE-based TTS model with zero-shot cloning capability).

Perfect voice cloning is still challenging. Synthesised segments might sometimes sound slightly off or have different energy than the original. However, for short corrections or added phrases, listeners often can’t tell the difference, especially when stitching and prosody are also handled well (see below).

Seamless Audio Reconstruction Techniques

Once we have the pieces (original audio with certain words marked for removal, and new synthesised audio for inserted or replaced words) the final step is to reconstruct the audio so that it sounds coherent and natural. Several techniques help achieve seamless edits:

Word-Level Cutting with Crossfades: When deleting or replacing audio, cut at points that won’t introduce a click or unnatural transition. A short crossfade (10-50 ms) overlaps a tiny portion of the end of one clip with the start of the next, fading audio out and in to smooth the join. This avoids abrupt cutoffs or jumps in ambience, and can soften differences in energy levels. Many editing tools do this automatically; Descript, for instance, has technology to make cuts “basically undetectable.”
Maintaining Room Tone and Background Audio: If the original audio has background room tone, noise, or reverb, a cut could cause that background to change suddenly. A strategy is to extract a bit of “room tone” from the original and crossfade it underneath edits. When inserting synthesised speech, add a low-level background noise to match the original noise floor. Descript’s Overdub synthesises in the “same acoustic environment,” implying they model the background/echo so inserted words have the same room sound as the surrounding audio.
Prosody and Timing Adjustment: One challenge with inserting synthesised words into natural speech is matching prosody (pitch, speed, and intonation) so the insertion doesn’t sound flat or out-of-context. A technique is to generate the synthetic audio with context: synthesise the whole sentence including the new word, then extract only the needed portion. Alternatively, DSP techniques like TD-PSOLA can slightly time-stretch or pitch-shift the synthetic audio to better match neighbouring real speech. Research by Morrison et al. (2021) on context-aware speech editing notes that simple cut-and-paste edits often have mismatched intonation or rhythm at boundaries (“prosody discontinuities”), and proposes generating a smooth pitch and duration contour that blends with the context.
Length and Timing: When removing words, close the gap so the word appears never to have been there. Forced alignment provides precise word timings for surgical removal. The editor may also slightly adjust pause length between surrounding words to maintain believable rhythm. Human speech has natural variation in pause length, so this requires some care.
Synthesising Longer Segments: For larger rewrites (an entire sentence or additional line), it may be better to generate the whole new sentence with a voice clone so its internal prosody is consistent, then crossfade it into the original at sentence boundaries or breath pauses. This avoids multiple small splice points at the cost of losing original audio for that span.

In summary, smooth reconstruction follows the same principles a skilled human audio editor would apply (cut at natural breakpoints, use fades to hide the seams, maintain consistency in noise and pacing) but in an automated way. Some advanced systems incorporate AI-based prosody correction, adjusting synthesised parts to better match surrounding speech. Even basic methods like crossfading and careful alignment can yield seamless edits.

Existing Tools and Projects

This text-based editing paradigm has moved from research into real products in recent years. Descript pioneered much of this functionality for podcasts and videos, and other tools and open-source projects have followed.

Descript · Commercial app (Windows/macOS): Comprehensive text-based audio/video editor. Automatically transcribes and aligns text to the waveform so edits in text apply to correct audio regions, rendering cuts that are “basically undetectable.” Standout feature is Overdub (voice cloning): users train a clone of their voice (~10 min of audio) and generate new words in that voice. Also includes one-click filler word removal, silence gap deletion, and noise reduction (Studio Sound). Integrates all components: transcription, alignment, cloud TTS voice cloning, and automatic crossfades.
Hindenburg PRO 2 · Commercial DAW (Windows/macOS): A professional audio editor geared toward radio and podcasts. Introduced offline transcription and text-based editing, allowing users to edit audio “like in a Word Document.” Performs transcription on-device (no cloud needed). Does not offer AI voice cloning; used for editing out mistakes and rearranging content, with any new lines recorded manually.
Audapolis · Open-source (Windows/Linux/macOS): An open-source project inspired by Descript’s workflow. Automatically transcribes audio and provides a word-processor-like interface for editing spoken word audio. All processing is local (no cloud services), making it suitable for sensitive material. Supports cutting, copying, and reordering via text. No integrated TTS voice cloning for adding new words.
Reduct.video · Commercial web platform: A text-based video editing tool created by the team behind the Gentle aligner. Uses automated transcription and alignment to let users edit video by editing the transcript, primarily by cutting or selecting snippets. Does not generate new speech, but demonstrates the versatility of transcript alignment for editing.
Research Prototypes · Academic code: Active research into improving text-based speech editing. Notable papers include VoCo (Adobe’s early prototype for word insertion), EditSpeech (2021, partial inference to regenerate only edited portions of a spectrogram), and CampNet / FluentEditor (2022-2024, modelling acoustic context to smooth over edits). FluentEditor (Interspeech 2024) performs cut, copy, paste on text and applies neural prosody correction so the result sounds natural in context. As these techniques mature, future tools can incorporate them for even more seamless edits.

Many components discussed above are also available as modular SDKs. For example, NVIDIA’s Riva SDK offers ASR and TTS (including voice adaptation) that could be used to build a custom text-based editor. Open-source frameworks like ESPnet or Fairseq have ASR and TTS models that a developer could integrate. The challenge is gluing them together with a user interface that non-technical users find intuitive, but projects like Audapolis show that it’s feasible even outside large companies.

Conclusion

Developing an app for transcript-based audio editing involves bringing together several cutting-edge speech technologies: accurate transcription (to get the words right), forced alignment (to tie words to audio), voice cloning TTS (to generate new words in the same voice), and intelligent audio splicing (to hide the edits). Each of these areas has mature tools or APIs available in 2025, and some existing products have already proven the concept for English spoken-word content.

By leveraging these components (e.g. Whisper or Deepgram for ASR, Montreal Forced Aligner or Gentle for alignment, ElevenLabs or an open-source SV2TTS for generation, and applying good editing practices) one can build a system where editing an audiobook or podcast is as easy as editing a text document, with the edited audio sounding like it was never altered. The result is a dramatic boost in productivity for content creators and audio engineers working with speech content, turning hours of tedious waveform editing into a simple, high-level text revision process.

$ ls incoming/

$ ls outgoing/

Design

This post is licensed under CC BY 4.0 by the author.