Code: https://github.com/jordandare/echo-tts

Demo: https://huggingface.co/spaces/jordand/echo-tts-preview

Recently, we have been working on a diffusion text-to-speech model, named Echo, following our prior work on Parakeet. This work was made possible through the TPU Research Cloud (TRC) program, to which we are deeply grateful.

Parakeet, which is autoregressive in DAC token space, struggled with consistency - - it would often take many tries to generate a semantically correct sample - - and exhibited poor voice cloning capability. Given our goal of generating high quality speech samples with performant voice cloning (and an indifference towards real-time interaction capabilities, which may favor autoregressive methods), we felt it reasonable to try a diffusion approach.

We train a 2.4B DiT, Echo, that is able to generate audio segments of up to 30 seconds, conditioned on target text and up to two minutes of speaker reference audio. We generate Fish Speech S1-DAC latents and as a result are able to produce 44.1kHz audio.

Our model is quite fast at generating single samples (at least relative to frontier open-source autoregressive approaches). For example, given a 120-second audio prompt, we are able to generate 30 seconds of audio in only 1.45 seconds on an A100 (RTF < 0.05).

Below are samples, and further below is a more detailed description of the model.

Samples

Details

Dataset

We collect a dataset consisting of around 160K hours of podcast-like audio. Similar to Parakeet, we split the data into (≤) 30-second segments and transcribe all segments individually with (our) WhisperD, a Whisper model fine-tuned to output speaker diarization, disfluencies, and non-verbal events, based on Whisper.

Autoencoder

We use the Fish-Speech S1-DAC autoencoder, which is licensed under CC BY-NC-SA 4.0 (and therefore the full generation pipeline in our setup is non-commercial). S1-DAC is a discrete codec that encodes 44.1kHz audio into 10 discrete residual codebooks at ~21.5Hz. Thus, ~30 seconds of audio corresponds to a (640, 10) integer array.

We choose to encode/store/load all of our data discretely as codes. Since we will use diffusion to model the data, we will need to represent the codes as continuous latents. Similar to DAC, S1-DAC forms its pre-decoder continuous latent by summing individual up-projections of each of the 10 codebook entries; these entries themselves have dimension 8, and are up-projected to dimension 1024. Thus, the resulting 1024-dimensional output subspace must have rank ≤ 80. We hoped to (in some sense) preserve the geometry of the pre-decoder latent space, so rather than concatenate codebook entries, we apply PCA to the pre-decoder latents and extract the first 80 indices of the rotated latents.⁴

Model

Echo is a 2.4B DiT (though decoder compute per diffusion timestep is closer to a 1.4B transformer with seq_len = 640 and kv_seq_len ≤ 2048).⁵ The architecture is relatively simple.

Echo consists of a speaker reference transformer, text transformer, and diffusion decoder:

• The speaker reference transformer processes up to two minutes of (clean) speaker reference latents. This originally corresponds to a latent sequence length of 2560. To reduce sequence length relative to target, we patch these latents (patch_size = 4) so that the transformer sequence length is 640. This encoder is causal.⁶
• The text transformer processes utf-8 bytes of maximum length 768 during training, with bidirectional attention (and appropriate masking).
• The diffusion decoder denoises noised latents (sequence length 640). It is a traditional transformer with:
  • Self-attention replaced with joint-self-cross-attention, where the keys/values consist of the self(denoiser) keys/values concatenated with the speaker and text transformer output keys/values.⁷
  • AdaLN with LoRA to reduce param count, with a tanh applied on gate.

We use a SwiGLU MLP, QK-norm, RoPE (though in decoder applied to only half the heads), and gated attention.

Speaker reference selection

We need training tuples of the form (speaker_reference_audio, target_text, target_audio); while we have the target text and audio from the aforementioned segmentation, extracting speaker reference audio is slightly more involved. At this stage, our data consists of long audio samples (generally 30–60 minutes, think podcast-like; we will use the term “episode” moving forward) that have been segmented into <30-second clips. The most naive approach is to, for a given target clip, randomly select a subset of (distinct) segments from the same episode and concatenate up to 120 seconds of latents. This works reasonably well, though we try to improve upon this by diarizing each episode and selecting other clips based on a fuzzy/heuristic approach that tries to match speaker composition of reference concatenation and target.⁸

The upside of such a setup is that it is simple and allows us to drop in any single-or-multi-speaker audio clip(s) as speaker reference, without any additional labeling/transcribing/processing. The downside is that if we have two speaker reference clips where the speakers are in different “acoustic environments” and use the concatenation of these clips as the speaker reference, generating conversation between the two speakers is likely out-of-distribution for the model. Another downside is that given a reference clip of two speakers conversing, there is little (if any) control over the correspondence between [S1], [S2], …, and the reference speakers, and sampling over different random seeds might be necessary to obtain a desired permutation.

That said, these two missing behaviors/functions could be (relatively) straightforwardly fine-tuned into the model given the proper data. (And it might be possible to synthesize such data via diarization and audio augmentation.)

Training

We train on a TPU v4-64 pod in JAX/flax (linen). We use Muon (which seemed to outperform Adam in our smaller ablations) with a batch size of 768 for 800K steps, with a WSD LR schedule. Computation is performed in BF16 (weights and optimizer states are kept in FP32). We train with independent speaker condition dropout (10%) and text condition dropout (10%) for enabling CFG at inference.

Since many audio segments are less than 30 seconds (i.e., latent sequence length < 640), we zero-pad all latent sequences to length 640 and treat this padded, fixed-sized latent as the denoiser target. During inference, we sample starting from noise with shape (batch, 640, 80), and crop the generated audio by heuristically identifying trailing regions of (near) zeros in the generated latent.

We train with a Rectified Flow setup, using a stratified logit-normal timestep distribution.

Sampling

We try a few different sampling methods, generally consisting of Euler sampling with the RF ODE. Below is an example of sampling hyperparameters.

• Number of steps (30 generally works well)
• CFG (select one version below) and choose guidance scales
  • Joint unconditional (2× NFE)
    • v_pred = v_pred_cond + w * (v_pred_cond - v_pred_uncond) where v_pred_uncond drops both text and speaker conditions
  • Independent guidance (3× NFE)
    • v_pred = v_pred_cond + w_text * (v_pred_cond - v_pred_uncond_text) + w_speaker * (v_pred_cond - v_pred_uncond_speaker)
  • Alternating guidance (2× NFE)
    • v_pred_cond + w_M * (v_pred_cond - v_pred_uncond_M) where M alternates between text and speaker each step
• CFG_min_t, CFG_max_t (we find 0.5 for min T often works well)
• initial_noise_scale
  • (sometimes scaling the initial noise by a factor of 0.8 or 0.9 reduces artifacts; as far as I understand, this is totally unprincipled and suggestive that something is “off”; should be looked into further)
• Temporal score rescaling
  • Anecdotally (and with low confidence) k = 1.2, sigma = 3.0 may result in more robust/consistent/less-“artifacty”/flatter samples, while k = 0.96, sigma = 3.0 results in sharper samples (maybe higher cloning potential/ceiling for certain voices)?

We have not conducted a formal evaluation of different sampling hyperparameters. In our experience, the “Independent guidance” CFG option works well and has the benefit of decoupling the text and speaker guidance scales (though alternating guidance also decouples, and even joint unconditional CFG seems to work well in many cases). We have also at various points tried APG, Karras SDEs, CFG++, DPM-Solver++, etc. We leave further investigation and evaluation of such methods to future work.

(Dynamic-)Block-wise diffusion

Our model as described above has the following limitations:

1. It always will generate (zero-padded) 30-second chunks, even when the user anticipates the content will be much shorter (this still will generate proper-length content but spends more compute than necessary; consider time-to-first-byte (TTFB)).
2. We cannot generate audio continuations (e.g., audio from seconds 10 to 20 given audio from the first 10 seconds). This also hinders generating longer-than-30-second samples via overlapping window approaches.

We propose fine-tuning Echo to perform dynamic-block-wise diffusion, which addresses both these points.

Specifically, we introduce a latent-prefix encoder (which shares the same architecture as the speaker reference encoder) that will causally process the clean (640, 80) target latents.⁹ During fine-tuning, we randomly partition x_t ∈ (640, 80) into (up to) 3 separate segments¹⁰ with block indices (0, 1, 2). In the decoder joint-self-cross-attention, the noised latents only can attend to other noised latents within the same block. We extend the existing keys/values with the (kv-projected) latent-prefix outputs¹¹ and mask so that each noised latent attends only to latent-prefix keys from (strictly) previous blocks.

This allows us to specify any block-size to generate at inference: We simply process all previously generated (or given) latents with the latent-prefix encoder¹² and then sample from the RF ODE with initial x_t ∈ (desired_length, 80)¹³. Since S1-DAC is causal, we can stream blocks of generated audio. If we want to decrease TTFB, we can first generate a short block and immediately decode it (and then generate longer blocks if desired).

To fine-tune this behavior, we initialize the latent-prefix encoder from the speaker reference encoder, initialize new w_k_latentprefix/w_v_latentprefix, and adapt the latent-prefix encoder (along with the MLP/attention of the decoder) with LoRA (all weights besides the LoRA layers and new w_k/w_v are frozen). We add RoPE (with patch-adjusted frequency) to the latent-prefix keys. We fine-tune with Adam for 100K steps. (Earlier versions of our training setup employed block-wise methods during pretraining, but for Echo we chose to leave this for fine-tuning.)

It is unlikely that we will include the block-wise fine-tuned weights in our initial release.

Acknowledgments

We are immensely grateful to the TPU Research Cloud program for their continued support. We’d also like to thank Professor Duan, Ge Zhu, and the AIR lab (https://labsites.rochester.edu/air/) at University of Rochester. Lastly, we thank Anaheim and https://huggingface.co/MrDragonFox from the OpenSesame audio research Discord server for their valuable early feedback and advice.

Audio reference samples are brief excerpts used for technical demonstration purposes only. All rights belong to original creators.

Bibtex

To cite this blog post, please use:

bibtex @misc{darefsky2025echo, author = {Darefsky, Jordan}, title = {Echo-TTS}, year = {2025}, url = {https://jordandarefsky.com/blog/2025/Echo/} }

In a previous blog post, we described Parakeet, an autoregressive approach to generating natural, conversational speech. Here, we introduce our more recent work that explores using diffusion instead of autoregression.

Overview

In this blog post, I describe my work with Ge Zhu and Professor Zhiyao Duan to develop an initial version of a text-to-speech (TTS) model we call Parakeet. The research presented in this post was completed early last fall and was supported by Google’s TPU Research Cloud (TRC) program. This project would not have been possible without their immense generosity.

Parakeet takes in as input a text prompt, optionally containing multiple speakers or non-verbal events like “laughter,” and outputs up to 30 seconds of corresponding audio. In designing Parakeet, we had two main goals in mind for our model:

1. To be architecturally simple — as close to end-to-end as possible.
2. To be able to produce natural, conversational speech. This includes generating multi-speaker samples, as well as generating common “events” found in dialogue, such as laughter and coughing.

A brief overview of our methodology is as follows:

1. We curate a ~100,000 hour dataset of audio-transcription pairs. ~60,000 hours come from Spotify Podcast Dataset ; we fine-tune a Whisper model to provide speaker/event annotated transcriptions and then backtranslate the entirety of the Spotify Podcast audio using this model.
2. We train an autoregressive transformer to predict audio tokens (specifically, DAC codes), conditioned on raw transcription text. We develop a modified classifier-free guidance (CFG) technique, which we call CFG-filter, to improve quality.

We plan to release our fine-tuned whisper models and possibly the generative model (and/or future improved versions). The generative model would have to be released under a non-commercial license due to our datasets.

This project is a work in progress, but below are samples from our model, followed by a more detailed methodology. These samples are cherry-picked (in general, best-of-generally-between-4-and-16-samples). We hope to eventually improve our model so that our outputs are more consistently high-quality.

Demo

Unconditional speech generation

Parakeet is able to generate conversational audio (right column) given a text prompt (left column). As seen in text prompts below, [S1], [S2], etc., are used to denote different speakers. Non-verbal events such as “laughs” can be specified in parentheses.

Audio prompting / speaker cloning

Our model is able to perform zero-shot voice cloning simply by prefilling the decoder with an audio prompt and setting the text condition to be the concatenation of text from the audio prompt and the desired text.

In the below table for each sample, the audio column contains the audio prompt and the full audio continuation (including the prompt).

Dataset Methodology

Our dataset is comprised of three different sources, Spotify Podcast Dataset, LibriVox, and Common Voice.

Spotify Dataset and WhisperD

Spotify Podcast Dataset consists of over 100,000 podcast episodes totaling approximately 60,000 hours of audio (the dataset is no longer maintained by Spotify as of December 2023). Though it provides transcriptions, they are machine-generated (and not of great quality) and unsuitable for training a generative model. We first split the podcast into segments of up to 30 seconds in length using pyannote and obtain approximately 9,000,000 segments.

Ideally, our transcriptions would contain both speaker labels as well as events (i.e. “[S1] Hey! [S2] (sighs) Um, how’s it going?”). We initially tried prompting Whisper to elicit this style of transcription but found the model struggled to do so. We thus fine-tune Whisper-v2-large on a small dataset of (audio, ideal-annotation) pairs. We call this model WhisperD (for dialogue).

The initial version of WhisperD was created through a two-stage fine-tuning process, where we first fine-tune on lower-quality automatic transcriptions and then fine-tune again on higher-quality human data. Our reasoning was that human-annotated transcriptions are expensive to obtain, and that a larger dataset of automatic transcriptions could be inexpensively used to roughly calibrate Whisper to perform speaker annotation before a higher-quality fine-tuning pass. We used online transcription services for both the automatic and human transcriptions to transcribe random subsets of our podcast data. The automatic portion consisted of ~20 hours, while the human portion consisted of slightly under 2 hours.

After the fine-tuning process, we’re able to generate transcriptions such as:

We use this model to generate transcriptions for all of the VAD-split podcast segments. We also trained what we hoped would be a better version of WhisperD with a more sophisticated method involving bootstrapping, though meaningful evaluation turned out to be challenging (manual inspection seemed to be more informative than many of the eval methods we devised). We also generate transcriptions using this newer WhisperD and randomly sample which transcription to use when training our generative model later.

Lastly, we train two additional WhisperD models and use them to transcribe subsets of our dataset:

1. Fluent WhisperD: It may be unintuitive for users to write convincing-sounding disfluencies into text prompts. If we train our model (a percentage of the time) with “fluentized” text (accompanied with a tag), the model will learn to optionally generate disfluencies even when none are present in the text prompt. This functionality also may be useful in cases where one wants to fine-tune on a single speaker (i.e. train on fluentized text only and allow the modelto learn disfluency patterns) or generate disfluent audio continuations given fluent text. We add an “[F]” tag to the start of fluentized text prompts.
2. Fuzzy WhisperD: One possible issue with synthetic transcriptions is that if the transcriptions all have the same style, our generative model may not be robust to user input. We thus use GPT to generate stylistically-varied versions of a set of transcriptions, and then fine-tune Whisper on these “fuzzied” transcriptions. Though one could argue the fuzzying could be done by a text-only model, 1) using a Whisper model was practical / convenient given our pipeline and 2) it’s theoretically possible (albeit practically unlikely) that audio-aware fuzzing may provide benefits.

During generative training later, with some relatively small probability, transcriptions are replaced with either fluent or fuzzy versions.

LibriVox Dataset

LibriVox is an archive of public domain audiobooks. We take ~30,000 hours, and we attempt to balance speaker durations during sampling. We again perform VAD to split into 30 second (maximum) segments and use Whisper-v2-medium to generate transcriptions. Interestingly, we find that the pretrained Whisper models omit text such as “This is a LibriVox recording…,” (this is likely due to Whisper training on these samples and using ground-truth transcriptions that only contain book text) so we very briefly fine-tune Whisper-medium with examples containing these sorts of phrases.

Common Voice Dataset

We also use the English subset of Common Voice 14 . It contains ~3,300 hours of audio and has high speaker diversity, containing over 85,000 unique speakers. Since the median segment length is short (< 10 seconds), we sometimes concatenate segments (randomly choosing from 1 to 5 segments per sample) from the same speaker. We use the provided transcriptions, and when we concatenate audio, we concatenate text with the vertical bar “|”. Though zero-shot speaker cloning can be naively achieved through audio continuation, it’s possible including these concatenated segments in our train dataset will facilitate more in-distribution cloning during inference.

Modeling Methodology

DAC Prediction

Rather than predicting raw audio directly, we instead predict latent audio codes / tokens, using the pre-trained Descript-Audio-Codec (DAC) as our autoencoder. DAC is a residual-quantized VAE with 9 residual codebooks, each with vocab-size 1024, and processes 44,100 kHz monophonic audio with a time-wise downsampling rate of 512. Thus, a second of 44,100 kHz audio will be encoded into ~86 x 9 integer codes ranging from 0-1023. Note that DAC is trained with random dropout of higher-level codebooks, so lower-level codebooks tend to contain most of the semantic information. We process the entirety of our dataset with DAC and cache the corresponding token sequences.

We now train a model to autoregressively predict these DAC tokens, conditioned on raw text. In the model featured in this blog post, we use an encoder-decoder transformer, though we’ve also experimented with decoder-only models.

Given that there are 9 residual codebooks, we need to make a decision on how to predict the 86 x 9 (= 774) codes per second. Though flattening would be an option, this would leave us with 30-second sequence lengths of 23,220 (the ratio of attention flops to MLP flops would be high with a vanilla setup given relatively small model size), and the model might be inclined to spend more-than-optimal compute on predicting less important residual codebooks (though this depends on training dynamics, as it’s possible the model will learn to “think ahead” in the residual streams of less-important codebook levels).

There are two more practical options with which we experimented.

1. Hierarchical-transformer: Similar to Hourglass Transformers and MegaByte , as well as UniAudio , who concurrently trained a text-to-audio model using such an approach. The core idea is to have a big transformer whose input is the full residual stack, and a small transformer (applied at every timestep) that takes in the big transformer’s final hidden state and predicts the 9 residual codes. The benefit of this approach is that it is maximally expressive (it can model any distribution of codes) but the downside is its architectural complexity.
2. Predicting codes jointly / delay-pattern prediction: Naively, one can jointly predict all 9 residual codes at each timestep (i.e. 9 separate linear layers (or MLPs) projecting the final hidden state to each residual code logit vector). However, this would assume that all 9 codes for a particular timestep are conditionally independent (given previous timesteps), which is likely an unreasonable assumption. A clever method (from Kharitonov et al. and adapted in MusicGen is to manufacture a more favorable conditional independence assumption by shifting higher level codebooks rightward. See the below figure, or the description in MusicGen. Rather than use a uniform delay for all codebooks, we use a more significant delay for the second residual level. Specifically, we use a delay pattern of \((8, 9, 10, \ldots, 15)\). We also apply a 4x loss weight on the first level predictions.

We found that delay pattern prediction seemed to match hierarchical transformer setups in performance. We also tried combining delay-pattern with hierarchical prediction, which we found improved results over hierarchical prediction without any codebook delay but still was not obviously superior to the non-hierarchical approach. Given this, we choose to train a non-hierarchical model with a delay pattern. Note that if the conditional independence assumed by the delay-pattern proves to be a bottleneck in terms of quality, we could convert our model to a hierarchical one by training a small transformer to replace the linear output projections.

Training Details

We train a 3B parameter encoder-decoder transformer model. The encoder’s input is raw bytes of text, which closely corresponds to English characters, as the majority of our data is English aside from a small portion of podcast data.

The encoder (text) sequence length is 768, while the decoder (DAC token) length is 2048. To enable classifier-free guidance (see later section) at inference, we drop out the text condition for 15% of samples during training.

The encoder has model size 1536 with 12 layers and 16 heads. The decoder has model size 2560 with 32 layers and uses GQA, with 32 query heads and 8 KV heads. The model uses rotary positional embeddings in self-attention modules and SwiGLU activation. We train our model with batch size 256 for 110,000 steps with a cosine learning rate decay, with peak learning rate 2e-4. We train on a v3-256 TPU pod, provided graciously by TPU research cloud, implementing parallel training in JAX / FLAX . We train with data-parallelism across all devices and shard the model and optimizer via intra-host FSDP.

After training, we fine-tune the model on a small higher-quality subset of our dataset (we filter using PESQ ). Since the cosine schedule had not fully annealed by 110,000 steps, this approach is slightly analogous to MiniCPM . We found this emperically helps with generating higher quality unconditional samples. Since the training of this model, we’ve also explored adding high-quality tags to text conditions and/or performing dataset filtering. One (admittedly hacky) trick to encourage better unconditional generations would be to omit loss terms for the first $n$ seconds of poor quality samples.

Classifier-Free Guidance

First introduced in diffusion models , classifier-free guidance (CFG) has also been adapted to text-to-something autoregressive models .

CFG in autoregressive models involves training both a conditional model \(P_c(x_t \vert x_{1,\ldots,t-1}, c)\), where c represents the text condition, and an unconditional model \(P_u(x_t \vert x_{1,\ldots,t-1})\). In practice the unconditional model is learned by dropping out the text condition for a small portion of batch samples during training. Then, when sampling a particular code, given conditional logits \(l_c\) and unconditional logits \(l_u\), the final logits \(l_{cfg}\) are obtained with:

where \(\alpha\) is a hyperparameter scalar controlling the degree of guidance. After CFG, we can apply top-k or top-p sampling; we choose top-k with k = 50.

When we apply CFG to Parakeet sampling, quality is significantly improved. However, on inspecting generations, there tends to be a dramatic speed-up over the duration of the sample (i.e. the rate of speaking increases significantly over time). Our intuition for this problem is as follows: Say that is our model is (at some level) predicting phonemes and the ground truth distribution for the next phoneme occuring is 25% at a given timestep. Our conditional model may predict 20%, but because our uncondtional model cannot see the text transcription, its prediction for the correct next phoneme will be much lower, say 5%. With a reasonable level of CFG, because \((l_c - l_u)\) will be large for the correct next phoneme, we’ll obtain a much higher final probability, say 50%, which biases our generation towards faster speech. And this effect compounds! After an initial speed-up, a good model would now “think” it’s predicting for a fast speaker, so the conditional model will predict even higher probabilities for the next phoneme occuring soon, leading to an even larger bias after CFG is applied.

Here is an audio sample with CFG (with \(\alpha = 3\)), and for reference a sample without any CFG.

The quality and text-alignment of the CFG sample is higher (and the difference is often larger than the above examples), but the speed-up is a significant issue.

CFG-filter

To address this, we introduce CFG-filter, a modification to CFG that mitigates the speed drift. The idea is to first apply the CFG calculation to obtain a new set of logits \(l_{cfg}\) as before, but rather than use these logits to sample, we use these logits to obtain a top-k mask to apply to our original conditional logits. Intuitively, this serves to constrict the space of possible “phonemes” to text-aligned phonemes without heavily biasing the relative probabilities of these phonemes (or for example, start next word vs pause more).

Here is the same text prompt with CFG-filter:

The pseudocode is as follows:

  def mask_by_top_k(l_c: jax.Array, cfg_logits: jax.Array, k: int) -> jax.Array:
    # returns x where x[i] = l_c[i] if cfg_logits[i] in top k elements of cfg_logits else -inf
  
  def cfg_filter(l_c: jax.Array, l_u: jax.Array, alpha: float, k: int):
    cfg_logits = l_c + alpha * (l_c - l_u)
    sample_logits = mask_by_top_k(l_c, cfg_logits, k)
    return sample_logits

Note that we can optionally reapply a smaller level of CFG after masking (i.e. rather than returning sample logits as is, we could set \(\text{new_sample_logits} = \text{sample_logits} + \beta (\text{sample_logits} - l_u)\), where intuitively we should have $\beta < \alpha$. We can also again apply top-k or top-p sampling before returning the final sample_logits.

Future Work

Improving Autoregressive Parakeet

A reasonable next step for improving autoregressive versions of Parakeet would be to test replacing DAC with a different autoencoder. It’s possible that given DAC’s relatively high number of codes per second (86 base codes per second, and 774 codes including residual levels), it may not be particularly conducive to autoregressive generation. Training a new RQ-VAE with some combination of a higher downsampling rate and less residual levels (with a possibly larger vocab size) would be a sensible starting point. It also might be exploring FSQ as a potential quantization method.

Though higher autoencoder compression might result in worse decoder reconstruction quality, one could train a more powerful decoder (for example, diffusion) to mitigate this. However, a caveat is that ideally the latents / codes should contain most of the perceptible audio information, as leaving decision making / ambiguitity to the decoder might complicate audio prompt continuation.

Diffusion

Some of our more recent work has involved transitioning from autoregression to diffusion. A downside of the current autoregression approach is that mistakes in the sampling process cannot be corrected. Diffusion allows (hand-wavingly) for such correction, but a vanilla approach would involve generating 30 second segments regardless of text prompt length or training a duration predictor, both of which have disadvantages. Additionally, for text-to-speech, the approach of generating entire long segments at once may have theoretical downsides having to do with a certain time-wise asymmetry (e.g. it’s easier to predict the phoneme occuring at the 2-second-mark than the 28-second-mark); a more thorough exploration is left to future blog posts.

Some of our work on diffusion involved splitting audio latent segments into “blocks” to allow for autoregressive sampling of blocks, and concurrently Aran Komatsuzaki proposed a particularly elegant block-wise approach , explained here. I think this approach could be especially practical for longer generations, e.g. video. A limitation of this method is that it complicates solving inverse problems relative to single-block / vanilla diffusion; however, in preliminary experiments, we’ve found that it’s possible to adapt a diffusion model trained on small blocks to generate longer sequences via fine-tuning.

We hope to make more progress on this and share our work in the future.

Fully-Streaming TTS (Audio Turing Test)

While Parakeet could be adopted for AI-Human conversational TTS, a much more important project would involve a fully-streaming TTS, a project to which I have given some thought since the development of our initial Parakeet model. Rather than current conversational TTS systems, which involve a human talking, the AI waiting for the human to finish talking, the AI responding, etc., a fully-streaming TTS would involve a constant stream of AI audio output, which could include laughter, interruptions, etc. A well-performing streaming TTS model might be able to pass the audio version of a Turing test: a human might not be able to tell whether they’re speaking to another human or an AI system. Though it would require significant engineering work, I believe this project is relatively low-hanging fruit (with a high ratio of impressiveness to engineering capital). I’ve been thinking about different approaches and am excited to see what the future holds in this area.

Bibtex

To cite this blog post, please use:

bibtex @misc{darefsky2024parakeet, author = {Darefsky, Jordan and Zhu, Ge and Duan, Zhiyao}, title = {Parakeet}, year = {2024}, url = {https://jordandarefsky.com/blog/2024/parakeet/} }