Latent Spectral Equivariance

Anonymous Authors

TL;DR Structure audio VAE latents along time and frequency with temporal scale equivariance and spectral granularity equivariance, improving diffusion modeling without changing the architecture or adding loss terms.

Overview of LatentEQ: matched temporal and spectral transformations connect latent representations to target spectrograms.
Figure 1. Overview of LatentEQ. Temporal scale equivariance and spectral granularity equivariance pair coarsenings of the latent along its temporal and channel axes with matched coarsenings of the target spectrogram.

Abstract

Scalable audio generation systems typically cascade a pretrained variational autoencoder (VAE) for audio compression and a latent diffusion model for distribution modeling. However, existing VAEs prioritize reconstruction fidelity without explicitly considering how readily the resulting latent distribution can be modeled by diffusion, potentially limiting generation quality. Recent work highlights the importance of aligning latent representations with the structure of the underlying data for diffusion modeling.

Building on this direction, we introduce Latent Spectral Equivariance (LatentEQ), a framework that introduces equivariance into audio VAE optimization by pairing latent transformations with corresponding spectrogram transformations. Specifically, temporal scale equivariance applies matched low-pass filtering along time, aligning slowly varying latent components with slowly varying acoustic structure. Spectral granularity equivariance pairs latent channel prefixes with frequency-smoothed spectrograms, organizing channels to progressively capture spectral envelopes and fine harmonic detail.

We instantiate LatentEQ on the VAEs of two well-known audio generation systems, Stable Audio Open and MMAudio, through fine-tuning without changing their architecture or adding loss terms. Comprehensive results demonstrate improved generation quality across waveform- and spectrogram-based VAEs and both text-to-audio and video-to-audio tasks, highlighting LatentEQ as an effective principle for learning structured audio latents.

Key Contributions

  1. Temporal Scale
    Equivariance

    Pairs temporal low-pass filtering of the latent with the same coarsening of the target spectrogram, connecting slow latent variation to slowly varying acoustic structure.

  2. Spectral Granularity
    Equivariance

    Pairs a prefix of latent channels with spectral smoothing of the target, so leading channels capture broad spectral envelopes and later channels progressively add fine detail.

  3. One Principle Across
    Audio Generation Systems

    Fine-tunes waveform and spectrogram VAEs without changing their architecture or adding loss terms. The frozen VAE then provides structured latents for standard diffusion training.

Text-to-Audio

AudioCaps

01.

Water trickles as birds chirp then drums begin to play

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 1 mel AudioLDM, AudioCaps example 1 mel Tango, AudioCaps example 1 mel MMAudio, AudioCaps example 1 mel LatentEQ (Spectrogram), AudioCaps example 1 mel TangoFlux, AudioCaps example 1 mel AudioX, AudioCaps example 1 mel Stable Audio Open, AudioCaps example 1 mel LatentEQ (Waveform), AudioCaps example 1 mel
Audio

02.

A quick repeat of applause followed by continued applause

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 2 mel AudioLDM, AudioCaps example 2 mel Tango, AudioCaps example 2 mel MMAudio, AudioCaps example 2 mel LatentEQ (Spectrogram), AudioCaps example 2 mel TangoFlux, AudioCaps example 2 mel AudioX, AudioCaps example 2 mel Stable Audio Open, AudioCaps example 2 mel LatentEQ (Waveform), AudioCaps example 2 mel
Audio

03.

A train horn blows as a train approaches with warning bells ringing

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 3 mel AudioLDM, AudioCaps example 3 mel Tango, AudioCaps example 3 mel MMAudio, AudioCaps example 3 mel LatentEQ (Spectrogram), AudioCaps example 3 mel TangoFlux, AudioCaps example 3 mel AudioX, AudioCaps example 3 mel Stable Audio Open, AudioCaps example 3 mel LatentEQ (Waveform), AudioCaps example 3 mel
Audio

04.

An infant and woman laughing as water splashes and birds chirp in the background

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 4 mel AudioLDM, AudioCaps example 4 mel Tango, AudioCaps example 4 mel MMAudio, AudioCaps example 4 mel LatentEQ (Spectrogram), AudioCaps example 4 mel TangoFlux, AudioCaps example 4 mel AudioX, AudioCaps example 4 mel Stable Audio Open, AudioCaps example 4 mel LatentEQ (Waveform), AudioCaps example 4 mel
Audio

05.

Tires squealing followed by an engine revving

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 5 mel AudioLDM, AudioCaps example 5 mel Tango, AudioCaps example 5 mel MMAudio, AudioCaps example 5 mel LatentEQ (Spectrogram), AudioCaps example 5 mel TangoFlux, AudioCaps example 5 mel AudioX, AudioCaps example 5 mel Stable Audio Open, AudioCaps example 5 mel LatentEQ (Waveform), AudioCaps example 5 mel
Audio

06.

Multiple basketballs bouncing on a hard surface and shoes squeaking as a man shouts in the distance

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 6 mel AudioLDM, AudioCaps example 6 mel Tango, AudioCaps example 6 mel MMAudio, AudioCaps example 6 mel LatentEQ (Spectrogram), AudioCaps example 6 mel TangoFlux, AudioCaps example 6 mel AudioX, AudioCaps example 6 mel Stable Audio Open, AudioCaps example 6 mel LatentEQ (Waveform), AudioCaps example 6 mel
Audio

07.

Sirens ring and a vehicle speeds closer

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 7 mel AudioLDM, AudioCaps example 7 mel Tango, AudioCaps example 7 mel MMAudio, AudioCaps example 7 mel LatentEQ (Spectrogram), AudioCaps example 7 mel TangoFlux, AudioCaps example 7 mel AudioX, AudioCaps example 7 mel Stable Audio Open, AudioCaps example 7 mel LatentEQ (Waveform), AudioCaps example 7 mel
Audio

08.

Bells chime and ring

Ground Truth AudioLDM Tango MMAudio LatentEQ (Spectrogram) TangoFlux AudioX Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, AudioCaps example 8 mel AudioLDM, AudioCaps example 8 mel Tango, AudioCaps example 8 mel MMAudio, AudioCaps example 8 mel LatentEQ (Spectrogram), AudioCaps example 8 mel TangoFlux, AudioCaps example 8 mel AudioX, AudioCaps example 8 mel Stable Audio Open, AudioCaps example 8 mel LatentEQ (Waveform), AudioCaps example 8 mel
Audio

Video-to-Audio

VGGSound

01.

Ground TruthV-AURAFrierenAudioXMMAudioLatentEQ (Spectrogram)
MelGround Truth MelV-AURA MelFrieren MelAudioX MelMMAudio MelLatentEQ (Spectrogram) Mel
Video

02.

Ground TruthV-AURAFrierenAudioXMMAudioLatentEQ (Spectrogram)
MelGround Truth MelV-AURA MelFrieren MelAudioX MelMMAudio MelLatentEQ (Spectrogram) Mel
Video

03.

Ground TruthV-AURAFrierenAudioXMMAudioLatentEQ (Spectrogram)
MelGround Truth MelV-AURA MelFrieren MelAudioX MelMMAudio MelLatentEQ (Spectrogram) Mel
Video

04.

Ground TruthV-AURAFrierenAudioXMMAudioLatentEQ (Spectrogram)
MelGround Truth MelV-AURA MelFrieren MelAudioX MelMMAudio MelLatentEQ (Spectrogram) Mel
Video

05.

Ground TruthV-AURAFrierenAudioXMMAudioLatentEQ (Spectrogram)
MelGround Truth MelV-AURA MelFrieren MelAudioX MelMMAudio MelLatentEQ (Spectrogram) Mel
Video

06.

Ground TruthV-AURAFrierenAudioXMMAudioLatentEQ (Spectrogram)
MelGround Truth MelV-AURA MelFrieren MelAudioX MelMMAudio MelLatentEQ (Spectrogram) Mel
Video

Video-and-Text-to-Audio

VGGSound-Omni

01.

Electronic dance music with a funky bassline and steady beat plays as a DJ rhythmically scratches and stutters a male vocal sample, creating a percussive effect.

Ground TruthHunyuanVideo FoleyAudioXMMAudioThinkSoundLatentEQ (Spectrogram)
MelGround Truth MelHunyuanVideo Foley MelAudioX MelMMAudio MelThinkSound MelLatentEQ (Spectrogram) Mel
Video

02.

A man's declarative voice speaks in a South Asian language, accompanied by the persistent background gobbling and cooing of a flock of turkeys.

Ground TruthHunyuanVideo FoleyAudioXMMAudioThinkSoundLatentEQ (Spectrogram)
MelGround Truth MelHunyuanVideo Foley MelAudioX MelMMAudio MelThinkSound MelLatentEQ (Spectrogram) Mel
Video

03.

A male speaks excitedly in Japanese over dramatic orchestral battle music and video game combat sounds, including whooshes and a creature's roar.

Ground TruthHunyuanVideo FoleyAudioXMMAudioThinkSoundLatentEQ (Spectrogram)
MelGround Truth MelHunyuanVideo Foley MelAudioX MelMMAudio MelThinkSound MelLatentEQ (Spectrogram) Mel
Video

04.

A single electronic tone and mechanical clicks precede a whirring sound. A rapid series of loud explosions follows, ending with a deafening metallic crash and a final click.

Ground TruthHunyuanVideo FoleyAudioXMMAudioThinkSoundLatentEQ (Spectrogram)
MelGround Truth MelHunyuanVideo Foley MelAudioX MelMMAudio MelThinkSound MelLatentEQ (Spectrogram) Mel
Video

05.

Upbeat dance music plays as children joyfully chant in unison, accompanied by rhythmic clapping and the steady thud of a jump rope hitting the floor.

Ground TruthHunyuanVideo FoleyAudioXMMAudioThinkSoundLatentEQ (Spectrogram)
MelGround Truth MelHunyuanVideo Foley MelAudioX MelMMAudio MelThinkSound MelLatentEQ (Spectrogram) Mel
Video

06.

A fire truck's piercing siren wails continuously, accompanied by a blaring horn. A male voice speaks briefly over the roaring engine as the truck speeds by.

Ground TruthHunyuanVideo FoleyAudioXMMAudioThinkSoundLatentEQ (Spectrogram)
MelGround Truth MelHunyuanVideo Foley MelAudioX MelMMAudio MelThinkSound MelLatentEQ (Spectrogram) Mel
Video

VAE Reconstruction

Mel-spectrograms and audio of the same 10-second input reconstructed by each VAE.

Sound

01.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Sound example 1 mel AudioLDM, Sound example 1 mel Make-An-Audio 2, Sound example 1 mel MMAudio, Sound example 1 mel LatentEQ (Spectrogram), Sound example 1 mel HunyuanVideo Foley, Sound example 1 mel εar-VAE, Sound example 1 mel Stable Audio Open, Sound example 1 mel LatentEQ (Waveform), Sound example 1 mel
Audio

02.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Sound example 2 mel AudioLDM, Sound example 2 mel Make-An-Audio 2, Sound example 2 mel MMAudio, Sound example 2 mel LatentEQ (Spectrogram), Sound example 2 mel HunyuanVideo Foley, Sound example 2 mel εar-VAE, Sound example 2 mel Stable Audio Open, Sound example 2 mel LatentEQ (Waveform), Sound example 2 mel
Audio

03.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Sound example 3 mel AudioLDM, Sound example 3 mel Make-An-Audio 2, Sound example 3 mel MMAudio, Sound example 3 mel LatentEQ (Spectrogram), Sound example 3 mel HunyuanVideo Foley, Sound example 3 mel εar-VAE, Sound example 3 mel Stable Audio Open, Sound example 3 mel LatentEQ (Waveform), Sound example 3 mel
Audio

04.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Sound example 4 mel AudioLDM, Sound example 4 mel Make-An-Audio 2, Sound example 4 mel MMAudio, Sound example 4 mel LatentEQ (Spectrogram), Sound example 4 mel HunyuanVideo Foley, Sound example 4 mel εar-VAE, Sound example 4 mel Stable Audio Open, Sound example 4 mel LatentEQ (Waveform), Sound example 4 mel
Audio

05.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Sound example 5 mel AudioLDM, Sound example 5 mel Make-An-Audio 2, Sound example 5 mel MMAudio, Sound example 5 mel LatentEQ (Spectrogram), Sound example 5 mel HunyuanVideo Foley, Sound example 5 mel εar-VAE, Sound example 5 mel Stable Audio Open, Sound example 5 mel LatentEQ (Waveform), Sound example 5 mel
Audio

Music

01.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Music example 1 mel AudioLDM, Music example 1 mel Make-An-Audio 2, Music example 1 mel MMAudio, Music example 1 mel LatentEQ (Spectrogram), Music example 1 mel HunyuanVideo Foley, Music example 1 mel εar-VAE, Music example 1 mel Stable Audio Open, Music example 1 mel LatentEQ (Waveform), Music example 1 mel
Audio

02.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Music example 2 mel AudioLDM, Music example 2 mel Make-An-Audio 2, Music example 2 mel MMAudio, Music example 2 mel LatentEQ (Spectrogram), Music example 2 mel HunyuanVideo Foley, Music example 2 mel εar-VAE, Music example 2 mel Stable Audio Open, Music example 2 mel LatentEQ (Waveform), Music example 2 mel
Audio

03.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Music example 3 mel AudioLDM, Music example 3 mel Make-An-Audio 2, Music example 3 mel MMAudio, Music example 3 mel LatentEQ (Spectrogram), Music example 3 mel HunyuanVideo Foley, Music example 3 mel εar-VAE, Music example 3 mel Stable Audio Open, Music example 3 mel LatentEQ (Waveform), Music example 3 mel
Audio

04.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Music example 4 mel AudioLDM, Music example 4 mel Make-An-Audio 2, Music example 4 mel MMAudio, Music example 4 mel LatentEQ (Spectrogram), Music example 4 mel HunyuanVideo Foley, Music example 4 mel εar-VAE, Music example 4 mel Stable Audio Open, Music example 4 mel LatentEQ (Waveform), Music example 4 mel
Audio

05.

Ground Truth AudioLDM Make-An-Audio 2 MMAudio LatentEQ (Spectrogram) HunyuanVideo Foley εar-VAE Stable Audio Open LatentEQ (Waveform)
Mel Ground Truth, Music example 5 mel AudioLDM, Music example 5 mel Make-An-Audio 2, Music example 5 mel MMAudio, Music example 5 mel LatentEQ (Spectrogram), Music example 5 mel HunyuanVideo Foley, Music example 5 mel εar-VAE, Music example 5 mel Stable Audio Open, Music example 5 mel LatentEQ (Waveform), Music example 5 mel
Audio

Figure 2, Enlarged

Temporal scale equivariance. Top row: the target spectrogram and its versions low-pass filtered at 9.4, 5.4 and 2.7 Hz. Lower rows: spectrograms decoded from the latent filtered at the same cutoffs, for the SAO VAE and for LatentEQ, with the mean log-magnitude per frame overlaid. Spectral granularity equivariance. Top row: the target spectrogram and its versions smoothed to the granularity matched to 48, 32 and 16 latent channels. Lower rows: spectrograms decoded from the first 48, 32 and 16 latent channels, for the SAO VAE and for LatentEQ.
Figure 2. Qualitative comparison of the pretrained SAO VAE and its LatentEQ counterpart under temporal low-pass filtering (first panel) and prefix-channel truncation (second panel), where the top row shows the original and matched coarsened spectrogram targets and the lower rows show the decoded outputs. The curves overlaid in the first panel trace the mean log-magnitude per frame.