Codec SUPERB

Abstract: The neural audio codec has attracted increasing interest as a highly effective method for audio compression and representation. By transforming continuous audio into discrete tokens, it facilitates the use of large language modelling (LLM) techniques in audio processing. In this talk, we will report recent progress in neural audio codecs, with a particular focus on SemantiCodec, a new neural audio codec for ultra-low bit rate audio compression and tokenization. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised AudioMAE, discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are then used to reconstruct audio via a diffusion-model-based decoder. SemantiCodec presents several advantages over previous codecs, which typically operate at high bitrates, are confined to narrow domains like speech, and lack the semantic information essential for effective language modelling. First, SemantiCodec compresses audio into fewer than 100 tokens per second across various audio types, including speech, general audio, and music, while maintaining high-quality output. Second, it preserves substantially richer semantic information from audio compared to all evaluated codecs. We will illustrate these benefits through benchmarking and conclude by discussing potential directions for future research in this field.

Bio: Wenwu Wang is a Professor in Signal Processing and Machine Learning, University of Surrey, UK. He is also an AI Fellow at the Surrey Institute for People Centred Artificial Intelligence. His current research interests include signal processing, machine learning and perception, artificial intelligence, machine audition (listening), and statistical anomaly detection. He has (co)-authored over 300 papers in these areas. He has been recognized as a (co-)author or (co)-recipient of more than 15 accolades, including the 2022 IEEE Signal Processing Society Young Author Best Paper Award, ICAUS 2021 Best Paper Award, DCASE 2020 and 2023 Judge’s Award, DCASE 2019 and 2020 Reproducible System Award, and LVA/ICA 2018 Best Student Paper Award. He is an Associate Editor (2020-2025) for IEEE/ACM Transactions on Audio Speech and Language Processing, and an Associate Editor (2024-2026) for IEEE Transactions on Multimedia. He was a Senior Area Editor (2019-2023) and Associate Editor (2014-2018) for IEEE Transactions on Signal Processing. He is the elected Chair (2023-2024) of IEEE Signal Processing Society (SPS) Machine Learning for Signal Processing Technical Committee, a Board Member (2023-2024) of IEEE SPS Technical Directions Board, the elected Chair (2025-2027) and Vice Chair (2022-2024) of the EURASIP Technical Area Committee on Acoustic Speech and Music Signal Processing, an elected Member (2021-2026) of the IEEE SPS Signal Processing Theory and Methods Technical Committee. He has been on the organising committee of INTERSPEECH 2022, IEEE ICASSP 2019 & 2024, IEEE MLSP 2013 & 2024, and SSP 2009. He is Technical Program Co-Chair of IEEE MLSP 2025. He has been an invited Keynote or Plenary Speaker on more than 20 international conferences and workshops.

Abstract: Neural speech codecs promise high-quality speech at low bitrates but face challenges like increased model complexity and suboptimal quality. This talk presents two approaches to address these issues: generative de-quantization and personalization. With LaDiffCodec, we propose separating representation learning from information reconstruction. It combines a typical end-to-end codec that learns low-dimensional discrete tokens for compact representation and a latent diffusion model that de-quantizes these tokens into high-dimensional continuous space in a generative fashion. To prevent over-smooth speech, we employ “midway-infilling” during diffusion. Subjective tests show our model improves popular neural speech codecs' performance. Next, we introduce personalized neural speech codecs, where we explore personalizing codecs to specific user groups to reduce complexity and enhance perceptual quality. By learning speaker embeddings with a Siamese network from the LibriSpeech dataset, we cluster speakers based on perceptual similarity. Subjective tests reveal this strategy enables model compression without sacrificing—and even improving—speech quality. By decoupling key tasks and introducing personalization, these approaches address current limitations and pave the way for superior speech quality and efficiency at low bitrates in neural speech codecs.

Bio: Minje Kim is an associate professor in the Siebel School of Computing and Data Science at the University of Illinois at Urbana-Champaign and a Visiting Academic at Amazon Lab126. Before then, he was an associate professor at Indiana University (2016-2023). He earned his Ph.D. in Computer Science at UIUC (2016) after working as a researcher at ETRI, a national lab in Korea (2006 to 2011). During his career, he has focused on developing machine learning models for audio signal processing applications. He is a recipient of various awards, including the NSF Career Award (2021), IU Trustees Teaching Award (2021), IEEE SPS Best Paper Award (2020), Google and Starkey’s grants for outstanding student papers in ICASSP 2013 and 2014, respectively. He is an IEEE Senior Member and the Vice Chair of the IEEE Audio and Acoustic Signal Processing Technical Committee. He is serving on the editorial boards for IEEE/ACM T-ALSP and IEEE SPL as Senior Area Editor, EURASIP JASMP (Associate Editor), and IEEE OJSP (Consulting Associate Editor). He was the General Chair of IEEE WASPAA 2023 and also a reviewer, program committee member, or area chair for the major machine learning and signal processing venues. He is on more than 50 patents as an inventor.

Abstract: Building a universal audio foundation model for different audio generation tasks, such as text-to-speech, text-to-audio, singing voice synthesis, voice conversion, and speech dialogue, has attract great interest in the audio community. Audio codec and audio modeling strategies are two key points. In this talk, we review the development of audio codec and different audio modeling strategies in the literature, and show our recent works in audio codec and audio modeling methods. Lastly, we summarize the challenges and potential directions in development universal audio foundation model.

Bio: Dongchao Yang is a second-year Ph.D. student at The Chinese University of Hong Kong, supervised by Prof. Helen Meng. Before that, he received his master's degree from Peking University. His research interests encompass the extensive domain of speech and language intelligence, which includes audio foundation models, multi-modal large language models (MLLMs), text-to-speech synthesis (TTS), cross-modal representation learning, among other related areas. Currently, his work focuses on audio-text foundation models and audio codec models.

Abstract: We introduce VoiceCraft, a token infilling neural codec language model (NCLM), that achieves state-of-the-art (SOTA) performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts. We study the case of developing VoiceCraft, to understand how NCLM works in solving critical speech generation problems, how codec sets the foundation of SOTA NCLM, and the challenges in the development and evaluation.

VoiceCraft leverages Encodec to tokenize speech signals; it employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable token generation within an existing sequence. On speech editing tasks, VoiceCraft produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SOTA models including VALLE and XTTS-v2. The models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings. In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named RealEdit. We encourage audience to listen to the demos at this website.

Bio: Shang-Wen Li is a Research Lead and Manager at Meta’s Fundamental AI Research (FAIR) team, and he worked at Apple Siri, Amazon Alexa and AWS before joining FAIR. He completed his PhD in 2016 at MIT in the Spoken Language Systems group of Computer Science and Artificial Intelligence Laboratory (CSAIL). His recent research is focused on multimodal large language models, multimodal representation learning, and spoken language understanding.

Abstract: Audio analysis and audio synthesis require modeling long-term, complex phenomena and have historically been tackled in an asymmetric fashion, with specific analysis models that differ from their synthesis counterpart. In this presentation, we will introduce the concept of audio language models, a recent innovation aimed at overcoming these limitations. By discretizing audio signals using a neural audio codec, we can frame both audio generation and audio comprehension as similar autoregressive sequence-to-sequence tasks, capitalizing on the well-established Transformer architecture commonly used in language modeling. This approach unlocks novel capabilities in areas such as textless speech modeling, zero-shot voice conversion, text-to-music generation and even real-time spoken dialogue. Furthermore, we will illustrate how the integration of analysis and synthesis within a single model enables the creation of versatile audio models capable of handling a wide range of tasks involving audio as inputs or outputs. We will conclude by highlighting the promising prospects offered by these models and discussing the key challenges that lie ahead in their development.

Bio: Neil is co-founder and Chief Modeling Officer of the Kyutai non-profit research lab. Neil and team have recently presented Moshi (https://moshi.chat), the first real-time spoken dialogue model. He was previously at Google DeepMind, where he started and led a team working on generative audio, with contributions including Google’s first text-to-music API, a voice preserving speech-to-speech translation system, and the first neural audio codec that outperforms general-purpose audio codecs. Before that, Neil spent three years at Facebook AI Research, working on automatic speech recognition and audio understanding. He graduated with a PhD in machine learning from Ecole Normale Supérieure (Paris), and holds an MSc in machine learning from Ecole Normale Supérieure (Saclay) and an MSc in quantitative finance from Université Paris Dauphine. In parallel with his research activities, Neil teaches speech processing technologies at the École Normale Supérieure (Saclay).