Speech & Audio

Audio source separation, recognition, and understanding.

Our current research focuses on application of machine learning to estimation and inference problems in speech and audio processing. Topics include end-to-end speech recognition and enhancement, acoustic modeling and analysis, statistical dialog systems, as well as natural language understanding and adaptive multimodal interfaces.

  • Quick Links

  • Researchers

  • Awards

    •  AWARD    MERL team wins the Listener Acoustic Personalisation (LAP) 2024 Challenge
      Date: August 29, 2024
      Awarded to: Yoshiki Masuyama, Gordon Wichern, Francois G. Germain, Christopher Ick, and Jonathan Le Roux
      MERL Contacts: François Germain; Jonathan Le Roux; Gordon Wichern; Yoshiki Masuyama
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL's Speech & Audio team ranked 1st out of 7 teams in Task 2 of the 1st SONICOM Listener Acoustic Personalisation (LAP) Challenge, which focused on "Spatial upsampling for obtaining a high-spatial-resolution HRTF from a very low number of directions". The team was led by Yoshiki Masuyama, and also included Gordon Wichern, Francois Germain, MERL intern Christopher Ick, and Jonathan Le Roux.

        The LAP Challenge workshop and award ceremony was hosted by the 32nd European Signal Processing Conference (EUSIPCO 24) on August 29, 2024 in Lyon, France. Yoshiki Masuyama presented the team's method, "Retrieval-Augmented Neural Field for HRTF Upsampling and Personalization", and received the award from Prof. Michele Geronazzo (University of Padova, IT, and Imperial College London, UK), Chair of the Challenge's Organizing Committee.

        The LAP challenge aims to explore challenges in the field of personalized spatial audio, with the first edition focusing on the spatial upsampling and interpolation of head-related transfer functions (HRTFs). HRTFs with dense spatial grids are required for immersive audio experiences, but their recording is time-consuming. Although HRTF spatial upsampling has recently shown remarkable progress with approaches involving neural fields, HRTF estimation accuracy remains limited when upsampling from only a few measured directions, e.g., 3 or 5 measurements. The MERL team tackled this problem by proposing a retrieval-augmented neural field (RANF). RANF retrieves a subject whose HRTFs are close to those of the target subject at the measured directions from a library of subjects. The HRTF of the retrieved subject at the target direction is fed into the neural field in addition to the desired sound source direction. The team also developed a neural network architecture that can handle an arbitrary number of retrieved subjects, inspired by a multi-channel processing technique called transform-average-concatenate.
    •  
    •  AWARD    Jonathan Le Roux elevated to IEEE Fellow
      Date: January 1, 2024
      Awarded to: Jonathan Le Roux
      MERL Contact: Jonathan Le Roux
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL Distinguished Scientist and Speech & Audio Senior Team Leader Jonathan Le Roux has been elevated to IEEE Fellow, effective January 2024, "for contributions to multi-source speech and audio processing."

        Mitsubishi Electric celebrated Dr. Le Roux's elevation and that of another researcher from the company, Dr. Shumpei Kameyama, with a worldwide news release on February 15.

        Dr. Jonathan Le Roux has made fundamental contributions to the field of multi-speaker speech processing, especially to the areas of speech separation and multi-speaker end-to-end automatic speech recognition (ASR). His contributions constituted a major advance in realizing a practically usable solution to the cocktail party problem, enabling machines to replicate humans’ ability to concentrate on a specific sound source, such as a certain speaker within a complex acoustic scene—a long-standing challenge in the speech signal processing community. Additionally, he has made key contributions to the measures used for training and evaluating audio source separation methods, developing several new objective functions to improve the training of deep neural networks for speech enhancement, and analyzing the impact of metrics used to evaluate the signal reconstruction quality. Dr. Le Roux’s technical contributions have been crucial in promoting the widespread adoption of multi-speaker separation and end-to-end ASR technologies across various applications, including smart speakers, teleconferencing systems, hearables, and mobile devices.

        IEEE Fellow is the highest grade of membership of the IEEE. It honors members with an outstanding record of technical achievements, contributing importantly to the advancement or application of engineering, science and technology, and bringing significant value to society. Each year, following a rigorous evaluation procedure, the IEEE Fellow Committee recommends a select group of recipients for elevation to IEEE Fellow. Less than 0.1% of voting members are selected annually for this member grade elevation.
    •  
    •  AWARD    MERL team wins the Audio-Visual Speech Enhancement (AVSE) 2023 Challenge
      Date: December 16, 2023
      Awarded to: Zexu Pan, Gordon Wichern, Yoshiki Masuyama, Francois Germain, Sameer Khurana, Chiori Hori, and Jonathan Le Roux
      MERL Contacts: François Germain; Chiori Hori; Sameer Khurana; Jonathan Le Roux; Gordon Wichern; Yoshiki Masuyama
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL's Speech & Audio team ranked 1st out of 12 teams in the 2nd COG-MHEAR Audio-Visual Speech Enhancement Challenge (AVSE). The team was led by Zexu Pan, and also included Gordon Wichern, Yoshiki Masuyama, Francois Germain, Sameer Khurana, Chiori Hori, and Jonathan Le Roux.

        The AVSE challenge aims to design better speech enhancement systems by harnessing the visual aspects of speech (such as lip movements and gestures) in a manner similar to the brain’s multi-modal integration strategies. MERL’s system was a scenario-aware audio-visual TF-GridNet, that incorporates the face recording of a target speaker as a conditioning factor and also recognizes whether the predominant interference signal is speech or background noise. In addition to outperforming all competing systems in terms of objective metrics by a wide margin, in a listening test, MERL’s model achieved the best overall word intelligibility score of 84.54%, compared to 57.56% for the baseline and 80.41% for the next best team. The Fisher’s least significant difference (LSD) was 2.14%, indicating that our model offered statistically significant speech intelligibility improvements compared to all other systems.
    •  

    See All Awards for Speech & Audio

  • News & Events

    •  TALK    [MERL Seminar Series 2024] Samuel Clarke presents talk titled Audio for Object and Spatial Awareness
      Date & Time: Wednesday, October 30, 2024; 1:00 PM
      Speaker: Samuel Clarke, Stanford University
      MERL Host: Gordon Wichern
      Research Areas: Artificial Intelligence, Machine Learning, Robotics, Speech & Audio
      Abstract
      • Acoustic perception is invaluable to humans and robots in understanding objects and events in their environments. These sounds are dependent on properties of the source, the environment, and the receiver. Many humans possess remarkable intuition both to infer key properties of each of these three aspects from a sound and to form expectations of how these different aspects would affect the sound they hear. In order to equip robots and AI agents with similar if not stronger capabilities, our research has taken a two-fold path. First, we collect high-fidelity datasets in both controlled and uncontrolled environments which capture real sounds of objects and rooms. Second, we introduce differentiable physics-based models that can estimate acoustic properties of objects and rooms from minimal amounts of real audio data, then can predict new sounds from these objects and rooms under novel, “unseen” conditions.
    •  
    •  NEWS    MERL at the International Conference on Robotics and Automation (ICRA) 2024
      Date: May 13, 2024 - May 17, 2024
      Where: Yokohama, Japan
      MERL Contacts: Anoop Cherian; Radu Corcodel; Stefano Di Cairano; Chiori Hori; Siddarth Jain; Devesh K. Jha; Jonathan Le Roux; Diego Romeres; William S. Yerazunis
      Research Areas: Artificial Intelligence, Machine Learning, Optimization, Robotics, Speech & Audio
      Brief
      • MERL made significant contributions to both the organization and the technical program of the International Conference on Robotics and Automation (ICRA) 2024, which was held in Yokohama, Japan from May 13th to May 17th.

        MERL was a Bronze sponsor of the conference, and exhibited a live robotic demonstration, which attracted a large audience. The demonstration showcased an Autonomous Robotic Assembly technology executed on MELCO's Assista robot arm and was the collaborative effort of the Optimization and Robotics Team together with the Advanced Technology department at Mitsubishi Electric.

        MERL researchers from the Optimization and Robotics, Speech & Audio, and Control for Autonomy teams also presented 8 papers and 2 invited talks covering topics on robotic assembly, applications of LLMs to robotics, human robot interaction, safe and robust path planning for autonomous drones, transfer learning, perception and tactile sensing.
    •  

    See All News & Events for Speech & Audio

  • Research Highlights

  • Internships

    • SA0044: Internship - Multimodal scene-understanding

      We are looking for a graduate student interested in helping advance the field of multimodal scene understanding, focusing on scene understanding using natural language for robot dialog and/or indoor monitoring using a large language model. The intern will collaborate with MERL researchers to derive and implement new models and optimization methods, conduct experiments, and prepare results for publication. Internships regularly lead to one or more publications in top-tier venues, which can later become part of the intern''''s doctoral work. The ideal candidates are senior Ph.D. students with experience in deep learning for audio-visual, signal, and natural language processing. Good programming skills in Python and knowledge of deep learning frameworks such as PyTorch are essential. Multiple positions are available with flexible start date (not just Spring/Summer but throughout 2024) and duration (typically 3-6 months).

      Required Specific Experience

      • Experience with ROS2, C/C++, Python, and deep learning frameworks such as PyTorch are essential.

    • SA0040: Internship - Sound event and anomaly detection

      We are seeking graduate students interested in helping advance the fields of sound event detection/localization, anomaly detection, and physics informed deep learning for machine sounds. The interns will collaborate with MERL researchers to derive and implement novel algorithms, record data, conduct experiments, integrate audio signals with other sensors (electrical, vision, vibration, etc.), and prepare results for publication. Internships regularly lead to one or more publications in top-tier venues, which can later become part of the intern's doctoral work.

      The ideal candidates are senior Ph.D. students with experience in some of the following: audio signal processing, microphone array processing, physics informed machine learning, outlier detection, and unsupervised learning.

      Multiple positions are available with flexible start dates (not just Spring/Summer but throughout 2025) and duration (typically 3-6 months).

    • SA0045: Internship - Universal Audio Compression and Generation

      We are seeking graduate students interested in helping advance the fields of universal audio compression and generation. We aim to build a single generative model that can perform multiple audio generation tasks conditioned on multimodal context. The interns will collaborate with MERL researchers to derive and implement new models and optimization methods, conduct experiments, and prepare results for publication. Internships regularly lead to one or more publications in top-tier venues, which can later become part of the intern's doctoral work. The ideal candidates are Ph.D. students with experience in some of the following: deep generative modeling, large language models, neural audio codecs. The internship typically lasts 3-6 months.


    See All Internships for Speech & Audio

  • Recent Publications

    •  Cornell, S., Ebbers, J., Douwes, C., Martin-Morato, I., Harju, M., Mesaros, A., Serizel, R., "DCASE 2024 Task 4: Sound Event Detection with Heterogeneous Data and Missing Labels", Detection and Classification of Acoustic Scenes and Events (DCASE) Workshop, October 2024.
      BibTeX TR2024-146 PDF
      • @inproceedings{Cornell2024oct,
      • author = {Cornell, Samuele and Ebbers, Janek and Douwes, Constance and Martin-Morato, Irene and Harju, Manu and Mesaros, Annamaria and Serizel, Romain}},
      • title = {DCASE 2024 Task 4: Sound Event Detection with Heterogeneous Data and Missing Labels},
      • booktitle = {Detection and Classification of Acoustic Scenes and Events (DCASE) Workshop},
      • year = 2024,
      • month = oct,
      • url = {https://www.merl.com/publications/TR2024-146}
      • }
    •  Saijo, K., Wichern, G., Germain, F.G., Pan, Z., Le Roux, J., "TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement", International Workshop on Acoustic Signal Enhancement (IWAENC), September 2024.
      BibTeX TR2024-126 PDF Software
      • @inproceedings{Saijo2024sep2,
      • author = {Saijo, Kohei and Wichern, Gordon and Germain, François G and Pan, Zexu and Le Roux, Jonathan}},
      • title = {TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement},
      • booktitle = {International Workshop on Acoustic Signal Enhancement (IWAENC)},
      • year = 2024,
      • month = sep,
      • url = {https://www.merl.com/publications/TR2024-126}
      • }
    •  Yin, J., Luo, A., Du, Y., Cherian, A., Marks, T.K., Le Roux, J., Gan, C., "Disentangled Acoustic Fields For Multimodal Physical Scene Understanding", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 2024.
      BibTeX TR2024-125 PDF
      • @inproceedings{Yin2024sep,
      • author = {Yin, Jie and Luo, Andrew and Du, Yilun and Cherian, Anoop and Marks, Tim K. and Le Roux, Jonathan and Gan, Chuang}},
      • title = {Disentangled Acoustic Fields For Multimodal Physical Scene Understanding},
      • booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
      • year = 2024,
      • month = sep,
      • url = {https://www.merl.com/publications/TR2024-125}
      • }
    •  Bahrman, L., Fontaine, M., Le Roux, J., Richard, G., "Speech Dereverberation Constrained on Room Impulse Response Characteristics", Interspeech, DOI: 10.21437/​Interspeech.2024-1173, September 2024, pp. 622-626.
      BibTeX TR2024-121 PDF
      • @inproceedings{Bahrman2024sep,
      • author = {Bahrman, Louis and Fontaine, Mathieu and Le Roux, Jonathan and Richard, Gaël}},
      • title = {Speech Dereverberation Constrained on Room Impulse Response Characteristics},
      • booktitle = {Interspeech},
      • year = 2024,
      • pages = {622--626},
      • month = sep,
      • doi = {10.21437/Interspeech.2024-1173},
      • issn = {2958-1796},
      • url = {https://www.merl.com/publications/TR2024-121}
      • }
    •  Ebbers, J., Germain, F.G., Wichern, G., Le Roux, J., "Sound Event Bounding Boxes", Interspeech, DOI: 10.21437/​Interspeech.2024-2075, September 2024, pp. 562-566.
      BibTeX TR2024-118 PDF Software
      • @inproceedings{Ebbers2024sep,
      • author = {Ebbers, Janek and Germain, François G and Wichern, Gordon and Le Roux, Jonathan}},
      • title = {Sound Event Bounding Boxes},
      • booktitle = {Interspeech},
      • year = 2024,
      • pages = {562--566},
      • month = sep,
      • doi = {10.21437/Interspeech.2024-2075},
      • issn = {2958-1796},
      • url = {https://www.merl.com/publications/TR2024-118}
      • }
    •  Khurana, S., Hori, C., Laurent, A., Wichern, G., Le Roux, J., "ZeroST: Zero-Shot Speech Translation", Interspeech, DOI: 10.21437/​Interspeech.2024-1088, September 2024, pp. 392-396.
      BibTeX TR2024-122 PDF
      • @inproceedings{Khurana2024sep,
      • author = {Khurana, Sameer and Hori, Chiori and Laurent, Antoine and Wichern, Gordon and Le Roux, Jonathan}},
      • title = {ZeroST: Zero-Shot Speech Translation},
      • booktitle = {Interspeech},
      • year = 2024,
      • pages = {392--396},
      • month = sep,
      • doi = {10.21437/Interspeech.2024-1088},
      • issn = {2958-1796},
      • url = {https://www.merl.com/publications/TR2024-122}
      • }
    •  Pan, Z., Wichern, G., Germain, F.G., Saijo, K., Le Roux, J., "PARIS: Pseudo-AutoRegressIve Siamese Training for Online Speech Separation", Interspeech, DOI: 10.21437/​Interspeech.2024-1066, September 2024, pp. 582-586.
      BibTeX TR2024-124 PDF
      • @inproceedings{Pan2024sep,
      • author = {Pan, Zexu and Wichern, Gordon and Germain, François G and Saijo, Kohei and Le Roux, Jonathan}},
      • title = {PARIS: Pseudo-AutoRegressIve Siamese Training for Online Speech Separation},
      • booktitle = {Interspeech},
      • year = 2024,
      • pages = {582--586},
      • month = sep,
      • doi = {10.21437/Interspeech.2024-1066},
      • issn = {2958-1796},
      • url = {https://www.merl.com/publications/TR2024-124}
      • }
    •  Saijo, K., Wichern, G., Germain, F.G., Pan, Z., Le Roux, J., "Enhanced Reverberation as Supervision for Unsupervised Speech Separation", Interspeech, DOI: 10.21437/​Interspeech.2024-1241, September 2024, pp. 607-611.
      BibTeX TR2024-116 PDF Software
      • @inproceedings{Saijo2024sep,
      • author = {Saijo, Kohei and Wichern, Gordon and Germain, François G and Pan, Zexu and Le Roux, Jonathan}},
      • title = {Enhanced Reverberation as Supervision for Unsupervised Speech Separation},
      • booktitle = {Interspeech},
      • year = 2024,
      • pages = {607--611},
      • month = sep,
      • doi = {10.21437/Interspeech.2024-1241},
      • issn = {2958-1796},
      • url = {https://www.merl.com/publications/TR2024-116}
      • }
    See All Publications for Speech & Audio

  • Videos

  • Software & Data Downloads