Jonathan Le Roux

Jonathan Le Roux
Jonathan's Quick Links

  • Biography

    Jonathan completed his B.Sc. and M.Sc. in Mathematics at the École Normale Supérieure in Paris, France. Before joining MERL in 2011, he spent several years in Beijing and Tokyo. In Tokyo he worked as a postdoctoral researcher at NTT's Communication Science Laboratories. His research interests are in signal processing and machine learning applied to speech and audio.

  • Recent News & Events

    •  NEWS    MERL at the International Conference on Robotics and Automation (ICRA) 2024
      Date: May 13, 2024 - May 17, 2024
      Where: Yokohama, Japan
      MERL Contacts: Anoop Cherian; Radu Corcodel; Stefano Di Cairano; Chiori Hori; Siddarth Jain; Devesh K. Jha; Jonathan Le Roux; Diego Romeres; William S. Yerazunis
      Research Areas: Artificial Intelligence, Machine Learning, Optimization, Robotics, Speech & Audio
      Brief
      • MERL made significant contributions to both the organization and the technical program of the International Conference on Robotics and Automation (ICRA) 2024, which was held in Yokohama, Japan from May 13th to May 17th.

        MERL was a Bronze sponsor of the conference, and exhibited a live robotic demonstration, which attracted a large audience. The demonstration showcased an Autonomous Robotic Assembly technology executed on MELCO's Assista robot arm and was the collaborative effort of the Optimization and Robotics Team together with the Advanced Technology department at Mitsubishi Electric.

        MERL researchers from the Optimization and Robotics, Speech & Audio, and Control for Autonomy teams also presented 8 papers and 2 invited talks covering topics on robotic assembly, applications of LLMs to robotics, human robot interaction, safe and robust path planning for autonomous drones, transfer learning, perception and tactile sensing.
    •  
    •  NEWS    MERL Papers and Workshops at CVPR 2024
      Date: June 17, 2024 - June 21, 2024
      Where: Seattle, WA
      MERL Contacts: Petros T. Boufounos; Moitreya Chatterjee; Anoop Cherian; Michael J. Jones; Toshiaki Koike-Akino; Jonathan Le Roux; Suhas Lohit; Tim K. Marks; Pedro Miraldo; Jing Liu; Kuan-Chuan Peng; Pu (Perry) Wang; Ye Wang; Matthew Brand
      Research Areas: Artificial Intelligence, Computational Sensing, Computer Vision, Machine Learning, Speech & Audio
      Brief
      • MERL researchers are presenting 5 conference papers, 3 workshop papers, and are co-organizing two workshops at the CVPR 2024 conference, which will be held in Seattle, June 17-21. CVPR is one of the most prestigious and competitive international conferences in computer vision. Details of MERL contributions are provided below.

        CVPR Conference Papers:

        1. "TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models" by H. Ni, B. Egger, S. Lohit, A. Cherian, Y. Wang, T. Koike-Akino, S. X. Huang, and T. K. Marks

        This work enables a pretrained text-to-video (T2V) diffusion model to be additionally conditioned on an input image (first video frame), yielding a text+image to video (TI2V) model. Other than using the pretrained T2V model, our method requires no ("zero") training or fine-tuning. The paper uses a "repeat-and-slide" method and diffusion resampling to synthesize videos from a given starting image and text describing the video content.

        Paper: https://www.merl.com/publications/TR2024-059
        Project page: https://merl.com/research/highlights/TI2V-Zero

        2. "Long-Tailed Anomaly Detection with Learnable Class Names" by C.-H. Ho, K.-C. Peng, and N. Vasconcelos

        This work aims to identify defects across various classes without relying on hard-coded class names. We introduce the concept of long-tailed anomaly detection, addressing challenges like class imbalance and dataset variability. Our proposed method combines reconstruction and semantic modules, learning pseudo-class names and utilizing a variational autoencoder for feature synthesis to improve performance in long-tailed datasets, outperforming existing methods in experiments.

        Paper: https://www.merl.com/publications/TR2024-040

        3. "Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling" by X. Liu, Y-W. Tai, C-T. Tang, P. Miraldo, S. Lohit, and M. Chatterjee

        This work presents a new strategy for rendering dynamic scenes from novel viewpoints. Our approach is based on stratifying the scene into regions based on the extent of motion of the region, which is automatically determined. Regions with higher motion are permitted a denser spatio-temporal sampling strategy for more faithful rendering of the scene. Additionally, to the best of our knowledge, ours is the first work to enable tracking of objects in the scene from novel views - based on the preferences of a user, provided by a click.

        Paper: https://www.merl.com/publications/TR2024-042

        4. "SIRA: Scalable Inter-frame Relation and Association for Radar Perception" by R. Yataka, P. Wang, P. T. Boufounos, and R. Takahashi

        Overcoming the limitations on radar feature extraction such as low spatial resolution, multipath reflection, and motion blurs, this paper proposes SIRA (Scalable Inter-frame Relation and Association) for scalable radar perception with two designs: 1) extended temporal relation, generalizing the existing temporal relation layer from two frames to multiple inter-frames with temporally regrouped window attention for scalability; and 2) motion consistency track with a pseudo-tracklet generated from observational data for better object association.

        Paper: https://www.merl.com/publications/TR2024-041

        5. "RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation" by Z. Yang, J. Liu, P. Chen, A. Cherian, T. K. Marks, J. L. Roux, and C. Gan

        We leverage Large Language Models (LLM) for zero-shot semantic audio visual navigation. Specifically, by employing multi-modal models to process sensory data, we instruct an LLM-based planner to actively explore the environment by adaptively evaluating and dismissing inaccurate perceptual descriptions.

        Paper: https://www.merl.com/publications/TR2024-043

        CVPR Workshop Papers:

        1. "CoLa-SDF: Controllable Latent StyleSDF for Disentangled 3D Face Generation" by R. Dey, B. Egger, V. Boddeti, Y. Wang, and T. K. Marks

        This paper proposes a new method for generating 3D faces and rendering them to images by combining the controllability of nonlinear 3DMMs with the high fidelity of implicit 3D GANs. Inspired by StyleSDF, our model uses a similar architecture but enforces the latent space to match the interpretable and physical parameters of the nonlinear 3D morphable model MOST-GAN.

        Paper: https://www.merl.com/publications/TR2024-045

        2. “Tracklet-based Explainable Video Anomaly Localization” by A. Singh, M. J. Jones, and E. Learned-Miller

        This paper describes a new method for localizing anomalous activity in video of a scene given sample videos of normal activity from the same scene. The method is based on detecting and tracking objects in the scene and estimating high-level attributes of the objects such as their location, size, short-term trajectory and object class. These high-level attributes can then be used to detect unusual activity as well as to provide a human-understandable explanation for what is unusual about the activity.

        Paper: https://www.merl.com/publications/TR2024-057

        MERL co-organized workshops:

        1. "Multimodal Algorithmic Reasoning Workshop" by A. Cherian, K-C. Peng, S. Lohit, M. Chatterjee, H. Zhou, K. Smith, T. K. Marks, J. Mathissen, and J. Tenenbaum

        Workshop link: https://marworkshop.github.io/cvpr24/index.html

        2. "The 5th Workshop on Fair, Data-Efficient, and Trusted Computer Vision" by K-C. Peng, et al.

        Workshop link: https://fadetrcv.github.io/2024/

        3. "SuperLoRA: Parameter-Efficient Unified Adaptation for Large Vision Models" by X. Chen, J. Liu, Y. Wang, P. Wang, M. Brand, G. Wang, and T. Koike-Akino

        This paper proposes a generalized framework called SuperLoRA that unifies and extends different variants of low-rank adaptation (LoRA). Introducing new options with grouping, folding, shuffling, projection, and tensor decomposition, SuperLoRA offers high flexibility and demonstrates superior performance up to 10-fold gain in parameter efficiency for transfer learning tasks.

        Paper: https://www.merl.com/publications/TR2024-062
    •  

    See All News & Events for Jonathan

  • Awards

    •  AWARD    MERL team wins the Listener Acoustic Personalisation (LAP) 2024 Challenge
      Date: August 29, 2024
      Awarded to: Yoshiki Masuyama, Gordon Wichern, Francois G. Germain, Christopher Ick, and Jonathan Le Roux
      MERL Contacts: François Germain; Jonathan Le Roux; Gordon Wichern; Yoshiki Masuyama
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL's Speech & Audio team ranked 1st out of 7 teams in Task 2 of the 1st SONICOM Listener Acoustic Personalisation (LAP) Challenge, which focused on "Spatial upsampling for obtaining a high-spatial-resolution HRTF from a very low number of directions". The team was led by Yoshiki Masuyama, and also included Gordon Wichern, Francois Germain, MERL intern Christopher Ick, and Jonathan Le Roux.

        The LAP Challenge workshop and award ceremony was hosted by the 32nd European Signal Processing Conference (EUSIPCO 24) on August 29, 2024 in Lyon, France. Yoshiki Masuyama presented the team's method, "Retrieval-Augmented Neural Field for HRTF Upsampling and Personalization", and received the award from Prof. Michele Geronazzo (University of Padova, IT, and Imperial College London, UK), Chair of the Challenge's Organizing Committee.

        The LAP challenge aims to explore challenges in the field of personalized spatial audio, with the first edition focusing on the spatial upsampling and interpolation of head-related transfer functions (HRTFs). HRTFs with dense spatial grids are required for immersive audio experiences, but their recording is time-consuming. Although HRTF spatial upsampling has recently shown remarkable progress with approaches involving neural fields, HRTF estimation accuracy remains limited when upsampling from only a few measured directions, e.g., 3 or 5 measurements. The MERL team tackled this problem by proposing a retrieval-augmented neural field (RANF). RANF retrieves a subject whose HRTFs are close to those of the target subject at the measured directions from a library of subjects. The HRTF of the retrieved subject at the target direction is fed into the neural field in addition to the desired sound source direction. The team also developed a neural network architecture that can handle an arbitrary number of retrieved subjects, inspired by a multi-channel processing technique called transform-average-concatenate.
    •  
    •  AWARD    Jonathan Le Roux elevated to IEEE Fellow
      Date: January 1, 2024
      Awarded to: Jonathan Le Roux
      MERL Contact: Jonathan Le Roux
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL Distinguished Scientist and Speech & Audio Senior Team Leader Jonathan Le Roux has been elevated to IEEE Fellow, effective January 2024, "for contributions to multi-source speech and audio processing."

        Mitsubishi Electric celebrated Dr. Le Roux's elevation and that of another researcher from the company, Dr. Shumpei Kameyama, with a worldwide news release on February 15.

        Dr. Jonathan Le Roux has made fundamental contributions to the field of multi-speaker speech processing, especially to the areas of speech separation and multi-speaker end-to-end automatic speech recognition (ASR). His contributions constituted a major advance in realizing a practically usable solution to the cocktail party problem, enabling machines to replicate humans’ ability to concentrate on a specific sound source, such as a certain speaker within a complex acoustic scene—a long-standing challenge in the speech signal processing community. Additionally, he has made key contributions to the measures used for training and evaluating audio source separation methods, developing several new objective functions to improve the training of deep neural networks for speech enhancement, and analyzing the impact of metrics used to evaluate the signal reconstruction quality. Dr. Le Roux’s technical contributions have been crucial in promoting the widespread adoption of multi-speaker separation and end-to-end ASR technologies across various applications, including smart speakers, teleconferencing systems, hearables, and mobile devices.

        IEEE Fellow is the highest grade of membership of the IEEE. It honors members with an outstanding record of technical achievements, contributing importantly to the advancement or application of engineering, science and technology, and bringing significant value to society. Each year, following a rigorous evaluation procedure, the IEEE Fellow Committee recommends a select group of recipients for elevation to IEEE Fellow. Less than 0.1% of voting members are selected annually for this member grade elevation.
    •  
    •  AWARD    MERL team wins the Audio-Visual Speech Enhancement (AVSE) 2023 Challenge
      Date: December 16, 2023
      Awarded to: Zexu Pan, Gordon Wichern, Yoshiki Masuyama, Francois Germain, Sameer Khurana, Chiori Hori, and Jonathan Le Roux
      MERL Contacts: François Germain; Chiori Hori; Sameer Khurana; Jonathan Le Roux; Gordon Wichern; Yoshiki Masuyama
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL's Speech & Audio team ranked 1st out of 12 teams in the 2nd COG-MHEAR Audio-Visual Speech Enhancement Challenge (AVSE). The team was led by Zexu Pan, and also included Gordon Wichern, Yoshiki Masuyama, Francois Germain, Sameer Khurana, Chiori Hori, and Jonathan Le Roux.

        The AVSE challenge aims to design better speech enhancement systems by harnessing the visual aspects of speech (such as lip movements and gestures) in a manner similar to the brain’s multi-modal integration strategies. MERL’s system was a scenario-aware audio-visual TF-GridNet, that incorporates the face recording of a target speaker as a conditioning factor and also recognizes whether the predominant interference signal is speech or background noise. In addition to outperforming all competing systems in terms of objective metrics by a wide margin, in a listening test, MERL’s model achieved the best overall word intelligibility score of 84.54%, compared to 57.56% for the baseline and 80.41% for the next best team. The Fisher’s least significant difference (LSD) was 2.14%, indicating that our model offered statistically significant speech intelligibility improvements compared to all other systems.
    •  
    •  AWARD    MERL Intern and Researchers Win ICASSP 2023 Best Student Paper Award
      Date: June 9, 2023
      Awarded to: Darius Petermann, Gordon Wichern, Aswin Subramanian, Jonathan Le Roux
      MERL Contacts: Jonathan Le Roux; Gordon Wichern
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • Former MERL intern Darius Petermann (Ph.D. Candidate at Indiana University) has received a Best Student Paper Award at the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023) for the paper "Hyperbolic Audio Source Separation", co-authored with MERL researchers Gordon Wichern and Jonathan Le Roux, and former MERL researcher Aswin Subramanian. The paper presents work performed during Darius's internship at MERL in the summer 2022. The paper introduces a framework for audio source separation using embeddings on a hyperbolic manifold that compactly represent the hierarchical relationship between sound sources and time-frequency features. Additionally, the code associated with the paper is publicly available at https://github.com/merlresearch/hyper-unmix.

        ICASSP is the flagship conference of the IEEE Signal Processing Society (SPS). ICASSP 2023 was held in the Greek island of Rhodes from June 04 to June 10, 2023, and it was the largest ICASSP in history, with more than 4000 participants, over 6128 submitted papers and 2709 accepted papers. Darius’s paper was first recognized as one of the Top 3% of all papers accepted at the conference, before receiving one of only 5 Best Student Paper Awards during the closing ceremony.
    •  
    •  AWARD    Joint CMU-MERL team wins DCASE2023 Challenge on Automated Audio Captioning
      Date: June 1, 2023
      Awarded to: Shih-Lun Wu, Xuankai Chang, Gordon Wichern, Jee-weon Jung, Francois Germain, Jonathan Le Roux, Shinji Watanabe
      MERL Contacts: François Germain; Jonathan Le Roux; Gordon Wichern
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • A joint team consisting of members of CMU Professor and MERL Alumn Shinji Watanabe's WavLab and members of MERL's Speech & Audio team ranked 1st out of 11 teams in the DCASE2023 Challenge's Task 6A "Automated Audio Captioning". The team was led by student Shih-Lun Wu and also featured Ph.D. candidate Xuankai Chang, Postdoctoral research associate Jee-weon Jung, Prof. Shinji Watanabe, and MERL researchers Gordon Wichern, Francois Germain, and Jonathan Le Roux.

        The IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events (DCASE Challenge), started in 2013, has been organized yearly since 2016, and gathers challenges on multiple tasks related to the detection, analysis, and generation of sound events. This year, the DCASE2023 Challenge received over 428 submissions from 123 teams across seven tasks.

        The CMU-MERL team competed in the Task 6A track, Automated Audio Captioning, which aims at generating informative descriptions for various sounds from nature and/or human activities. The team's system made strong use of large pretrained models, namely a BEATs transformer as part of the audio encoder stack, an Instructor Transformer encoding ground-truth captions to derive an audio-text contrastive loss on the audio encoder, and ChatGPT to produce caption mix-ups (i.e., grammatical and compact combinations of two captions) which, together with the corresponding audio mixtures, increase not only the amount but also the complexity and diversity of the training data. The team's best submission obtained a SPIDEr-FL score of 0.327 on the hidden test set, largely outperforming the 2nd best team's 0.315.
    •  
    •  AWARD    Best Poster Award and Best Video Award at the International Society for Music Information Retrieval Conference (ISMIR) 2020
      Date: October 15, 2020
      Awarded to: Ethan Manilow, Gordon Wichern, Jonathan Le Roux
      MERL Contacts: Jonathan Le Roux; Gordon Wichern
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • Former MERL intern Ethan Manilow and MERL researchers Gordon Wichern and Jonathan Le Roux won Best Poster Award and Best Video Award at the 2020 International Society for Music Information Retrieval Conference (ISMIR 2020) for the paper "Hierarchical Musical Source Separation". The conference was held October 11-14 in a virtual format. The Best Poster Awards and Best Video Awards were awarded by popular vote among the conference attendees.

        The paper proposes a new method for isolating individual sounds in an audio mixture that accounts for the hierarchical relationship between sound sources. Many sounds we are interested in analyzing are hierarchical in nature, e.g., during a music performance, a hi-hat note is one of many such hi-hat notes, which is one of several parts of a drumkit, itself one of many instruments in a band, which might be playing in a bar with other sounds occurring. Inspired by this, the paper re-frames the audio source separation problem as hierarchical, combining similar sounds together at certain levels while separating them at other levels, and shows on a musical instrument separation task that a hierarchical approach outperforms non-hierarchical models while also requiring less training data. The paper, poster, and video can be seen on the paper page on the ISMIR website.
    •  
    •  AWARD    Best Paper Award at the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 2019
      Date: December 18, 2019
      Awarded to: Xuankai Chang, Wangyou Zhang, Yanmin Qian, Jonathan Le Roux, Shinji Watanabe
      MERL Contact: Jonathan Le Roux
      Research Areas: Artificial Intelligence, Machine Learning, Speech & Audio
      Brief
      • MERL researcher Jonathan Le Roux and co-authors Xuankai Chang, Shinji Watanabe (Johns Hopkins University), Wangyou Zhang, and Yanmin Qian (Shanghai Jiao Tong University) won the Best Paper Award at the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2019), for the paper "MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition". MIMO-Speech is a fully neural end-to-end framework that can transcribe the text of multiple speakers speaking simultaneously from multi-channel input. The system is comprised of a monaural masking network, a multi-source neural beamformer, and a multi-output speech recognition model, which are jointly optimized only via an automatic speech recognition (ASR) criterion. The award was received by lead author Xuankai Chang during the conference, which was held in Sentosa, Singapore from December 14-18, 2019.
    •  
    •  AWARD    Best Student Paper Award at IEEE ICASSP 2018
      Date: April 17, 2018
      Awarded to: Zhong-Qiu Wang
      MERL Contact: Jonathan Le Roux
      Research Area: Speech & Audio
      Brief
      • Former MERL intern Zhong-Qiu Wang (Ph.D. Candidate at Ohio State University) has received a Best Student Paper Award at the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2018) for the paper "Multi-Channel Deep Clustering: Discriminative Spectral and Spatial Embeddings for Speaker-Independent Speech Separation" by Zhong-Qiu Wang, Jonathan Le Roux, and John Hershey. The paper presents work performed during Zhong-Qiu's internship at MERL in the summer 2017, extending MERL's pioneering Deep Clustering framework for speech separation to a multi-channel setup. The award was received on behalf on Zhong-Qiu by MERL researcher and co-author Jonathan Le Roux during the conference, held in Calgary April 15-20.
    •  
    •  AWARD    MERL's Speech Team Achieves World's 2nd Best Performance at the Third CHiME Speech Separation and Recognition Challenge
      Date: December 15, 2015
      Awarded to: John R. Hershey, Takaaki Hori, Jonathan Le Roux and Shinji Watanabe
      MERL Contact: Jonathan Le Roux
      Research Area: Speech & Audio
      Brief
      • The results of the third 'CHiME' Speech Separation and Recognition Challenge were publicly announced on December 15 at the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2015) held in Scottsdale, Arizona, USA. MERL's Speech and Audio Team, in collaboration with SRI, ranked 2nd out of 26 teams from Europe, Asia and the US. The task this year was to recognize speech recorded using a tablet in real environments such as cafes, buses, or busy streets. Due to the high levels of noise and the distance from the speaker's mouth to the microphones, this is very challenging task, where the baseline system only achieved 33.4% word error rate. The MERL/SRI system featured state-of-the-art techniques including multi-channel front-end, noise-robust feature extraction, and deep learning for speech enhancement, acoustic modeling, and language modeling, leading to a dramatic 73% reduction in word error rate, down to 9.1%. The core of the system has since been released as a new official challenge baseline for the community to use.
    •  
    •  AWARD    Awaya Prize Young Researcher Award
      Date: March 11, 2014
      Awarded to: Yuuki Tachioka
      Awarded for: "Effectiveness of discriminative approaches for speech recognition under noisy environments on the 2nd CHiME Challenge"
      Awarded by: Acoustical Society of Japan (ASJ)
      MERL Contact: Jonathan Le Roux
      Research Area: Speech & Audio
      Brief
      • MELCO researcher Yuuki Tachioka received the Awaya Prize Young Researcher Award from the Acoustical Society of Japan (ASJ) for "effectiveness of discriminative approaches for speech recognition under noisy environments on the 2nd CHiME Challenge", which was based on joint work with MERL Speech & Audio team researchers Shinji Watanabe, Jonathan Le Roux and John R. Hershey.
    •  
    •  AWARD    Awaya Prize Young Researcher Award
      Date: September 26, 2013
      Awarded to: Jonathan Le Roux
      Awarded for: "A new non-negative dynamical system for speech and audio modeling"
      Awarded by: Acoustical Society of Japan (ASJ)
      MERL Contact: Jonathan Le Roux
      Research Area: Speech & Audio
    •  
    •  AWARD    CHiME 2012 Speech Separation and Recognition Challenge Best Performance
      Date: June 1, 2013
      Awarded to: Yuuki Tachioka, Shinji Watanabe, Jonathan Le Roux and John R. Hershey
      Awarded for: "Discriminative Methods for Noise Robust Speech Recognition: A CHiME Challenge Benchmark"
      Awarded by: International Workshop on Machine Listening in Multisource Environments (CHiME)
      MERL Contact: Jonathan Le Roux
      Research Area: Speech & Audio
      Brief
      • The results of the 2nd 'CHiME' Speech Separation and Recognition Challenge are out! The team formed by MELCO researcher Yuuki Tachioka and MERL Speech & Audio team researchers Shinji Watanabe, Jonathan Le Roux and John Hershey obtained the best results in the continuous speech recognition task (Track 2). This very challenging task consisted in recognizing speech corrupted by highly non-stationary noises recorded in a real living room. Our proposal, which also included a simple yet extremely efficient denoising front-end, focused on investigating and developing state-of-the-art automatic speech recognition back-end techniques: feature transformation methods, as well as discriminative training methods for acoustic and language modeling. Our system significantly outperformed other participants. Our code has since been released as an improved baseline for the community to use.
    •  
    See All Awards for MERL

  • Research Highlights

  • Internships with Jonathan

    • SA0041: Internship - Audio separation, generation, and analysis

      We are seeking graduate students interested in helping advance the fields of generative audio, source separation, speech enhancement, spatial audio, and robust ASR in challenging multi-source and far-field scenarios. The interns will collaborate with MERL researchers to derive and implement new models and optimization methods, conduct experiments, and prepare results for publication. Internships regularly lead to one or more publications in top-tier venues, which can later become part of the intern's doctoral work.

      The ideal candidates are senior Ph.D. students with experience in some of the following: audio signal processing, microphone array processing, spatial audio reproduction, probabilistic modeling, deep generative modeling, and physics informed machine learning techniques (e.g., neural fields, PINNs, sound field and reverberation modeling).

      Multiple positions are available with flexible start dates (not just Spring/Summer but throughout 2025) and duration (typically 3-6 months).

    See All Internships at MERL

  • MERL Publications

    •  Saijo, K., Ebbers, J., Germain, F.G., Wichern, G., Le Roux, J., "Task-Aware Unified Source Separation", arXiv, October 2024.
      BibTeX arXiv
      • @article{Saijo2024oct,
      • author = {Saijo, Kohei and Ebbers, Janek and Germain, François G and Wichern, Gordon and Le Roux, Jonathan}},
      • title = {Task-Aware Unified Source Separation},
      • journal = {arXiv},
      • year = 2024,
      • month = oct,
      • url = {https://arxiv.org/abs/2410.23987v1}
      • }
    •  Saijo, K., Ebbers, J., Germain, F.G., Khurana, S., Wichern, G., Le Roux, J., "Leveraging Audio-Only Data for Text-Queried Target Sound Extraction", arXiv, September 2024.
      BibTeX arXiv
      • @article{Saijo2024sep3,
      • author = {{Saijo, Kohei and Ebbers, Janek and Germain, François G and Khurana, Sameer and Wichern, Gordon and Le Roux, Jonathan}},
      • title = {Leveraging Audio-Only Data for Text-Queried Target Sound Extraction},
      • journal = {arXiv},
      • year = 2024,
      • month = sep,
      • url = {https://arxiv.org/abs/2409.13152v1}
      • }
    •  Saijo, K., Wichern, G., Germain, F.G., Pan, Z., Le Roux, J., "TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement", International Workshop on Acoustic Signal Enhancement (IWAENC), September 2024.
      BibTeX TR2024-126 PDF Software
      • @inproceedings{Saijo2024sep2,
      • author = {Saijo, Kohei and Wichern, Gordon and Germain, François G and Pan, Zexu and Le Roux, Jonathan}},
      • title = {TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement},
      • booktitle = {International Workshop on Acoustic Signal Enhancement (IWAENC)},
      • year = 2024,
      • month = sep,
      • url = {https://www.merl.com/publications/TR2024-126}
      • }
    •  Yin, J., Luo, A., Du, Y., Cherian, A., Marks, T.K., Le Roux, J., Gan, C., "Disentangled Acoustic Fields For Multimodal Physical Scene Understanding", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 2024.
      BibTeX TR2024-125 PDF
      • @inproceedings{Yin2024sep,
      • author = {Yin, Jie and Luo, Andrew and Du, Yilun and Cherian, Anoop and Marks, Tim K. and Le Roux, Jonathan and Gan, Chuang}},
      • title = {Disentangled Acoustic Fields For Multimodal Physical Scene Understanding},
      • booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
      • year = 2024,
      • month = sep,
      • url = {https://www.merl.com/publications/TR2024-125}
      • }
    •  Ebbers, J., Germain, F.G., Wichern, G., Le Roux, J., "Sound Event Bounding Boxes", Interspeech, DOI: 10.21437/​Interspeech.2024-2075, September 2024, pp. 562-566.
      BibTeX TR2024-118 PDF Software
      • @inproceedings{Ebbers2024sep,
      • author = {Ebbers, Janek and Germain, François G and Wichern, Gordon and Le Roux, Jonathan}},
      • title = {Sound Event Bounding Boxes},
      • booktitle = {Interspeech},
      • year = 2024,
      • pages = {562--566},
      • month = sep,
      • doi = {10.21437/Interspeech.2024-2075},
      • issn = {2958-1796},
      • url = {https://www.merl.com/publications/TR2024-118}
      • }
    See All MERL Publications for Jonathan

  • Other Publications

    •  Masahiro Nakano, Jonathan Le Roux, Hirokazu Kameoka, Tomohiro Nakamura, Nobutaka Ono and Shigeki Sagayama, "Bayesian Nonparametric Spectrogram Modeling Based on Infinite Factorial Infinite Hidden Markov Model", IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), October 2011.
      BibTeX
      • @Inproceedings{Nakano2011WASPAA10,
      • author = {Nakano, Masahiro and Le Roux, Jonathan and Kameoka, Hirokazu and Nakamura, Tomohiro and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Bayesian Nonparametric Spectrogram Modeling Based on Infinite Factorial Infinite Hidden Markov Model},
      • booktitle = {IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)},
      • year = 2011,
      • month = oct
      • }
    •  Masahiro Nakano, Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono and Shigeki Sagayama, "Infinite-State Spectrum Model for Music Signal Analysis", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2011, pp. 1972-1975.
      BibTeX
      • @Inproceedings{Nakano2011ICASSP05,
      • author = {Nakano, Masahiro and Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Infinite-State Spectrum Model for Music Signal Analysis},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2011,
      • pages = {1972--1975},
      • month = may
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, Alain de Cheveigné and Shigeki Sagayama, "Computational Auditory Induction as a Missing-Data Model-Fitting Problem with Bregman Divergence", Speech Communication (Special issue on Perceptual and Statistical Audition), Vol. 53, No. 5, pp. 658-676, May-June 2011.
      BibTeX
      • @Article{LeRoux2011Specom05,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and de Cheveigne, Alain and Sagayama, Shigeki},
      • title = {Computational Auditory Induction as a Missing-Data Model-Fitting Problem with Bregman Divergence},
      • journal = {Speech Communication (Special issue on Perceptual and Statistical Audition)},
      • year = 2011,
      • volume = 53,
      • number = 5,
      • pages = {658--676},
      • month = {May-June}
      • }
    •  Hirokazu Kameoka, Takuya Yoshioka, Mariko Hamamura, Jonathan Le Roux and Kunio Kashino, "Statistical Model of Speech Signals Based on Composite Autoregressive System with Application to Blind Source Separation", International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA), September 2010, pp. 245-253.
      BibTeX
      • @Inproceedings{Kameoka2010LVA09,
      • author = {Kameoka, Hirokazu and Yoshioka, Takuya and Hamamura, Mariko and Le Roux, Jonathan and Kashino, Kunio},
      • title = {Statistical Model of Speech Signals Based on Composite Autoregressive System with Application to Blind Source Separation},
      • booktitle = {International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA)},
      • year = 2010,
      • pages = {245--253},
      • month = sep
      • }
    •  Hirokazu Kameoka, Jonathan Le Roux and Yasunori Ohishi, "A Statistical Model of Speech F0 Contours", ISCA Tutorial and Research Workshop on Statistical And Perceptual Audition (SAPA), September 2010, pp. 43-48.
      BibTeX
      • @Inproceedings{Kameoka2010SAPA09,
      • author = {Kameoka, Hirokazu and Le Roux, Jonathan and Ohishi, Yasunori},
      • title = {A Statistical Model of Speech F0 Contours},
      • booktitle = {ISCA Tutorial and Research Workshop on Statistical And Perceptual Audition (SAPA)},
      • year = 2010,
      • pages = {43--48},
      • month = sep
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono and Shigeki Sagayama, "Fast Signal Reconstruction from Magnitude STFT Spectrogram Based on Spectrogram Consistency", International Conference on Digital Audio Effects (DAFx), September 2010, pp. 397-403.
      BibTeX
      • @Inproceedings{LeRoux2010DAFx09,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Fast Signal Reconstruction from Magnitude STFT Spectrogram Based on Spectrogram Consistency},
      • booktitle = {International Conference on Digital Audio Effects (DAFx)},
      • year = 2010,
      • pages = {397--403},
      • month = sep
      • }
    •  Jonathan Le Roux, Emmanuel Vincent, Yuu Mizuno, Hirokazu Kameoka, Nobutaka Ono and Shigeki Sagayama, "Consistent Wiener Filtering: Generalized Time-Frequency Masking Respecting Spectrogram Consistency", International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA), September 2010, pp. 89-96.
      BibTeX
      • @Inproceedings{LeRoux2010LVA09,
      • author = {Le Roux, Jonathan and Vincent, Emmanuel and Mizuno, Yuu and Kameoka, Hirokazu and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Consistent Wiener Filtering: Generalized Time-Frequency Masking Respecting Spectrogram Consistency},
      • booktitle = {International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA)},
      • year = 2010,
      • pages = {89--96},
      • month = sep
      • }
    •  Masahiro Nakano, Jonathan Le Roux, Hirokazu Kameoka, Yu Kitano, Nobutaka Ono and Shigeki Sagayama, "Nonnegative Matrix Factorization with Markov-chained Bases for Modeling Time-varying patterns in Music Spectrograms", International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA), September 2010, pp. 149-156.
      BibTeX
      • @Inproceedings{Nakano2010LVA09,
      • author = {Nakano, Masahiro and Le Roux, Jonathan and Kameoka, Hirokazu and Kitano, Yu and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Nonnegative Matrix Factorization with Markov-chained Bases for Modeling Time-varying patterns in Music Spectrograms},
      • booktitle = {International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA)},
      • year = 2010,
      • pages = {149--156},
      • month = sep
      • }
    •  Masahiro Nakano, Hirokazu Kameoka, Jonathan Le Roux, Yu Kitano, Nobutaka Ono and Shigeki Sagayama, "Convergence-Guaranteed Multiplicative Algorithms for Non-Negative Matrix Factorization with Beta-Divergence", IEEE International Workshop on Machine Learning for Signal Processing (MLSP), August 2010.
      BibTeX
      • @Inproceedings{Nakano2010MLSP08,
      • author = {Nakano, Masahiro and Kameoka, Hirokazu and Le Roux, Jonathan and Kitano, Yu and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Convergence-Guaranteed Multiplicative Algorithms for Non-Negative Matrix Factorization with Beta-Divergence},
      • booktitle = {IEEE International Workshop on Machine Learning for Signal Processing (MLSP)},
      • year = 2010,
      • month = aug
      • }
    •  Eric Carlen, Maria C. Carvalho, Jonathan Le Roux, Michael Loss and Cédric Villani, "Entropy and Chaos in the Kac Model", Kinetic and Related Models, Vol. 3, No. 1, pp. 85-122, March 2010.
      BibTeX
      • @Article{Carlen2010KRM03,
      • author = {Carlen, Eric and Carvalho, Maria C. and Le Roux, Jonathan and Loss, Michael and Villani, Cedric},
      • title = {Entropy and Chaos in the Kac Model},
      • journal = {Kinetic and Related Models},
      • year = 2010,
      • volume = 3,
      • number = 1,
      • pages = {85--122},
      • month = mar
      • }
    •  Nobutaka Ono, Kenichi Miyamoto, Hirokazu Kameoka, Jonathan Le Roux, Yuuki Uchiyama, Emiru Tsunoo, Takuya Nishimoto and Shigeki Sagayama, "Harmonic and Percussive Sound Separation and Its Application to MIR-Related Tasks" in Advances in Music Information Retrieval, Ras, Z. W. and Wieczorkowska, A., Eds., vol. 274 of Studies in Computational Intelligence, pp. 213-236, Springer, 2010.
      BibTeX
      • @Incollection{Ono2010Springer,
      • author = {Ono, Nobutaka and Miyamoto, Kenichi and Kameoka, Hirokazu and Le Roux, Jonathan and Uchiyama, Yuuki and Tsunoo, Emiru and Nishimoto, Takuya and Sagayama, Shigeki},
      • title = {Harmonic and Percussive Sound Separation and Its Application to MIR-Related Tasks},
      • booktitle = {Advances in Music Information Retrieval},
      • year = 2010,
      • editor = {Ras, Z. W. and Wieczorkowska, A.},
      • volume = 274,
      • series = {Studies in Computational Intelligence},
      • pages = {213--236},
      • publisher = {Springer}
      • }
    •  Jonathan Le Roux, "Exploiting Regularities in Natural Acoustical Scenes for Monaural Audio Signal Estimation, Decomposition, Restoration and Modification", March 2009, The University of Tokyo & Université Paris VI--Pierre et Marie Curie.
      BibTeX
      • @Phdthesis{LeRoux2009PhD03,
      • author = {Le Roux, Jonathan},
      • title = {Exploiting Regularities in Natural Acoustical Scenes for Monaural Audio Signal Estimation, Decomposition, Restoration and Modification},
      • school = {The University of Tokyo & Universite Paris VI--Pierre et Marie Curie},
      • year = 2009,
      • month = mar
      • }
    •  Jonathan Le Roux, Alain de Cheveigné and Lucas C. Parra, "Adaptive Template Matching with Shift-Invariant Semi-NMF", Advances in Neural Information Processing Systems (Proc. NIPS), Koller, D. and Bengio, Y. and Shuurmans, D. and Bottou, L., Eds., 2009.
      BibTeX
      • @Inproceedings{LeRoux2008NIPS12,
      • author = {Le Roux, Jonathan and de Cheveigne, Alain and Parra, Lucas C.},
      • title = {Adaptive Template Matching with Shift-Invariant Semi-NMF},
      • booktitle = {Advances in Neural Information Processing Systems (Proc. NIPS)},
      • year = 2009,
      • editor = {Koller, D. and Bengio, Y. and Shuurmans, D. and Bottou, L.},
      • address = {Cambridge, MA},
      • publisher = {The MIT Press}
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, Alain de Cheveigné and Shigeki Sagayama, "Computational Auditory Induction by Missing-Data Non-Negative Matrix Factorization", ISCA Workshop on Statistical and Perceptual Audition (SAPA), September 2008, pp. 1-6.
      BibTeX
      • @Inproceedings{LeRoux2008SAPA09a,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and de Cheveigne, Alain and Sagayama, Shigeki},
      • title = {Computational Auditory Induction by Missing-Data Non-Negative Matrix Factorization},
      • booktitle = {ISCA Workshop on Statistical and Perceptual Audition (SAPA)},
      • year = 2008,
      • pages = {1--6},
      • month = sep
      • }
    •  Jonathan Le Roux, Nobutaka Ono and Shigeki Sagayama, "Explicit Consistency Constraints for STFT Spectrograms and Their Application to Phase Reconstruction", ISCA Workshop on Statistical and Perceptual Audition (SAPA), September 2008, pp. 23-28.
      BibTeX
      • @Inproceedings{LeRoux2008SAPA09b,
      • author = {Le Roux, Jonathan and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Explicit Consistency Constraints for STFT Spectrograms and Their Application to Phase Reconstruction},
      • booktitle = {ISCA Workshop on Statistical and Perceptual Audition (SAPA)},
      • year = 2008,
      • pages = {23--28},
      • month = sep
      • }
    •  Nobutaka Ono, Ken-Ichi Miyamoto, Jonathan Le Roux, Hirokazu Kameoka and Shigeki Sagayama, "Separation of a Monaural Audio Signal into Harmonic/Percussive Components by Complementary Diffusion on Spectrogram", European Signal Processing Conference (EUSIPCO), August 2008.
      BibTeX
      • @Inproceedings{Ono2008EUSIPCO08,
      • author = {Ono, Nobutaka and Miyamoto, Ken-Ichi and Le Roux, Jonathan and Kameoka, Hirokazu and Sagayama, Shigeki},
      • title = {Separation of a Monaural Audio Signal into Harmonic/Percussive Components by Complementary Diffusion on Spectrogram},
      • booktitle = {European Signal Processing Conference (EUSIPCO)},
      • year = 2008,
      • month = aug
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, Shigeki Sagayama and Alain de Cheveigné, "Modulation Analysis of Speech Through Orthogonal FIR Filterbank Optimization", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 2008, pp. 4189-4192.
      BibTeX
      • @Inproceedings{LeRoux2008ICASSP04,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and Sagayama, Shigeki and de Cheveigne, Alain},
      • title = {Modulation Analysis of Speech Through Orthogonal FIR Filterbank Optimization},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2008,
      • pages = {4189--4192},
      • month = apr
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono and Shigeki Sagayama, "On the Interpretation of I-Divergence-Based Distribution-Fitting as a Maximum-Likelihood Estimation Problem," Tech. Rep. METR 2008-11, The University of Tokyo, March 2008.
      BibTeX
      • @Techreport{LeRoux2008TechRep03,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {On the Interpretation of I-Divergence-Based Distribution-Fitting as a Maximum-Likelihood Estimation Problem},
      • institution = {The University of Tokyo},
      • year = 2008,
      • number = {METR 2008-11},
      • month = mar
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, Alain de Cheveigné and Shigeki Sagayama, "Single Channel Speech and Background Segregation through Harmonic-Temporal Clustering", IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), October 2007, pp. 279-282.
      BibTeX
      • @Inproceedings{LeRoux2007WASPAA10,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and de Cheveigne, Alain and Sagayama, Shigeki},
      • title = {Single Channel Speech and Background Segregation through Harmonic-Temporal Clustering},
      • booktitle = {IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)},
      • year = 2007,
      • pages = {279--282},
      • month = oct
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, Alain de Cheveigné and Shigeki Sagayama, "Single and Multiple F0 Contour Estimation Through Parametric Spectrogram Modeling of Speech in Noisy Environments", IEEE Transactions on Audio, Speech and Language Processing, Vol. 15, No. 4, pp. 1135-1145, May 2007.
      BibTeX
      • @Article{LeRoux2007IEEETASLP05,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and de Cheveigne, Alain and Sagayama, Shigeki},
      • title = {Single and Multiple F0 Contour Estimation Through Parametric Spectrogram Modeling of Speech in Noisy Environments},
      • journal = {IEEE Transactions on Audio, Speech and Language Processing},
      • year = 2007,
      • volume = 15,
      • number = 4,
      • pages = {1135--1145},
      • month = may
      • }
    •  Jonathan Le Roux, Hirokazu Kameoka, Nobutaka Ono, Alain de Cheveigné and Shigeki Sagayama, "Harmonic-Temporal Clustering of Speech for Single and Multiple F0 Contour Estimation in Noisy Environments", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 2007, vol. 4, pp. 1053-1056.
      BibTeX
      • @Inproceedings{LeRoux2007ICASSP04,
      • author = {Le Roux, Jonathan and Kameoka, Hirokazu and Ono, Nobutaka and de Cheveigne, Alain and Sagayama, Shigeki},
      • title = {Harmonic-Temporal Clustering of Speech for Single and Multiple F0 Contour Estimation in Noisy Environments},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2007,
      • volume = 4,
      • pages = {1053--1056},
      • month = apr
      • }
    •  Alain de Cheveigné, Jonathan Le Roux and Jonathan Z. Simon, "MEG Signal Denoising based on Time-Shift PCA", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 2007, vol. 1, pp. 317-320.
      BibTeX
      • @Inproceedings{deCheveigne2007ICASSP04,
      • author = {de Cheveigne, Alain and Le Roux, Jonathan and Simon, Jonathan Z.},
      • title = {MEG Signal Denoising based on Time-Shift PCA},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2007,
      • volume = 1,
      • pages = {317--320},
      • month = apr
      • }
    •  Erik McDermott, Timothy J. Hazen, Jonathan Le Roux, Atsushi Nakamura and Shigeru Katagiri, "Discriminative Training for Large Vocabulary Speech Recognition Using Minimum Classification Error", IEEE Transactions on Audio, Speech and Language Processing, Vol. 15, No. 1, pp. 203-223, January 2007.
      BibTeX
      • @Article{McDermott2007IEEETASLP03,
      • author = {McDermott, Erik and Hazen, Timothy J. and Le Roux, Jonathan and Nakamura, Atsushi and Katagiri, Shigeru},
      • title = {Discriminative Training for Large Vocabulary Speech Recognition Using Minimum Classification Error},
      • journal = {IEEE Transactions on Audio, Speech and Language Processing},
      • year = 2007,
      • volume = 15,
      • number = 1,
      • pages = {203--223},
      • month = jan
      • }
    •  Hirokazu Kameoka, Jonathan Le Roux, Nobutaka Ono and Shigeki Sagayama, "Speech Analyzer Using a Joint Estimation Model of Spectral Envelope and Fine Structure", ISCA International Conference on Spoken Language Processing (ICSLP/Interspeech), September 2006, pp. 2502-2505.
      BibTeX
      • @Inproceedings{Kameoka2006Interspeech09,
      • author = {Kameoka, Hirokazu and Le Roux, Jonathan and Ono, Nobutaka and Sagayama, Shigeki},
      • title = {Speech Analyzer Using a Joint Estimation Model of Spectral Envelope and Fine Structure},
      • booktitle = {ISCA International Conference on Spoken Language Processing (ICSLP/Interspeech)},
      • year = 2006,
      • pages = {2502--2505},
      • month = sep
      • }
    •  Jonathan Le Roux and Erik McDermott, "Optimization Methods for Discriminative Training", ISCA European Conference on Speech Communication and Technology (Eurospeech/Interspeech), September 2005, pp. 3341-3344.
      BibTeX
      • @Inproceedings{LeRoux2005Eurospeech09,
      • author = {Le Roux, Jonathan and McDermott, Erik},
      • title = {Optimization Methods for Discriminative Training},
      • booktitle = {ISCA European Conference on Speech Communication and Technology (Eurospeech/Interspeech)},
      • year = 2005,
      • pages = {3341--3344},
      • month = sep
      • }

  • Software & Data Downloads

  • Videos

  • MERL Issued Patents

    • Title: "A Method and System for Scene-Aware Audio-Video Representation"
      Inventors: Cherian, Anoop; Chatterjee, Moitreya; Le Roux, Jonathan
      Patent No.: 12,056,213
      Issue Date: Aug 6, 2024
    • Title: "Method and System for Detecting Anomalous Sound"
      Inventors: Wichern, Gordon P; Chakrabarty, Ankush; Wang, Zhongqiu; Le Roux, Jonathan
      Patent No.: 11,978,476
      Issue Date: May 7, 2024
    • Title: "Long-context End-to-end Speech Recognition System"
      Inventors: Hori, Takaaki; Moritz, Niko; Hori, Chiori; Le Roux, Jonathan
      Patent No.: 11,978,435
      Issue Date: May 7, 2024
    • Title: "Artificial Intelligence System for Sequence-to-Sequence Processing With Attention Adapted for Streaming Applications"
      Inventors: Moritz, Niko; Hori, Takaaki; Le Roux, Jonathan
      Patent No.: 11,810,552
      Issue Date: Nov 7, 2023
    • Title: "Low-latency speech separation using LC-BLSTM and Teacher-Student Learning"
      Inventors: AIHARA, RYO; HANAZAWA, TOSHIYUKI; OKATO, YOHEI; Wichern, Gordon P; Le Roux, Jonathan
      Patent No.: 11798574
      Issue Date: Oct 24, 2023
    • Title: "Method and System for Dereverberation of Speech Signals"
      Inventors: Wang, Zhongqiu; Wichern, Gordon P; Le Roux, Jonathan
      Patent No.: 11,790,930
      Issue Date: Oct 17, 2023
    • Title: "System and Method for Producing Metadata of an Audio Signal"
      Inventors: Moritz, Niko; Wichern, Gordon P; Hori, Takaaki; Le Roux, Jonathan
      Patent No.: 11,756,551
      Issue Date: Sep 12, 2023
    • Title: "Method and System for Scene-Aware Interaction"
      Inventors: Hori, Chiori; Cherian, Anoop; Chen, Siheng; Marks, Tim; Le Roux, Jonathan; Hori, Takaaki; Harsham, Bret A.; Vetro, Anthony; Sullivan, Alan
      Patent No.: 11,635,299
      Issue Date: Apr 25, 2023
    • Title: "Scene-Aware Video Encoder System and Method"
      Inventors: Cherian, Anoop; Hori, Chiori; Le Roux, Jonathan; Marks, Tim; Sullivan, Alan
      Patent No.: 11,582,485
      Issue Date: Feb 14, 2023
    • Title: "Manufacturing Automation using Acoustic Separation Neural Network"
      Inventors: Wichern, Gordon P; Le Roux, Jonathan; Pishdadian, Fatemeh
      Patent No.: 11,579,598
      Issue Date: Feb 14, 2023
    • Title: "An Artificial Intelligence System for Capturing Context by Dilated Self-Attention"
      Inventors: Moritz, Niko; Hori, Takaaki; Le Roux, Jonathan
      Patent No.: 11,557,283
      Issue Date: Jan 17, 2023
    • Title: "System and Method for Hierarchical Audio Source Separation"
      Inventors: Wichern, Gordon P; Le Roux, Jonathan; Manilow, Ethan
      Patent No.: 11,475,908
      Issue Date: Oct 18, 2022
    • Title: "System and Method for Detecting Adversarial Attacks"
      Inventors: Le Roux, Jonathan; Jayashankar, Tejas; Moulin, Pierre
      Patent No.: 11,462,211
      Issue Date: Oct 4, 2022
    • Title: "Low-latency Captioning System"
      Inventors: Hori, Chiori; Hori, Takaaki; Cherian, Anoop; Marks, Tim; Le Roux, Jonathan
      Patent No.: 11,445,267
      Issue Date: Sep 13, 2022
    • Title: "System and Method for Streaming end-to-end Speech Recognition with Asynchronous Decoders pruning prefixes using a joint label and frame information in transcribing technique"
      Inventors: Moritz, Niko; Hori, Takaaki; Le Roux, Jonathan
      Patent No.: 11,373,639
      Issue Date: Jun 28, 2022
    • Title: "Scene-Aware Video Dialog"
      Inventors: Geng, Shijie; Gao, Peng; Cherian, Anoop; Hori, Chiori; Le Roux, Jonathan
      Patent No.: 11,210,523
      Issue Date: Dec 28, 2021
    • Title: "System and Method for End-to-End Speech Recognition with Triggered Attention"
      Inventors: Moritz, Niko; Hori, Takaaki; Le Roux, Jonathan
      Patent No.: 11,100,920
      Issue Date: Aug 24, 2021
    • Title: "Method and System for Multi-Label Classification"
      Inventors: Hori, Takaaki; Hori, Chiori; Watanabe, Shinji; Hershey, John R.; Harsham, Bret A.; Le Roux, Jonathan
      Patent No.: 11,086,918
      Issue Date: Aug 10, 2021
    • Title: "Methods and Systems for Recognizing Simultaneous Speech by Multiple Speakers"
      Inventors: Le Roux, Jonathan; Hori, Takaaki; Settle, Shane; Seki, Hiroshi; Watanabe, Shinji; Hershey, John R.
      Patent No.: 10,811,000
      Issue Date: Oct 20, 2020
    • Title: "Methods and Systems for Enhancing Audio Signals Corrupted by Noise"
      Inventors: Le Roux, Jonathan; Watanabe, Shinji; Hershey, John R.; Wichern, Gordon P
      Patent No.: 10,726,856
      Issue Date: Jul 28, 2020
    • Title: "Method and Apparatus for Multi-Lingual End-to-End Speech Recognition"
      Inventors: Watanabe, Shinji; Hori, Takaaki; Seki, Hiroshi; Le Roux, Jonathan; Hershey, John R.
      Patent No.: 10,593,321
      Issue Date: Mar 17, 2020
    • Title: "Neural Networks for Transforming Signals"
      Inventors: Le Roux, Jonathan; Hershey, John R.; Weninger, Felix
      Patent No.: 10,592,800
      Issue Date: Mar 17, 2020
    • Title: "Methods and Systems for End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction"
      Inventors: Le Roux, Jonathan; Hershey, John R.; Wang, Zhongqiu; Wichern, Gordon P
      Patent No.: 10,529,349
      Issue Date: Jan 7, 2020
    • Title: "Method for Enhancing Audio Signal using Phase Information"
      Inventors: Erdogan, Hakan; Hershey, John R.; Watanabe, Shinji; Le Roux, Jonathan
      Patent No.: 9,881,631
      Issue Date: Jan 30, 2018
    • Title: "Method for Distinguishing Components of Signal of Environment"
      Inventors: Hershey, John R.; Le Roux, Jonathan; Watanabe, Shinji; Chen, Zhuo
      Patent No.: 9,685,155
      Issue Date: Jun 20, 2017
    • Title: "Source Signal Separation by Discriminatively-Trained Non-Negative Matrix Factorization"
      Inventors: Le Roux, Jonathan; Hershey, John R.; Weninger, Felix; Watanabe, Shinji
      Patent No.: 9,679,559
      Issue Date: Jun 13, 2017
    • Title: "Flat-Panel Acoustic Apparatus"
      Inventors: Le Roux, Jonathan; Hershey, John R.; Yerazunis, William S.; Boufounos, Petros T.; Daudet, Laurent
      Patent No.: 9,661,414
      Issue Date: May 23, 2017
    • Title: "Method for Processing Speech Signals Using an Ensemble of Speech Enhancement Procedures"
      Inventors: Le Roux, Jonathan; Watanabe, Shinji; Hershey, John R.
      Patent No.: 9,601,130
      Issue Date: Mar 21, 2017
    • Title: "Neural Networks for Transforming Signals"
      Inventors: Hershey, John R.; Le Roux, Jonathan; Weninger, Felix
      Patent No.: 9,582,753
      Issue Date: Feb 28, 2017
    • Title: "Method and System for Detecting Events in an Acoustic Signal Subject to Cyclo-Stationary Noise"
      Inventors: Hershey, John R.; Potluru, Vamsi K.; Le Roux, Jonathan
      Patent No.: 9,477,895
      Issue Date: Oct 25, 2016
    • Title: "Actions Prediction for Hypothetical Driving Conditions"
      Inventors: Harsham, Bret A.; Hershey, John R.; Le Roux, Jonathan; Nikovski, Daniel N.; Esenther, Alan W.
      Patent No.: 9,434,389
      Issue Date: Sep 6, 2016
    • Title: "Method for Distinguishing Components of an Acoustic Signal"
      Inventors: Hershey, John R.; Le Roux, Jonathan; Watanabe, Shinji; Chen, Zhuo
      Patent No.: 9,368,110
      Issue Date: Jun 14, 2016
    • Title: "Denoising Noisy Speech Signals using Probabilistic Model"
      Inventors: Le Roux, Jonathan; Hershey, John R.; Simsekli, Umut
      Patent No.: 9,324,338
      Issue Date: Apr 26, 2016
    • Title: "Method for Localizing Sources of Signals in Reverberant Environments Using Sparse Optimization"
      Inventors: Boufounos, Petros T.; Le Roux, Jonathan; Kang, Kang; Hershey, John R.
      Patent No.: 9,251,436
      Issue Date: Feb 2, 2016
    • Title: "Method and Apparatus for Processing Text with Variations in Vocabulary Usage"
      Inventors: Hershey, John R.; Le Roux, Jonathan; Heakulani, Creighton K.
      Patent No.: 9,251,250
      Issue Date: Feb 2, 2016
    • Title: "Method and System for Dynamically Adapting user Interfaces in Vehicle Navigation Systems to Minimize Interaction Complexity"
      Inventors: Nikovski, Daniel N.; Hershey, John R.; Harsham, Bret A.; Le Roux, Jonathan
      Patent No.: 9,170,119
      Issue Date: Oct 27, 2015
    • Title: "Method of Text Classification Using Discriminative Topic Transformation"
      Inventors: Hershey, John R.; Le Roux, Jonathan
      Patent No.: 9,069,798
      Issue Date: Jun 30, 2015
    • Title: "Indirect Model-Based Speech Enhancement"
      Inventors: Hershey, John R.; Le Roux, Jonathan
      Patent No.: 8,880,393
      Issue Date: Nov 4, 2014
    See All Patents for MERL