Publications

2025

1. EMNLP Findings preprint

   From Tower to Spire: Adding the speech modality to a translation specialist LLM

   Ambilduke, Kshitij, Peters, Ben, Sannigrahi, Sonal, Keshwani, Anil, Lam, Tsz Kin, Martins, Bruno, Martins, André FT, and Boito, Marcely Zanon

   arXiv preprint arXiv:2503.10620 2025

   PDF
2. IWSLT Track Winner

   NAVER LABS Europe Submission to the Instruction-following Track

   Lee, Beomseok, Boito, Marcely Zanon, Besacier, Laurent, and Calapodescu, Ioan

   In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT) 2025

   Abs PDF Poster

   In this paper we describe NAVER LABS Europe submission to the instruction-following speech processing short track at IWSLT 2025. We participate in the constrained settings, developing systems that can simultaneously perform ASR, ST, and SQA tasks from English speech input into the following target languages: Chinese, Italian, and German. Our solution leverages two pretrained modules: (1) a speech-to-LLM embedding projector trained using representations from the SeamlessM4T-v2-large speech encoder; and (2) LoRA adapters trained on text data on top of Llama-3.1-8B-Instruct. These modules are jointly loaded and further instruction-tuned for 1K steps on multilingual and multimodal data to form our final system submitted for evaluation.

2024

1. Interspeech

   mHuBERT-147: A Compact Multilingual HuBERT Model

   Boito, Marcely Zanon, Iyer, Vivek, Lagos, Nikolaos, Besacier, Laurent, and Calapodescu, Ioan

   In Interspeech 2024

   PDF Poster
2. ICASSP

   Multilingual Distilwhisper: Efficient Distillation of Multi-Task Speech Models Via Language-Specific Experts

   Ferraz, Thomas Palmeira, Boito, Marcely Zanon, Brun, Caroline, and Nikoulina, Vassilina

   In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024

   PDF Poster
3. CS&L Journal

   LeBenchmark 2.0: A standardized, replicable and enhanced framework for self-supervised representations of French speech

   Parcollet, Titouan, Nguyen, Ha, Evain, Solène, Boito, Marcely Zanon, Pupier, Adrien, Mdhaffar, Salima, Le, Hang, Alisamir, Sina, Tomashenko, Natalia, Dinarelli, Marco, Zhang, Shucong, Allauzen, Alexandre, Coavoux, Maximin, Estève, Yannick, Rouvier, Mickael, Goulian, Jerôme, Lecouteux, Benjamin, Portet, François, Rossato, Solange, Ringeval, Fabien, Schwab, Didier, and Besacier, Laurent

   Computer Speech & Language 2024

   Abs PDF

   Self-supervised learning (SSL) is at the origin of unprecedented improvements in many different domains including computer vision and natural language processing. Speech processing drastically benefitted from SSL as most of the current domain-related tasks are now being approached with pre-trained models. This work introduces LeBenchmark 2.0 an open-source framework for assessing and building SSL-equipped French speech technologies. It includes documented, large-scale and heterogeneous corpora with up to 14,000 h of heterogeneous speech, ten pre-trained SSL wav2vec 2.0 models containing from 26 million to one billion learnable parameters shared with the community, and an evaluation protocol made of six downstream tasks to complement existing benchmarks. LeBenchmark 2.0 also presents unique perspectives on pre-trained SSL models for speech with the investigation of frozen versus fine-tuned downstream models, task-agnostic versus task-specific pre-trained models as well as a discussion on the carbon footprint of large-scale model training. Overall, the newly introduced models trained on 14,000 h of French speech outperform multilingual and previous LeBenchmark SSL models across the benchmark but also required up to four times more energy for pre-training.

2023

1. IWSLT Track Winner

   NAVER LABS Europe’s Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track

   Gow-Smith, Edward, Berard, Alexandre, Boito, Marcely Zanon, and Calapodescu, Ioan

   In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT) 2023

   Abs PDF

   This paper presents NAVER LABS Europe’s systems for Tamasheq-French and Quechua-Spanish speech translation in the IWSLT 2023 Low-Resource track. Our work attempts to maximize translation quality in low-resource settings using multilingual parameter-efficient solutions that leverage strong pre-trained models. Our primary submission for Tamasheq outperforms the previous state of the art by 7.5 BLEU points on the IWSLT 2022 test set, and achieves 23.6 BLEU on this year’s test set, outperforming the second best participant by 7.7 points. For Quechua, we also rank first and achieve 17.7 BLEU, despite having only two hours of translation data. Finally, we show that our proposed multilingual architecture is also competitive for high-resource languages, outperforming the best unconstrained submission to the IWSLT 2021 Multilingual track, despite using much less training data and compute.

2022

1. Interspeech

   A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systems

   Boito, Marcely Zanon, Besacier, Laurent, Tomashenko, Natalia, and Estéve, Yannick

   In Interspeech 2022

   PDF
2. SIGUL

   Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings

   Boito, Marcely Zanon, Yusuf, Bolaji, Ondel, Lucas, Villavicencio, Aline, and Besacier, Laurent

   In Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under-Resourced Languages 2022

   Abs PDF Slides

   Documenting languages helps to prevent the extinction of endangered dialects - many of which are otherwise expected to disappear by the end of the century. When documenting oral languages, unsupervised word segmentation (UWS) from speech is a useful, yet challenging, task. It consists in producing time-stamps for slicing utterances into smaller segments corresponding to words, being performed from phonetic transcriptions, or in the absence of these, from the output of unsupervised speech discretization models. These discretization models are trained using raw speech only, producing discrete speech units that can be applied for downstream (text-based) tasks. In this paper we compare five of these models: three Bayesian and two neural approaches, with regards to the exploitability of the produced units for UWS. For the UWS task, we experiment with two models, using as our target language the Mboshi (Bantu C25), an unwritten language from Congo-Brazzaville. Additionally, we report results for Finnish, Hungarian, Romanian and Russian in equally low-resource settings, using only 4 hours of speech. Our results suggest that neural models for speech discretization are difficult to exploit in our setting, and that it might be necessary to adapt them to limit sequence length. We obtain our best UWS results by using Bayesian models that produce high quality, yet compressed, discrete representations of the input speech signal.

3. LREC

   Speech Resources in the Tamasheq Language

   Boito, Marcely Zanon, Bougares, Fethi, Barbier, Florentin, Gahbiche, Souhir, Barrault, Loïc, Rouvier, Mickael, and Estève, Yannick

   In Proceedings of the Thirteenth Language Resources and Evaluation Conference 2022

   Abs PDF Poster

   In this paper we present two datasets for Tamasheq, a developing language mainly spoken in Mali and Niger. These two datasets were made available for the IWSLT 2022 low-resource speech translation track, and they consist of collections of radio recordings from daily broadcast news in Niger (Studio Kalangou) and Mali (Studio Tamani). We share (i) a massive amount of unlabeled audio data (671 hours) in five languages: French from Niger, Fulfulde, Hausa, Tamasheq and Zarma, and (ii) a smaller 17 hours parallel corpus of audio recordings in Tamasheq, with utterance-level translations in the French language. All this data is shared under the Creative Commons BY-NC-ND 3.0 license. We hope these resources will inspire the speech community to develop and benchmark models using the Tamasheq language.

4. IWSLT Track Winner

   ON-TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks

   Boito, Marcely Zanon, Ortega, John, Riguidel, Hugo, Laurent, Antoine, Barrault, Loïc, Bougares, Fethi, Chaabani, Firas, Nguyen, Ha, Barbier, Florentin, Gahbiche, Souhir, and Estève, Yannick

   In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT) 2022

   Abs PDF Poster

   This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2022: low-resource and dialect speech translation. For the Tunisian Arabic-English dataset (low-resource and dialect tracks), we build an end-to-end model as our joint primary submission, and compare it against cascaded models that leverage a large fine-tuned wav2vec 2.0 model for ASR. Our results show that in our settings pipeline approaches are still very competitive, and that with the use of transfer learning, they can outperform end-to-end models for speech translation (ST). For the Tamasheq-French dataset (low-resource track) our primary submission leverages intermediate representations from a wav2vec 2.0 model trained on 234 hours of Tamasheq audio, while our contrastive model uses a French phonetic transcription of the Tamasheq audio as input in a Conformer speech translation architecture jointly trained on automatic speech recognition, ST and machine translation losses. Our results highlight that self-supervised models trained on smaller sets of target data are more effective to low-resource end-to-end ST fine-tuning, compared to large off-the-shelf models. Results also illustrate that even approximate phonetic transcriptions can improve ST scores.

5. IWSLT

   Findings of the IWSLT 2022 Evaluation Campaign

   Anastasopoulos, Antonios, Barrault, Loïc, Bentivogli, Luisa, Boito, Marcely Zanon, Bojar, Ondřej, Cattoni, Roldano, Currey, Anna, Dinu, Georgiana, Duh, Kevin, Elbayad, Maha, Emmanuel, Clara, Estève, Yannick, Federico, Marcello, Federmann, Christian, Gahbiche, Souhir, Gong, Hongyu, Grundkiewicz, Roman, Haddow, Barry, Hsu, Benjamin, Javorský, Dávid, Kloudová, Vĕra, Lakew, Surafel, Ma, Xutai, Mathur, Prashant, McNamee, Paul, Murray, Kenton, Nǎdejde, Maria, Nakamura, Satoshi, Negri, Matteo, Niehues, Jan, Niu, Xing, Ortega, John, Pino, Juan, Salesky, Elizabeth, Shi, Jiatong, Sperber, Matthias, Stüker, Sebastian, Sudoh, Katsuhito, Turchi, Marco, Virkar, Yogesh, Waibel, Alexander, Wang, Changhan, and Watanabe, Shinji

   In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT) 2022

   Abs PDF

   The evaluation campaign of the 19th International Conference on Spoken Language Translation featured eight shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Speech to speech translation, (iv) Low-resource speech translation, (v) Multilingual speech translation, (vi) Dialect speech translation, (vii) Formality control for speech translation, (viii) Isometric speech translation. A total of 27 teams participated in at least one of the shared tasks. This paper details, for each shared task, the purpose of the task, the data that were released, the evaluation metrics that were applied, the submissions that were received and the results that were achieved.

6. CAID

   Promises and Limitations of Self-supervised Learning for Automatic Speech Processing

   Maison, Lucas, Boito, Marcely Zanon, and Estève, Yannick

   In Actes de la 4ème Conference on Artificial Intelligence for Defense (CAID) 2022

   PDF
7. JEP French

   LeBenchmark, un référentiel d’évaluation pour le français oral

   Le, Hang, Alisamir, Sina, Dinarelli, Marco, Ringeval, Fabien, Evain, Solène, Nguyen, Ha, Boito, Marcely Zanon, Mdhaffar, Salima, Tong, Ziyi, Tomashenko, Natalia, and others,

   In 34e Journées d’étude sur la parole 2022

   PDF
8. JEP French

   Modèles neuronaux pré-appris par auto-supervision sur des enregistrements de parole en français

   Evain, Solène, Nguyen, Ha, Le, Hang, Boito, Marcely Zanon, Mdhaffar, Salima, Alisamir, Sina, Tong, Ziyi, Tomashenko, Natalia, Dinarelli, Marco, Parcollet, Titouan, and others,

   In 34e Journées d’étude sur la parole 2022

   PDF

2021

1. NeurIPS Benchmarks Track

   Task Agnostic and Task Specific Self-Supervised Learning from Speech with LeBenchmark

   Evain, Solène, Nguyen, Ha, Le, Hang, Zanon Boito, Marcely, Mdhaffar, Salima, Alisamir, Sina, Tong, Ziyi, Tomashenko, Natalia, Dinarelli, Marco, Parcollet, Titouan, Allauzen, Alexandre, Estève, Yannick, Lecouteux, Benjamin, Portet, François, Rossato, Solange, Ringeval, Fabien, Schwab, Didier, and besacier, laurent

   In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 2021

   PDF Poster
2. Interspeech

   LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech

   Evain, Solène, Nguyen, Ha, Le, Hang, Boito, Marcely Zanon, Mdhaffar, Salima, Alisamir, Sina, Tong, Ziyi, Tomashenko, Natalia, Dinarelli, Marco, Parcollet, Titouan, Allauzen, Alexandre, Estève, Yannick, Lecouteux, Benjamin, Portet, François, Rossato, Solange, Ringeval, Fabien, Schwab, Didier, and Besacier, Laurent

   In Interspeech 2021 2021

   PDF

2020

1. MT Journal

   Investigating alignment interpretability for low-resource NMT

   Boito, Marcely Zanon, Villavicencio, Aline, and Besacier, Laurent

   Machine Translation 2020

   PDF
2. LREC

   MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible

   Boito, Marcely Zanon, Havard, William, Garnerin, Mahault, Le Ferrand, Éric, and Besacier, Laurent

   In Proceedings of the Twelfth Language Resources and Evaluation Conference 2020

   Abs PDF

   The CMU Wilderness Multilingual Speech Dataset (Black, 2019) is a newly published multilingual speech dataset based on recorded readings of the New Testament. It provides data to build Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) models for potentially 700 languages. However, the fact that the source content (the Bible) is the same for all the languages is not exploited to date. Therefore, this article proposes to add multilingual links between speech segments in different languages, and shares a large and clean dataset of 8,130 parallel spoken utterances across 8 languages (56 language pairs). We name this corpus MaSS (Multilingual corpus of Sentence-aligned Spoken utterances). The covered languages (Basque, English, Finnish, French, Hungarian, Romanian, Russian and Spanish) allow researches on speech-to-speech alignment as well as on translation for typologically different language pairs. The quality of the final corpus is attested by human evaluation performed on a corpus subset (100 utterances, 8 language pairs). Lastly, we showcase the usefulness of the final product on a bilingual speech retrieval task.

3. SLTU

   Investigating Language Impact in Bilingual Approaches for Computational Language Documentation

   Boito, Marcely Zanon, Villavicencio, Aline, and Besacier, Laurent

   In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL) 2020

   Abs PDF

   For endangered languages, data collection campaigns have to accommodate the challenge that many of them are from oral tradition, and producing transcriptions is costly. Therefore, it is fundamental to translate them into a widely spoken language to ensure interpretability of the recordings. In this paper we investigate how the choice of translation language affects the posterior documentation work and potential automatic approaches which will work on top of the produced bilingual corpus. For answering this question, we use the MaSS multilingual speech corpus (Boito et al., 2020) for creating 56 bilingual pairs that we apply to the task of low-resource unsupervised word segmentation and alignment. Our results highlight that the choice of language for translation influences the word segmentation performance, and that different lexicons are learned by using different aligned translations. Lastly, this paper proposes a hybrid approach for bilingual word segmentation, combining boundary clues extracted from a non-parametric Bayesian model (Goldwater et al., 2009a) with the attentional word segmentation neural model from Godard et al. (2018). Our results suggest that incorporating these clues into the neural models’ input representation increases their translation and alignment quality, specially for challenging language pairs.

2019

1. Interspeech

   Empirical Evaluation of Sequence-to-Sequence Models for Word Discovery in Low-Resource Settings

   Boito, Marcely Zanon, Villavicencio, Aline, and Besacier, Laurent

   In Interspeech 2019 2019

   PDF Slides
2. LIFT

   How Does Language Influence Documentation Workflow? Unsupervised Word Discovery Using Translations in Multiple Languages

   Boito, Marcely Zanon, Villavicencio, Aline, and Besacier, Laurent

   In Journées Scientifiques du Groupement de Recherche: Linguistique Informatique, Formelle et de Terrain (LIFT). 2019

   PDF
3. IWSLT

   ON-TRAC Consortium End-to-End Speech Translation Systems for the IWSLT 2019 Shared Task

   Nguyen, Ha, Tomashenko, Natalia, Boito, Marcely Zanon, Caubriére, Antoine, Bougares, Fethi, Rouvier, Mickael, Besacier, Laurent, and Estéve, Yannick

   In Proceedings of the 16th International Conference on Spoken Language Translation 2019

   Abs PDF

   This paper describes the ON-TRAC Consortium translation systems developed for the end-to-end model task of IWSLT Evaluation 2019 for the English\textrightarrow Portuguese language pair. ON-TRAC Consortium is composed of researchers from three French academic laboratories: LIA (Avignon Université), LIG (Université Grenoble Alpes), and LIUM (Le Mans Université). A single end-to-end model built as a neural encoder-decoder architecture with attention mechanism was used for two primary submissions corresponding to the two EN-PT evaluations sets: (1) TED (MuST-C) and (2) How2. In this paper, we notably investigate impact of pooling heterogeneous corpora for training, impact of target tokenization (characters or BPEs), impact of speech input segmentation and we also compare our best end-to-end model (BLEU of 26.91 on MuST-C and 43.82 on How2 validation sets) to a pipeline (ASR+MT) approach.

2018

1. LREC

   A Very Low Resource Language Speech Corpus for Computational Language Documentation Experiments

   Godard, Pierre, Adda, Gilles, Adda-Decker, Martine, Benjumea, Juan, Besacier, Laurent, Cooper-Leavitt, Jamison, Kouarata, Guy-Noel, Lamel, Lori, Maynard, Hélène, Mueller, Markus, Rialland, Annie, Stueker, Sebastian, Yvon, François, and Boito, Marcely Zanon

   In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) 2018

   PDF
2. SLTU

   A Small Griko-Italian Speech Translation Corpus

   Boito, Marcely Zanon, Anastasopoulos, Antonios, Villavicencio, Aline, Besacier, Laurent, and Lekakou, Marika

   In 6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018) 2018

   PDF Slides
3. Interspeech

   Unsupervised Word Segmentation from Speech with Attention

   Godard, Pierre, Boito, Marcely Zanon, Ondel, Lucas, Berard, Alexandre, Yvon, François, Villavicencio, Aline, and Besacier, Laurent

   In Interspeech 2018 2018

   PDF Slides

2017

1. ASRU

   Unwritten languages demand attention too! Word discovery with encoder-decoder models

   Boito, Marcely Zanon, Bérard, Alexandre, Villavicencio, Aline, and Besacier, Laurent

   In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 2017

   PDF

2014

1. IBERAMIA

   Size does not matter. Frequency does. A study of features for measuring lexical complexity

   Wilkers, Rodrigo, Vecchia, Alessandro Dalla, Boito, Marcely Zanon, Padró, Muntsa, and Villavicencio, Aline

   In Ibero-American conference on artificial intelligence 2014

   PDF
2. TorPorEsp Portuguese

   Uma análise do perfil de entropia das estruturas sintáticas do português

   Boito, Marcely Zanon, Hagemann, Luiza, Wilkens, Rodrigo, and Villavicencio, Aline

   In ToRPorEsp workshop, BDBComp (Biblioteca Digital Brasileira de Computação) 2014

   PDF