BASIS-Bench
BASIS-Bench
Do Speech LLMs Know When to Turn a Deaf Ear?
1Université Paris-Saclay · 2NAVER LABS Europe
Bias And perSonalization In Speech LLMs Benchmark
Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions. However, little work investigates how this sensitivity can lead to stereotyping, and existing evaluations largely measure demographic disparities without distinguishing beneficial personalization from stereotypical behavior. We introduce BASIS-Bench, a benchmark designed to disentangle these two aspects through three axes: (1) demographic robustness, measuring disparities in content-based tasks across speakers and demographic classification capabilities; (2) stereotypical bias, measuring whether perceived demographic cues trigger unprompted stereotypical behavior; and (3) personalization, measuring whether models can tailor responses to demographic characteristics when explicitly requested. We evaluate eight recent speech LLMs and find that, despite often recognizing demographic cues, most fail to translate them into appropriate personalization. Explicit demographic labels improve personalization but also amplify stereotypical associations already present in the textual backbones. These findings highlight the need to train speech LLMs not only to perceive demographic cues, but to learn when they should shape their responses.
We distinguish three sources of demographic disparity, and design our evaluation framework to disentangle their contributions.
Disparities arising from the speech encoding process: how reliably linguistic content and demographic cues are represented and perceived. Probed by the demographic robustness axis.
Biases and personalization capabilities inherited from the foundation LLM, independently of the speech modality. Probed by the text modes.
How speech integration amplifies or dampens the behavior already present in the backbone. Probed by comparing equivalent speech and text modes.
We consider personalization and stereotypical bias under four evaluation settings, in order to better isolate acoustic, backbone and interaction effects.
Eight speech LLMs, one dot each, in every plot; up and to the right is better on every axis.
Hover or click a model to highlight it in every plot · hover any dot for its exact numbers · every plotted value is also in the “View as table” toggles
Before asking how models use demographic cues: can they reliably process speech and perceive them at all? Accuracy on content-based tasks (ASR + SQA) against demographic classification accuracy.
Same x for both dimensions — only the classification target changes
The hollow dots are the implicit anchors — they stay on screen. Selecting Explicit adds the filled dots, each tied to its anchor by a dashed link, so it is clear where every model starts and where it ends up. Speech + Explicit reproduces the paper’s Figure 2; the blue band is the Pareto frontier over the speech-implicit points. Axis ranges are fitted per panel, so compare positions within a panel, not across the two.
ICAT (1 − stereotypical bias) vs. personalization accuracy · Spoken StereoSet+ / BASIS-Health + BASIS-Product
ICAT (1 − stereotypical bias) vs. personalization accuracy · Spoken StereoSet+ / ELIP-Pair
“Current speech LLMs often fail to listen when demographics matter, yet fail to turn a deaf ear when they do not.”
BASIS-Bench releases three data resources. Every spoken question below is recorded in multiple voices, so the same content appears with opposite targets depending on the speaker — pick a voice and listen.
Stereotypical bias. We manually re-annotated Spoken StereoSet against an explicit definition of stereotypical bias, filtering the pitfalls catalogued in the Inventory of Pitfalls in Fairness Benchmark Datasets — stereotype conflation, invalid continuations, invalid perturbations — plus demographic-specific criteria (demographic leakage and invalid gender for age; inappropriate content and social perception for gender). The model hears a first-person context and picks the most plausible continuation; ICAT measures the skew between stereotypical and anti-stereotypical selections (100 = a perfectly balanced 50/50 split).
Gender personalization, framed as constrained gender recognition to avoid stereotypical associations: the question is gender-neutral, the options carry explicit gender markers, and the model — explicitly instructed to personalize — must pick the option matching the voice. Health questions are grounded in biological differences between male and female. A controlled diagnostic task, not a clinical benchmark (see the paper’s ethics statement).
Gender personalization on product recommendations: two otherwise identical options that differ only by an explicit gender marker (Men’s vs. Women’s). Brands are entirely fictional to prevent reliance on memorized brand associations.
Age personalization as open-ended generation, built on ELIP (Thonet et al., 2025): each question pairs a child reference answer — friendly, witty, funny and humorous tone, assuming a non-expert audience — with an adult one — formal style, assuming an expert audience. The model answers the spoken question and an LLM-as-a-judge classifies the response into the profile it is fit for.
If you use BASIS-Bench, please cite the paper.
@misc{basisbench2026,
title = {Personalization Without Stereotyping: Do Speech {LLMs} Know When to Turn a Deaf Ear?},
author = {Ambilduke, Kshitij and Brun, Caroline and Thonet, Thibaut and Nikoulina, Vassilina and Boito, Marcely Zanon},
year = {2026},
eprint = {2610.13550},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.13550}
}