Paper Introduces Benchmark For Controllable Speaking-Style Captioning

Speaking-style captioning asks a model to put into words how something was said, not just what was said. Existing systems tend to produce one general description per clip, covering timbre, pacing and emotion all at once, which is of limited use to an application that only needs to know whether a speaker sounded urgent, or whether their accent matches a target profile. The paper reframes the task as instruction following: a user specifies which dimension of delivery they want described, and the model answers that instruction specifically rather than defaulting to a catch-all summary.

To make that claim testable, the authors built StyleInstructCaps, a dataset and evaluation framework rather than a single leaderboard number. It checks several things a caption could fail at independently: whether the words are grounded in details actually present in the metadata, whether the model invents attributes that are not supported by the audio, whether it actually follows the instruction it was given rather than ignoring it, whether performance holds up on instructions and audio sources it was never trained on, and whether its description of one speaker stays consistent across different clips of that speaker.

The headline finding is a gap between two capabilities that sound similar but are not. Models that follow instructions accurately, in the narrow sense of producing an answer shaped the way the prompt asked for, do not reliably produce captions that are actually grounded in the speech itself. The paper also reports that training on data built specifically for this instruction-diverse, dimension-specific setup produces clear improvements across groundedness, controllability and speaker consistency, rather than general-purpose audio-language training alone closing that gap.

The numbers

The benchmark was tested against nine state-of-the-art audio-language models, covering a spread of current approaches to audio understanding rather than a single family. The evaluation framework measures five separate properties: metadata groundedness, hallucination, instruction-following ability, generalization to unseen instructions and datasets, and speaker-style consistency, which lets a team diagnose which of those a given model is weak on instead of reading a single blended score. The paper was accepted at AACL 2026, and the authors have released the dataset, evaluation code and benchmark publicly alongside the preprint, so other groups can run their own models against the same checks rather than taking the reported results on faith.