CapTune: Adapting Non-Speech Captions With Anchored Generative Models

1University of Michigan 2Rochester Institute of Technology

ASSETS ’25

Five-part overview built around a cartoon cat whose original caption is '[Cat sound]'. (1) Alternative captions arranged on axes of Level of Detail and Expressiveness, from '[Sound]' (less detailed, less expressive) to '[Shrill, persistent meowing echoing through the quiet street]' (more detailed, more expressive). (2) A captioner marks a bounded region of this grid as acceptable. (3) A viewer selects one cell inside that region. (4) The same caption in four sound representations: default, source-focused '[Cat meows cheerfully nearby]', onomatopoeia '[Mrrrow! Mrrrow!]', and sensory '[Light, musical cries drift on the breeze]'. (5) Genre-aligned variants for five visual styles, such as '[Metallic screech rings out, sharp and unrelenting]' for a robotic cat.
CapTune maps non-speech captions into a two-dimensional space of Level of Detail and Expressiveness (1). Captioners bound the region of acceptable transformations (2), and viewers select their preference within it (3). Captions can further vary in sound representation (4) and be aligned with the genre and tone of the scene (5).

Abstract

Non-speech captions are essential to the video experience of deaf and hard of hearing (DHH) viewers, yet conventional approaches often overlook the diversity of their preferences. We present CapTune, a system that enables customization of non-speech captions based on DHH viewers’ needs while preserving creator intent. CapTune allows caption authors to define safe transformation spaces using concrete examples and empowers viewers to personalize captions across four dimensions: level of detail, expressiveness, sound representation method, and genre alignment. Evaluations with seven caption creators and twelve DHH participants showed that CapTune supported creators’ creative control while enhancing viewers’ emotional engagement with content. Our findings also reveal trade-offs between information richness and cognitive load, tensions between interpretive and descriptive representations of sound, and the context-dependent nature of caption preferences.

System Architecture

CapTune is organized as a pipeline from creator specification to viewer-specific caption generation. Given a video and its caption file, the system first isolates the non-speech captions and extracts audiovisual context for each one: an audiovisual language model processes the surrounding video segment and its audio track to produce a scene-based description. In the creator-facing interface, a large language model estimates where each original caption lies along two dimensions, level of detail and expressiveness, and the creator generates a lower and an upper anchor caption that bound the range of acceptable transformations. These anchors, the scene descriptions, and the video metadata are packaged with the original captions. In the viewer-facing interface, viewers specify their preferences through direct manipulation or natural language, including their desired position within the creator-defined range, a sound representation method, and genre alignment. At transformation time, the system expresses each request relative to the anchors, quantifying how far the target lies between the lower and upper anchor captions and how much it departs from the current caption, and passes these ratios to the language model together with the anchor captions and scene context. The model therefore interpolates between concrete, creator-approved references rather than generating captions from scratch, keeping every transformation within the creator’s intent.

System diagram. A video frame captioned '[Cooing]' and its audio track feed two components: a Creator-facing Interface, where the creator generates anchor captions and creates video metadata, and an Audiovisual LM (e.g., AV-Flamingo) that produces scene-based video descriptions. These outputs, together with preferences from a Viewer-facing Interface, are passed to Large Language Models, which output a transformed caption: '00:02:50 → 00:04:20 [Gentle, rhythmic cooing]'.

BibTeX

@inproceedings{huang2025captune,
  title     = {CapTune: Adapting Non-Speech Captions With Anchored Generative Models},
  author    = {Huang, Jeremy Zhengqi and de Lacerda Pataca, Calu{\~a} and Wu, Liang-Yuan and Jain, Dhruv},
  booktitle = {Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility},
  series    = {ASSETS '25},
  year      = {2025},
  numpages  = {18},
  location  = {Denver, CO, USA},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3663547.3746346},
  url       = {https://doi.org/10.1145/3663547.3746346}
}