CapTune: Adapting Non-Speech Captions With Anchored Generative Models

Jeremy Zhengqi Huang, Caluã De Lacerda Pataca, Liang-Yuan Wu Dhruv Jain
University of Michigan, Rochester Institute of Technology
International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2025)
Preprint Code (Available Soon) arXiv
Placeholder description for teaser figure. Replace with detailed image description.

Overview of the CapTune system. (1) Non-speech caption transformations are mapped within a two-dimensional space defined by the Level of Detail and Expressiveness. (2) (2) Captioners define the bounds of this space for a given caption file. (3) Viewers select their preferences within the defined range. (4) Captions can be transformed using one of three sound representation methods: source-focused, onomatopoeia, or sensory quality-based. (5) Captions can also be aligned with genre tone and style for narrative coherence.

Abstract

Non-speech captions are essential to the video experience of deaf and hard of hearing (DHH) viewers, yet conventional approaches often overlook the diversity of their preferences. We present CapTune, a system that enables customization of non-speech captions based on DHH viewers' needs while preserving creator intent. CapTune allows caption authors to define safe transformation spaces using concrete examples and empowers viewers to personalize captions across four dimensions: level of detail, expressiveness, sound representation method, and genre alignment. Evaluations with seven caption creators and twelve DHH participants showed that CapTune supported creators' creative control while enhancing viewers' emotional engagement with content. Our findings also reveal trade-offs between information richness and cognitive load, tensions between interpretive and descriptive representations of sound, and the context-dependent nature of caption preferences.

System Architecture

Placeholder description for system architecture figure. Replace with detailed image description.

Given the video segment, captions, and the audio track, CapTune's Creator Tool empowers creators to prepare video metadata and anchor captions that define the safe transformation boundary that guides generative AI behaviors, while Context Extractor summarizes the segment' audio. The Viewer Client then enables viewers to specify their viewing preferences. These preferences, together with the anchor captions and context information, are passed to a generative model that rewrite the caption for that time span while preserving the narrative intent. The rewritten caption is then displayed in sync with the video playback.

Video Presentation

BibTeX

@article{captune_2025,
  title={CapTune: Adapting Non-Speech Captions With Anchored Generative Models},
  author={Jeremy Zhengqi Huang, Caluã de Lacerda Pataca, Liang-Yuan Wu, Dhruv Jain},
  journal={The 27th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2025)},
  year={2025},
  url={https://arxiv.org/abs/2508.19971}
}