MTS is the department’s Mind, Technology, and Society speaker series. It is hosted by a different faculty member each semester. Founded by a generous gift from Professors Robert Glushko and Pamela Samuelson, MTS brings researchers and industry professionals from across the globe to present a variety of interdisciplinary work in cognitive science. See our UCMerced CogSci youtube channel for videos of past MTS talks!
CIS graduate students, faculty, and staff, and all who are interested are invited! Members of other departments at UC Merced as well as the general public are encouraged to attend. (Note: current CIS Ph.D. students are required to attend MTS each semester in residence, to fulfill their COGS 250 course requirement).
Dr Elisa Kreiss talk "From Images to Words: The Problem of Information Selection" will be from 2-3:30pm in SSM 104
Abstract: Visual media increasingly dominates modern (online) communication, but this heavily visual landscape creates substantial barriers for those who cannot access it. To facilitate non-visual accessibility, we need to translate visual information into linguistic descriptions. What appears straightforward on the surface is in fact a fundamental communicative challenge: when moving between modalities, what exact information should be selected and conveyed? While modern vision–language models achieve ostensibly "superhuman" benchmark results on image-to-text tasks, they frequently underperform in real communicative settings. I argue that this gap reflects a deeper problem of information selection rooted in communicative principles: effective image descriptions require not just stating what is literally true about an image, but deciding what should be said given a specific communicative goal and audience. Drawing on cognitive experiments with sighted participants and people who are blind or have low vision, I show that communicative goals strongly shape which visual information people choose to express, and that current AI tasks and evaluation methods fail to capture this nuance. I conclude by discussing recent work in which we start to quantify informativity in image descriptions, and how information selection varies across cultures, highlighting a broader research agenda for understanding and modeling cross-modal communication.
Bio: Elisa Kreiss is an Assistant Professor of Communication & Computer Science (by courtesy) at UCLA. She is the director of the Coalas (Computation and Language for Society) Lab and a co-director of the UCLA NLP Group. Previously, she completed a PhD in Linguistics at Stanford. Elisa investigates how we produce and understand language situated in the visual world. Her work combines tools from natural language processing, psycholinguistics, and human-computer interaction to advance our understanding of how communicative context shapes language use. Her research has direct applications to image accessibility – the challenge of (automatically) generating image descriptions for blind and low vision users. Elisa’s work has been supported by the UCLA Society of Hellman Fellows Award, several Google Research Awards, the National Science Foundation, Stanford’s Human-centered AI initiative, and Stanford’s Accelerator for Learning. Her work has been recognized with a Best Paper Award at the NLP for Positive Impact Workshop at EMNLP 2024.
For more information or to sign up for email announcements, please contact the talk series organizer: cis-mts-lead@lists.ucmerced.edu.
