Shaun Kane

Jon E. Froehlich, Jared Hwang, Zeyu Wang et al. · uw

Interactive digital maps have revolutionized how people travel and learn about the world; however, they rely on pre-existing structured data in GIS databases (e.g., road networks, POI indices), limiting their ability to address geo-visual questions related to what the world looks like. We introduce our vision for Geo-Visual Agents--multimodal AI agents capable of understanding and responding to nuanced visual-spatial inquiries about the world by analyzing large-scale repositories of geospatial images, including streetscapes (e.g., Google Street View), place-based photos (e.g., TripAdvisor, Yelp), and aerial imagery (e.g., satellite photos) combined with traditional GIS data sources. We define our vision, describe sensing and interaction approaches, provide three exemplars, and enumerate key challenges and opportunities for future work.

Jon E. Froehlich, Alexander Fiannaca, Nimer Jaber et al. · uw

Interactive streetscape mapping tools such as Google Street View (GSV) and Meta Mapillary enable users to virtually navigate and experience real-world environments via immersive 360° imagery but remain fundamentally inaccessible to blind users. We introduce StreetReaderAI, the first-ever accessible street view tool, which combines context-aware, multimodal AI, accessible navigation controls, and conversational speech. With StreetReaderAI, blind users can virtually examine destinations, engage in open-world exploration, or virtually tour any of the over 220 billion images and 100+ countries where GSV is deployed. We iteratively designed StreetReaderAI with a mixed-visual ability team and performed an evaluation with eleven blind users. Our findings demonstrate the value of an accessible street view in supporting POI investigations and remote route planning. We close by enumerating key guidelines for future work.

Shaun Kane, Richard Ladner, Clayton Lewis

Access to content and services online is increasingly important for everyone, including people with disabilities. National commitments, including the Americans with Disabilities Act, and international resolutions, including the United Nations Declaration of the Rights of Persons with Disabilities, call for work to ensure that people with disabilities can participate fully in the online world. Gains in education, employment and health, as well as in civic engagement, social participation, and personal independence will follow from enhanced inclusion online. Research in many areas of computer science, including recognition technology, natural language processing, personalization, software architecture, and others, is needed to secure these benefits. Organizing this research calls for partnerships among academic researchers, federal agencies, and commercial organizations, as well as effective division of labor and cooperation between computer scientists, behavioral scientists, advocacy groups, and consumers.