StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI
This addresses the problem of inaccessible digital street navigation for blind users, representing a novel application rather than an incremental improvement.
The researchers tackled the inaccessibility of street view tools like Google Street View for blind users by developing StreetReaderAI, a context-aware multimodal AI tool that enables virtual exploration and route planning, with an evaluation involving eleven blind users showing its value in POI investigations and remote route planning.
Interactive streetscape mapping tools such as Google Street View (GSV) and Meta Mapillary enable users to virtually navigate and experience real-world environments via immersive 360° imagery but remain fundamentally inaccessible to blind users. We introduce StreetReaderAI, the first-ever accessible street view tool, which combines context-aware, multimodal AI, accessible navigation controls, and conversational speech. With StreetReaderAI, blind users can virtually examine destinations, engage in open-world exploration, or virtually tour any of the over 220 billion images and 100+ countries where GSV is deployed. We iteratively designed StreetReaderAI with a mixed-visual ability team and performed an evaluation with eleven blind users. Our findings demonstrate the value of an accessible street view in supporting POI investigations and remote route planning. We close by enumerating key guidelines for future work.