Video In Sentences Out
This addresses the challenge of automated video captioning for applications in accessibility or surveillance, but it appears incremental as it builds on existing event recognition techniques.
The authors tackled the problem of generating sentential descriptions from video by recognizing events, objects, and their roles, resulting in a system that outputs structured sentences detailing actions, participants, and spatial relations.
We present a system that produces sentential descriptions of video: who did what to whom, and where and how they did it. Action class is rendered as a verb, participant objects as noun phrases, properties of those objects as adjectival modifiers in those noun phrases,spatial relations between those participants as prepositional phrases, and characteristics of the event as prepositional-phrase adjuncts and adverbial modifiers. Extracting the information needed to render these linguistic entities requires an approach to event recognition that recovers object tracks, the track-to-role assignments, and changing body posture.