HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding
This work addresses the need for effective multimodal AI in human-centric applications, representing an incremental improvement by specializing existing omni-model architectures for this domain.
The paper tackled the problem of multimodal understanding in human-centric scenes by developing HumanOmni, a large vision-speech language model, which achieved advanced capabilities in tasks like emotion recognition and action understanding through a dataset of over 2.4 million video clips and 14 million instructions.
In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.