Penetrative AI: Making LLMs Comprehend the Physical World
This addresses the challenge of integrating common-sense human knowledge into LLMs for real-world applications, enabling new uses in cyber-physical systems, though it appears incremental as it builds on existing LLM capabilities.
The paper tackles the problem of extending Large Language Models (LLMs) to interact with and reason about the physical world using IoT sensors and actuators, termed 'Penetrative AI', and finds that LLMs like ChatGPT show considerable proficiency in interpreting sensor data and reasoning about physical tasks.
Recent developments in Large Language Models (LLMs) have demonstrated their remarkable capabilities across a range of tasks. Questions, however, persist about the nature of LLMs and their potential to integrate common-sense human knowledge when performing tasks involving information about the real physical world. This paper delves into these questions by exploring how LLMs can be extended to interact with and reason about the physical world through IoT sensors and actuators, a concept that we term "Penetrative AI". The paper explores such an extension at two levels of LLMs' ability to penetrate into the physical world via the processing of sensory signals. Our preliminary findings indicate that LLMs, with ChatGPT being the representative example in our exploration, have considerable and unique proficiency in employing the embedded world knowledge for interpreting IoT sensor data and reasoning over them about tasks in the physical realm. Not only this opens up new applications for LLMs beyond traditional text-based tasks, but also enables new ways of incorporating human knowledge in cyber-physical systems.