Seon Gyeom Kim

Jae Young Choi, Seon Gyeom Kim, Hyungjun Yoon et al.

Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.

Seon Gyeom Kim, Jae Young Choi, Ryan Rossi et al.

The field of Multimodal Large Language Models (MLLMs) has made remarkable progress in visual understanding tasks, presenting a vast opportunity to predict the perceptual and emotional impact of charts. However, it also raises concerns, as many applications of LLMs are based on overgeneralized assumptions from a few examples, lacking sufficient validation of their performance and effectiveness. We introduce Chart-to-Experience, a benchmark dataset comprising 36 charts, evaluated by crowdsourced workers for their impact on seven experiential factors. Using the dataset as ground truth, we evaluated capabilities of state-of-the-art MLLMs on two tasks: direct prediction and pairwise comparison of charts. Our findings imply that MLLMs are not as sensitive as human evaluators when assessing individual charts, but are accurate and reliable in pairwise comparisons.

Reuben Luera, Ryan Rossi, Franck Dernoncourt et al.

In this work, we research user preferences to see a chart, table, or text given a question asked by the user. This enables us to understand when it is best to show a chart, table, or text to the user for the specific question. For this, we conduct a user study where users are shown a question and asked what they would prefer to see and used the data to establish that a user's personal traits does influence the data outputs that they prefer. Understanding how user characteristics impact a user's preferences is critical to creating data tools with a better user experience. Additionally, we investigate to what degree an LLM can be used to replicate a user's preference with and without user preference data. Overall, these findings have significant implications pertaining to the development of data tools and the replication of human preferences using LLMs. Furthermore, this work demonstrates the potential use of LLMs to replicate user preference data which has major implications for future user modeling and personalization research.

Jaehoon Pyun, Younggeol Cho, Seon Gyeom Kim et al.

We propose an interaction technique called "Point & Select." It enables a driver to directly enter a point of interest (POI) into the in-vehicle infotainment system while driving in a city. Point & Select enables the driver to directly indicate with a finger, identify, adjust (if required), and finally confirm the POI on the screen by using buttons on the steering wheel. Based on a comparative evaluation of two conditions (driving-only and driving with input-task) on a simulator, we demonstrated the feasibility of the interaction in the driving context from the perspective of driver performance and interaction usability at speeds of 30, 50, and 70 km/h. Although the interaction usage and speed partially affected the driver's mental load, all the participants drove at an acceptable level in each condition. They carried out the task successfully with a success rate of 96.9% and task completion time of 1.82 seconds on average.