MindMirror: A Local-First Multimodal State-Aware Support System for Digital Workers
For digital workers, this paper presents a privacy-preserving, user-controllable tool for state reflection and supportive interaction, though the contribution is incremental as it combines existing techniques (emotion recognition, LLMs) in a new application context.
MindMirror is a local-first multimodal system that helps digital workers manage fatigue, anxiety, and task blockage by integrating facial expression recognition, text/speech input, and LLM-based response generation. The fine-tuned emotion recognition model achieved 94.49% accuracy (from 59.66%) on a benchmark, and formative user feedback with six participants indicated positive reception of the local-first design and structured reflection workflow.
Digital workers often experience fatigue, anxiety, reduced attention, and task blockage during prolonged computer-based work. Existing productivity tools mainly focus on task completion, while general-purpose AI chatbots require users to formulate clear prompts before receiving useful help. This paper presents MindMirror, a local-first multimodal state-aware support system for digital workers. MindMirror integrates camera-based facial expression cues, text input, optional speech interaction, structured blockage reflection, local large language model (LLM)-based response generation, and daily/weekly review reports. The system forms a closed workflow of state checking, manual correction, structured articulation, suggestion generation, and state review. The current prototype follows a local-first design, while optional speech services may rely on third-party APIs when enabled. It is implemented with a Web frontend, Flask backend, an emotion recognition model, an Ollama-hosted Qwen model, Chart.js visualization, and local JSON/LocalStorage records. We evaluate the emotion recognition module on an independent seven-class image-level facial expression benchmark containing 6,767 images. The fine-tuned Hugging Face model improves accuracy from 59.66% to 94.49% over a non-fine-tuned checkpoint baseline, an absolute gain of 34.83 percentage points. We further validate the prototype through endpoint-level reliability tests, voice-interaction latency tests, and a small formative user feedback study with six digital workers. Results suggest that users value the local-first design, manual correction mechanism, and structured reflection workflow. MindMirror is not intended for psychological diagnosis; instead, it serves as a lightweight, user-controllable tool for state reflection and supportive interaction.