Can LLMs Test Terminal User Interfaces?
This paper provides the first dedicated testing methodology and benchmark for TUI applications, which are common in developer tools but lack testing support, offering practical insights for developers and researchers.
The paper investigates automated testing for Terminal User Interfaces (TUIs), finding that only 12% of existing test code exercises the interface and 45% of those tests never send input. They benchmark four LLMs against random exploration on a headless TUI benchmark, showing no model dominates and random is a strong baseline, but LLM guidance is more efficient per interaction and uniquely reaches input-gated faults.
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.