Jessica Staddon

Isha Chadalavada, Tianhui Huang, Jessica Staddon

Users increasingly query LLM-enabled web chatbots for help with scam defense. The Consumer Financial Protection Bureau's complaints database is a rich data source for evaluating LLM performance on user scam queries, but currently the corpus does not distinguish between scam and non-scam fraud. We developed an LLM ensemble approach to distinguishing scam and fraud CFPB complaints and describe initial findings regarding the strengths and weaknesses of LLMs in the scam defense context.

Vijay Prakash, Kevin Lee, Arkaprabha Bhattacharya et al.

Answering end user security questions is challenging. While large language models (LLMs) like GPT, LLAMA, and Gemini are far from error-free, they have shown promise in answering a variety of questions outside of security. We studied LLM performance in the area of end user security by qualitatively evaluating 3 popular LLMs on 900 systematically collected end user security questions. While LLMs demonstrate broad generalist ``knowledge'' of end user security information, there are patterns of errors and limitations across LLMs consisting of stale and inaccurate answers, and indirect or unresponsive communication styles, all of which impacts the quality of information received. Based on these patterns, we suggest directions for model improvement and recommend user strategies for interacting with LLMs when seeking assistance with security.

Adwait Nadkarni, William Enck, Somesh Jha et al.

Policy specification for personal user data is a hard problem, as it depends on many factors that cannot be predetermined by system developers. Simultaneously, systems are increasingly relying on users to make security decisions. In this paper, we propose the approach of Policy by Example (PyBE) for specifying user-specific security policies. PyBE brings the benefits of the successful approach of programming by example (PBE) for program synthesis to the policy specification domain. In PyBE, users provide policy examples that specify if actions should be allowed or denied in certain scenarios. PyBE then predicts policy decisions for new scenarios. A key aspect of PyBE is its use of active learning to enable users to correct potential errors in their policy specification. To evaluate PyBE's effectiveness, we perform a feasibility study with expert users. Our study demonstrates that PyBE correctly predicts policies with 76% accuracy across all users, a significant improvement over naive approaches. Finally, we investigate the causes of inaccurate predictions to motivate directions for future research in this promising new domain.