LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
This framework addresses the critical need for accessible and privacy-preserving tools for non-technical domain experts and compliance officers to audit LLMs, which is essential for responsible AI deployment and regulatory compliance.
This paper introduces LLM-FACETS, an open-source framework designed to help non-technical practitioners evaluate the transparency and accountability of Large Language Models. It provides a browser-accessible interface and a plugin architecture, ensuring data privacy by keeping deterministic metric calculations on a self-hosted server and giving users full control over external API calls for LLM-judge metrics. The framework operationalizes transparency via token-level log-probability visualization, multi-judge consensus, and RAG Triad metrics to detect hallucinations.
Assessing whether Large Language Models outputs are factually grounded, epistemically calibrated, and methodologically reproducible is a prerequisite for responsible AI deployment. Yet auditing LLMs remains inaccessible to non-technical practitioners: existing tools require programming expertise and non-trivial environment setup, and cloud-hosted platforms transmit evaluation data to external services, creating barriers for domain experts and compliance officers legally responsible for AI oversight. We introduce LLM-FACETS (LLM FActuality Cross-EvaluaTion System): an open-source framework with a browser-accessible interface and a plugin architecture, structured around three practitioner profiles (technical experts, domain experts, compliance officers) that mirror the stakeholder categories identified in the EU AI Act and the NIST AI Risk Management Framework. The architecture makes data flows explicit: deterministic metrics (BLEU, ROUGE, BERTScore) run entirely within the self-hosted server with no outbound transmission; LLM-judge metrics contact external APIs explicitly, with users retaining full credential control. The framework operationalizes transparency through three mechanisms: token-level log-probability visualization for epistemic uncertainty, multi-judge consensus to mitigate judge bias, and RAG Triad metrics (Faithfulness, Answer Relevance, Context Relevance) to detect and localize hallucinations. A plugin architecture allows any new metric or dataset to be integrated without modifying the evaluation pipeline. The open-source implementation enables cross-checking across multiple metrics targeting the same property, ensuring reproducibility and decoupling AI accountability from the teams building the systems assessed. We verify the framework through cross-validation of 18 metric implementations against canonical reference libraries.