PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluation
This addresses the burden for resource-constrained practitioners dealing with multiple AI policies, offering a scalable automated framework for AI governance.
The paper tackles the problem of multi-policy AI compliance evaluation by presenting PASTA, a scalable tool that integrates innovations like a model-card format and LLM-powered engine, achieving expert-aligned judgments (ρ ≥ .626) and evaluating five policies in under two minutes at about $3.
AI compliance is becoming increasingly critical as AI systems grow more powerful and pervasive. Yet the rapid expansion of AI policies creates substantial burdens for resource-constrained practitioners lacking policy expertise. Existing approaches typically address one policy at a time, making multi-policy compliance costly. We present PASTA, a scalable compliance tool integrating four innovations: (1) a comprehensive model-card format supporting descriptive inputs across development stages; (2) a policy normalization scheme; (3) an efficient LLM-powered pairwise evaluation engine with cost-saving strategies; and (4) an interface delivering interpretable evaluations via compliance heatmaps and actionable recommendations. Expert evaluation shows PASTA's judgments closely align with human experts ($ρ\geq .626$). The system evaluates five major policies in under two minutes at approximately \$3. A user study (N = 12) confirms practitioners found outputs easy-to-understand and actionable, introducing a novel framework for scalable automated AI governance.