Towards Measurement Theory for Artificial Intelligence
This foundational work aims to benefit researchers, practitioners, and regulators by providing a unified framework for measuring AI, though it is incremental as it outlines a program rather than presenting a fully developed theory.
The paper tackles the lack of a formal measurement theory for AI, proposing a program to develop such a theory to enable comparisons between systems, connect AI evaluations with risk analysis, and highlight the contingency of AI capability on measurement choices.
We motivate and outline a programme for a formal theory of measurement of artificial intelligence. We argue that formalising measurement for AI will allow researchers, practitioners, and regulators to: (i) make comparisons between systems and the evaluation methods applied to them; (ii) connect frontier AI evaluations with established quantitative risk analysis techniques drawn from engineering and safety science; and (iii) foreground how what counts as AI capability is contingent upon the measurement operations and scales we elect to use. We sketch a layered measurement stack, distinguish direct from indirect observables, and signpost how these ingredients provide a pathway toward a unified, calibratable taxonomy of AI phenomena.