STAR: Statistical Tests with Auditable Results
This addresses the issue of data dredging and trust in scientific claims for researchers and institutions, though it is incremental as it builds on existing cryptographic and secure computation techniques.
The authors tackled the problem of p-hacking and false discoveries in scientific studies by developing STAR, a system that provides mathematically provable guarantees for false discovery control and generates auditable certificates for statistical tests, demonstrating practicality in real-world scenarios.
We present STAR: a novel system aimed at solving the complex issue of "p-hacking" and false discoveries in scientific studies. STAR provides a concrete way for ensuring the application of false discovery control procedures in hypothesis testing, using mathematically provable guarantees, with the goal of reducing the risk of data dredging. STAR generates an efficiently auditable certificate which attests to the validity of each statistical test performed on a dataset. STAR achieves this by using several cryptographic techniques which are combined specifically for this purpose. Under-the-hood, STAR uses a decentralized set of authorities (e.g., research institutions), secure computation techniques, and an append-only ledger which together enable auditing of scientific claims by 3rd parties and matches real world trust assumptions. We implement and evaluate a construction of STAR using the Microsoft SEAL encryption library and SPDZ multi-party computation protocol. Our experimental evaluation demonstrates the practicality of STAR in multiple real world scenarios as a system for certifying scientific discoveries in a tamper-proof way.