ACHO: Adaptive Conformal Hyperparameter Optimization
This work addresses hyperparameter optimization for machine learning practitioners by offering a more flexible method, though it appears incremental as it builds on existing conformal prediction ideas.
The paper tackled the problem of hyperparameter search frameworks relying on strict distributional assumptions by proposing an optimization framework based on conformal confidence intervals, which achieved satisfactory interval coverage and superior tuning performance compared to random search in benchmarks on random forests and convolutional neural networks.
Several novel frameworks for hyperparameter search have emerged in the last decade, but most rely on strict, often normal, distributional assumptions, limiting search model flexibility. This paper proposes a novel optimization framework based on upper confidence bound sampling of conformal confidence intervals, whose weaker assumption of exchangeability enables greater choice of search model architectures. Several such architectures were explored and benchmarked on hyperparameter search of random forests and convolutional neural networks, displaying satisfactory interval coverage and superior tuning performance to random search.