Scorio.jl: A Julia package for ranking stochastic responses
This work provides a tool for researchers and practitioners in machine learning and data analysis to compare ranking methods, but it is incremental as it builds on existing ranking techniques within a new software package.
The authors introduced Scorio.jl, a Julia package for evaluating and ranking systems based on repeated responses to shared tasks, offering a unified interface for various ranking methods and demonstrating its utility through pilot experiments on synthetic data, runtime scaling, and stability.
Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks. It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions. We describe the package design, position it relative to existing Julia tools, and report pilot experiments on synthetic rank recovery, stability under limited trials, and runtime scaling.