Isabela Maria Carneiro Albuquerque

Felipe Maia Polo, Aida Nematzadeh, Virginia Aglietti et al.

Moving beyond evaluations that collapse performance across heterogeneous prompts toward fine-grained evaluation at the prompt level, or within relatively homogeneous subsets, is necessary to diagnose generative models' strengths and weaknesses. Such fine-grained evaluations, however, suffer from a data bottleneck: human gold-standard labels are too costly at this scale, while automated ratings are often misaligned with human judgment. To resolve this challenge, we propose a novel statistical model based on tensor factorization that merges cheap autorater data with a limited set of human gold-standard labels. Specifically, our approach uses autorater scores to pretrain latent representations of prompts and generative models, and then aligns those pretrained representations to human preferences using a small calibration set. This sample-efficient methodology is robust to autorater quality, more accurately predicts human preferences on a per-prompt basis than standard baselines, and provides tight confidence intervals for key statistical parameters of interest. We also showcase the practical utility of our method by constructing granular leaderboards based on prompt qualities and by estimating model performance solely from autorater scores, eliminating the need for additional human annotations.

F. B. Lima Neto, I. M. C. Albuquerque, J. B. Monteiro Filho

Optimization problems with more than one objective consist in a very attractive topic for researchers due to its applicability in real-world situations. Over the years, the research effort in the Computational Intelligence field resulted in algorithms able to achieve good results by solving problems with more than one conflicting objective. However, these techniques do not exhibit the same performance as the number of objectives increases and become greater than 3. This paper proposes an adaptation of the metaheuristic Fish School Search to solve optimization problems with many objectives. This adaptation is based on the division of the candidate solutions in clusters that are specialized in solving a single-objective problem generated by the decomposition of the original problem. For this, we used concepts and ideas often employed by state-of-the-art algorithms, namely: (i) reference points and lines in the objectives space; (ii) clustering process; and (iii) the decomposition technique Penalty-based Boundary Intersection. The proposed algorithm was compared with two state-of-the-art bio-inspired algorithms. Moreover, a version of the proposed technique tailored to solve multi-modal problems was also presented. The experiments executed have shown that the performance obtained by both versions is competitive with state-of-the-art results.