Astitwa Sarthak Lathe

Benjamin Feuer, Chiung-Yi Tseng, Astitwa Sarthak Lathe et al.

LLM-judged benchmarks are increasingly used to evaluate complex model behaviors, yet their design introduces failure modes absent in conventional ground-truth based benchmarks. We argue that without tight objectives and verifiable constructions, benchmark rankings can produce high-confidence rankings that are in fact largely noise. We introduce two mechanisms to diagnose these issues. Schematic adherence quantifies how much of a judge's overall verdict is explained by the explicit evaluation schema, revealing unexplained variance when judges deviate from their own rubric. Psychometric validity aggregates internal consistency and discriminant validity signals to quantify irreducible uncertainty in any benchmarking run. Applying these tools to Arena-Hard Auto, we find severe schema incoherence and factor collapse across popular judges: for example, unexplained variance exceeding 90 percent for DeepSeek-R1-32B and factor correlations above 0.93 for most criteria. We also show that the ELO-style aggregation used by Arena-Hard Auto collapses and masks genuine ranking uncertainty. Our results highlight design failures that undermine validity and offer actionable principles for building better-scoped, reliability-aware LLM-judged benchmarks. We released our code and dataset at https://github.com/penfever/judgment-to-noise

Kavita Dubey, Anant Agarwal, Astitwa Sarthak Lathe et al.

Heart Failure is a major component of healthcare expenditure and a leading cause of mortality worldwide. Despite higher inter-rater variability, endomyocardial biopsy (EMB) is still regarded as the standard technique, used to identify the cause (e.g. ischemic or non-ischemic cardiomyopathy, coronary artery disease, myocardial infarction etc.) of unexplained heart failure. In this paper, we focus on identifying cardiomyopathy as ischemic or non-ischemic. For this, we propose and implement a new unified architecture comprising CNN (inception-V3 model) and bidirectional LSTM (BiLSTM) with self-attention mechanism to predict the ischemic or non-ischemic to classify cardiomyopathy using histopathological images. The proposed model is based on self-attention that implicitly focuses on the information outputted from the hidden layers of BiLSTM. Through our results we demonstrate that this framework carries a high learning capacity and is able to improve the classification performance.