Marc Roth

Kyle Waters, Lucas Nuzzi, Tadhg Looram et al.

AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written benchmarks have proven effective at measuring AI reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted multimodal Terminus-2 agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains.

Panagiotis Aivasiliotis, Andreas Göbel, Matthias Lanzinger et al.

The Weisfeiler--Leman (WL) algorithm is one of the most influential heuristics for the graph isomorphism problem. The expressive power of WL has been extensively studied in the contexts of descriptive complexity, logics, graph neural networks, and the theory of homomorphism indistinguishabily. Notably, two graphs are indistinguishable by the $k$-dimensional WL algorithm if and only if they are indistinguishable by homomorphism-counts from graphs of treewidth at most $k$. An intrinsic question is to find a natural version of the WL algorithm for relational structures of higher arity admitting an equivalent characterisation via homomorphism indistinguishability along bounded generalised hypertree width (GHW). Scheidt and Schweikardt solved this for $k=1$ by defining the RCR algorithm and showing indistinguishability from $α$-acyclic structures. In this work, we resolve this for all $k\ge1$: we develop $k$-RCR and show that two structures $\mathcal{A}$ and $\mathcal{B}$ are insdistinguishable by $k$-RCR if and only if they have the same homomorphism-counts from all structures $\mathcal{C}$ of generalised hypertreewidth $\le k$. Moreover, we introduce a ``fractional'' version of $k$-RCR and show that two structures are insdistinguishable by fractional $k$-RCR if and only if they have the same homomorphism-counts from all structures with (a variant of) fractional hypertreewidth at most $k$. Last, we develop $k$-HyperOWL, the first relational WL algorithm operating directly on a relational structure. We show that $k$-HyperOWL is as expressive as $k$-RCR and that, given a structure $\mathcal{A}$, $k$-HyperOWL can compute $t$ iterative refinements in time $O(t|\mathcal{A}|^{k+1})$. Moreover, the colouring produced by $k$-HyperOWL can be used as a constructive preprocessing routine for counting homomorphisms from structures of generalised hypertreewidth $\le k$.

Madhumitha Krishnakumar, Marc Roth

Introduced by Lee, Ko, and Shin (VLDB 2020), a hypergraph motif is a connected subhypergraph consisting of three hyperedges whose intersections satisfy a prescribed pattern. Such patterns are represented by Venn diagrams $\mathcal{V}\in\{0,1\}^7$, indicating which of the seven regions determined by three sets must be empty or non-empty. Lee et al. designed and implemented exact and approximate algorithms for counting, in a hypergraph $G$, the motifs specified by $\mathcal{V}$; their algorithms run in worst-case cubic time in the number of hyperedges of $G$. This cubic worst case can occur even for hypergraphs of bounded rank, and already for $2$-uniform hypergraphs, that is, for simple graphs. In this work, we give a complete fine-grained picture of the parameterised complexity of exact hypergraph motif counting with respect to the rank of the input hypergraph. We use $\tilde{O}$ to hide polylogarithmic factors in the input size. First, we show that every Venn diagram $\mathcal{V}$ admits an exact counting algorithm running in FPT-near-quadratic time, \[ f(\mathsf{rank}(G))\cdot \tilde{O}(|E(G)|^2), \] for some computable function $f$. Second, we precisely characterise when this can be improved to FPT-near-linear time. We prove that such an algorithm exists exactly for the degenerate Venn diagrams, namely those that force one of the three hyperedges to be fully contained in another. For all non-degenerate Venn diagrams, we show that no FPT-near-linear-time algorithm exists unless either the Triangle Hypothesis or the Hyperclique Hypothesis fails. Exact hypergraph motif counting is thus always fixed-parameter near-quadratic in the rank, and the degenerate Venn diagrams are precisely the cases admitting fixed-parameter near-linear time.