Lucian Popa

Yannis Katsis, Sara Rosenthal, Kshitij Fadnis et al. · ibm-research

Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.

Marina Danilevsky, Kristjan Greenewald, Chulaka Gunasekara et al. · ibm-research

In the developer community for large language models (LLMs), there is not yet a clean pattern analogous to a software library, to support very large scale collaboration. Even for the commonplace use case of Retrieval-Augmented Generation (RAG), it is not currently possible to write a RAG application against a well-defined set of APIs that are agreed upon by different LLM providers. Inspired by the idea of compiler intrinsics, we propose some elements of such a concept through introducing a library of LLM Intrinsics for RAG. An LLM intrinsic is defined as a capability that can be invoked through a well-defined API that is reasonably stable and independent of how the LLM intrinsic itself is implemented. The intrinsics in our library are released as LoRA adapters on HuggingFace, and through a software interface with clear structured input/output characteristics on top of vLLM as an inference platform, accompanied in both places with documentation and code. This article describes the intended usage, training details, and evaluations for each intrinsic, as well as compositions of multiple intrinsics.

Bogdan Groza, Lucian Popa, Pal-Stefan Murvay

Previous research works have endorsed the use of delays and clock skews for detecting intrusions or fingerprinting ECUs on the CAN bus. Similar techniques have been also proposed for establishing a time-covert cryptographic authentication channel, in this way cleverly removing the need for cryptographic material inside the limited payload of CAN frames. The main shortcoming of such works is the limited security level that can be achieved under normal CAN-bus traffic. In this work we endeavour to test the limits of the achievable security level by relying on optimization algorithms for scheduling CAN frames. Under practical bus allocations that are based on real-world scenarios, we are able to extract around 4--5 bits of authentication data from each frame which leads to an efficient intrusion detection and authentication mechanism. By accumulating covert channel data over several consecutive frames, we can achieve higher security levels that are in line with current security demands. To prove the correctness of our approach, we present experiments on state-of-the-art automotive-grade controllers (Infineon Aurix) and bus measurements with the use of industry standard tools, i.e., CANoe.