CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography
This addresses the problem of assessing and improving LLMs for cryptography research and development, though it is incremental as it focuses on dataset creation and benchmarking rather than a new method.
The authors tackled the lack of data for evaluating and training large language models (LLMs) in cryptography by creating CryptoQA, a large-scale question-answering dataset with over two million pairs, and benchmarked 15 state-of-the-art LLMs, revealing significant performance deficits, particularly in formal reasoning and mathematical knowledge.
Large language models (LLMs) excel at many general-purpose natural language processing tasks. However, their ability to perform deep reasoning and mathematical analysis, particularly for complex tasks as required in cryptography, remains poorly understood, largely due to the lack of suitable data for evaluation and training. To address this gap, we present CryptoQA, the first large-scale question-answering (QA) dataset specifically designed for cryptography. CryptoQA contains over two million QA pairs drawn from curated academic sources, along with contextual metadata that can be used to test the cryptographic capabilities of LLMs and to train new LLMs on cryptographic tasks. We benchmark 15 state-of-the-art LLMs on CryptoQA, evaluating their factual accuracy, mathematical reasoning, consistency, referencing, backward reasoning, and robustness to adversarial samples. In addition to quantitative metrics, we provide expert reviews that qualitatively assess model outputs and establish a gold-standard baseline. Our results reveal significant performance deficits of LLMs, particularly on tasks that require formal reasoning and precise mathematical knowledge. This shows the urgent need for LLM assistants tailored to cryptography research and development. We demonstrate that, by using CryptoQA, LLMs can be fine-tuned to exhibit better performance on cryptographic tasks.