Benchmarking Retrieval-Augmented Generation: Challenging Domain-specific Datasets Generation Method

Retrieval-Augmented Generation (RAG) enhances the answering capabilities of large language models (LLMs) by leveraging knowledge retrieved from external corpora, significantly mitigating hallucination issues in LLMs. RAG systems have been widely applied across various domains. However, existing RAG systems often rely on LLMs and retrievers trained on general-purpose datasets, which perform poorly when dealing with domain-specific tasks. Furthermore, current RAG benchmark datasets fail to cover all domains, posing challenges in evaluating RAG systems' performance in specific domains. To address these limitations, we propose a challenging domain-specific datasets generation method for RAG benchmarking (RAGCDs). This method automatically generates domain-specific RAG benchmark datasets with retrieval challenges and answering difficulties based on any given corpus. The generated datasets enable rigorous evaluation of the performance of RAG systems in specific domains. We detail the process of constructing domain-specific benchmark datasets using RAGCDs and select corpora from two distinct fields, X-ray Photoelectron Spectroscopy (XPS) and medical research papers, to create the CDs-xps and CDs-medical datasets. This demonstrates the generality of our method. Our experiments validate that the RAGCDs-generated datasets exhibit exceptional quality. In terms of retrieval, the datasets pose challenges comparable to those of specialized retrieval benchmarks. Regarding answering difficulty, even the most advanced LLMs struggle significantly, achieving a score ratio of only 0.391 and 0.471 in the absence of retrieval support. We hope that RAGCDs will effectively promote the development and optimization of RAG systems tailored for specific domains. Implementation and data are available at https://github.com/xiaowuxiaowu1/RAGCDs.