Evaluating user performance with RAG-based generative AI: A scenario-based experiment on AI-assisted information retrieval

Abstract

Recent advances in generative artificial intelligence (GenAI) have enabled users to interact with AI models through conversational interfaces. However, because these models rely on pre-trained and static datasets, they often struggle to provide accurate or current information, particularly in specialized domains. Retrieval-augmented generation (RAG) addresses this limitation by integrating large language models with access to external, real-time data sources. While prior research has largely emphasized system-level evaluations, limited attention has been given to user-centered performance outcomes. This study bridges that gap by investigating how RAG-based tools affect user performance in information-seeking tasks. Guided by Task–technology fit (TTF) theory, we conducted a 2 × 2 scenario-based experiment manipulating RAG functionality and task complexity. Participants completed search tasks using either standard LLMs or RAG-enhanced systems. User performance was assessed in terms of accuracy, completeness, and relevance. The findings are expected to offer empirical insights into the practical value of RAG systems and inform the design of GenAI tools for knowledge-intensive applications. © 2026 Elsevier Ltd