Evaluating user performance with RAG-based generative AI: A scenario-based experiment on AI-assisted information retrieval
Citations

WEB OF SCIENCE

0
Citations

SCOPUS

0

초록

Recent advances in generative artificial intelligence (GenAI) have enabled users to interact with AI models through conversational interfaces. However, because these models rely on pre-trained and static datasets, they often struggle to provide accurate or current information, particularly in specialized domains. Retrieval-augmented generation (RAG) addresses this limitation by integrating large language models with access to external, real-time data sources. While prior research has largely emphasized system-level evaluations, limited attention has been given to user-centered performance outcomes. This study bridges that gap by investigating how RAG-based tools affect user performance in information-seeking tasks. Guided by Task–technology fit (TTF) theory, we conducted a 2 × 2 scenario-based experiment manipulating RAG functionality and task complexity. Participants completed search tasks using either standard LLMs or RAG-enhanced systems. User performance was assessed in terms of accuracy, completeness, and relevance. The findings are expected to offer empirical insights into the practical value of RAG systems and inform the design of GenAI tools for knowledge-intensive applications. © 2026 Elsevier Ltd

키워드

Generative artificial intelligenceInformation retrievalRetrieval-augmented generationTask-technology fitUser performanceTASK COMPLEXITYINTERRATER RELIABILITYRELEVANCE
제목
Evaluating user performance with RAG-based generative AI: A scenario-based experiment on AI-assisted information retrieval
저자
Sagynbayeva, AktilekPyo, AjinYoon, Sang-HyeakYang, Sung-Byung
DOI
10.1016/j.chb.2026.108952
발행일
2026-07
유형
Article
저널명
Computers in Human Behavior
180
페이지
1 ~ 12