Public values in public R&D through natural language processing

Abstract

Given South Korea’s recent 16.6% reduction in research and development (R&D) budgets for 2023, there is an urgent need for more efficient and strategic R&D policy management. Previous studies evaluating R&D outputs have primarily relied on quantitative metrics or expert opinions, making it challenging to assess qualitative outcomes systematically. However, existing literature lacks data-driven, scalable approaches that go beyond counting outputs to systematically capture and interpret the broader societal and economic impacts of public R&D, particularly in emerging and converging technology fields. This study introduces a novel framework for evaluating the social value of public R&D outputs by focusing on public value (PV) as a critical metric. The approach integrates conventional R&D outputs with external data sources, leveraging advanced natural language processing and deep learning techniques. Specifically, NLP-based document parsing was applied to segment full-text patents into subsections (e.g., Background, Technical Problem, Advantageous Effects), and a large language model (LLM) classifier was used to categorize each segment into predefined public value types. Utilizing a comprehensive dataset of 1642 patents and 422 news columns, the patents were sourced from the National Technology Information Service database of publicly funded AI R&D projects and matched with full-text USPTO records, while news columns were selected from AI-related opinion pieces to capture societal perspectives. Focusing on patents ensures consistency but limits representativeness, classification depends on LLM performance, and PV subjectivity is addressed through our systematic framework. From these sources, the study extracted public value elements across six types: industrial advancements, safe society, sustainable environment, job creation, human health, and convenience of life. These six categories were defined based on national policy documents and technology impact assessments, and operationalized in this study by identifying and classifying text segments from patents and news columns that explicitly reflect each public value type. Our analysis focuses on the top five R&D programs, which were selected based on the highest patent counts among publicly funded AI-related programs, encompassing both educational/workforce development initiatives and key technology development projects. By integrating multiple data sources, including patents and news columns, with advanced NLP and LLM, this study demonstrates a novel approach to R&D policy evaluation that yields richer, evidence-based insights for policymakers and practitioners. The findings offer broader implications for public R&D policy by providing evidence-based insights that inform program prioritization, resource allocation, and the design of impact-oriented evaluation systems. In doing so, the study advances existing practices in R&D policy management by presenting a systematic, data-driven framework for evaluating the societal impacts of public R&D. © 2025 Elsevier B.V., All rights reserved.