Simulating the people's voice: Leveraging algorithmic fidelity to assess ChatGPT's performance in modeling public opinion on Chinese government policies

초록

Traditional public opinion surveys face persistent challenges related to cost, sample representativeness, and respondent willingness. These limitations have encouraged growing interest in using large language models (LLMs) to generate silicon samples as synthetic substitutes for human data. Although previous studies report high algorithmic fidelity in Western contexts, much less is known about whether globally trained LLMs can reproduce public attitudes in regulated and nonWestern information environments. Using nationally representative data from the Chinese General Social Survey (CGSS 2021), this study evaluates ChatGPT's ability to simulate Chinese public opinion on ten policy issues by comparing human responses with demographic-conditioned silicon samples. Analyses of response rates, response distributions, and demographic subgroups show that LLM outputs approximate human attitudes on low-sensitivity and consensus-oriented topics, but diverge systematically on culturally embedded and governance-sensitive issues. Silicon samples also produce near-complete response rates, which fails to capture human patterns of strategic non-response, and show larger misalignment among politically embedded and highly educated subgroups. Robustness diagnostics across model generations reveal strong cross-model structural stability but continued limitations when the model is applied in different sociopolitical contexts. These findings reconceptualize algorithmic fidelity as a context-sensitive construct and extend Pattern Correspondence into a multidimensional framework that incorporates response rates, response distributions, and demographic subgroup patterns. Overall, the study highlights both the potential and the limits of using LLMs to simulate public opinion in non-Western settings, emphasizing the need for culturally grounded calibration, transparent reporting, and cautious use in policy-relevant domains.

키워드