Good-enough but more error-prone: Garden-path processing in GPT models

Abstract

This research explores the syntactic processing of Large Language Models (LLMs), specifically GPT-3.5 and GPT-4, by comparing them to human processors, focusing on garden-path sentences. These structures are challenging for even proficient human processors, often causing misinterpretations that persist despite reanalysis, revealing the ‘good-enough’ nature of human syntactic processing. This study aims to determine if LLMs exhibit a similar ‘good-enough’ syntactic processing as humans and whether more advanced models exhibit a more human-like processing. In a series of experiments, we examined how models handle garden-path sentences such as “While the man hunted the deer ran into the woods,” through a comprehension questions task. A key focus was whether misinterpretations in the target phrases (“hunted the deer”) erroneously affected the global interpretation of the sentence. Results showed that LLMs display patterns similar to humans, including lingering misinterpretations and the ability to utilize linguistic cues such as plausibility, phrase length, and verb type. This suggests that LLMs mimic human ‘good-enough’ syntactic processing through probabilistic next-word prediction, including making human-like errors. However, LLMs also showed vulnerability to garden-path structures, showing a higher rate of errors compared to humans, likely due to inherent features of their processing mechanisms.