Agent-Specific Prompt Engineering for LLM-Guided RL Exploration

초록

Recently, Large Language Model (LLM)-based Reinforcement Learning (RL) has gained attention as a new approach to improving exploration efficiency. However, traditional LLM-exploration relies on human-in-the-loop processes when encoding agent-specific features, expressed as numerical and symbolic representations, into natural language. This process is affected by human-induced bias, which can underestimate or overestimate certain elements of agent-specific features, leading to deviations in agents’ exploration performance. In this paper, we propose an LLM-exploration method that considers agent-specific features while minimizing human intervention. For each agent, we first generate Action Possible Models (APMs) that estimate the feasibility of all actions in each state. Based on these APMs, prompts are generated, which allow the LLM to consider agent-specific features. The proposed LLM-exploration reduces deviations in exploration performance among agents.We conduct experiments with different agents in the Street Fighter III: 3rd Strike environment, and the proposed method achieves an average win rate improvement of 19.3% compared to ϵ-greedy exploration, while also reducing performance deviations across different agents. We further demonstrate the scalability of the proposed method by applying it to the Reacher-v4 and MiniGrid environments. Overall, we contribute to reinforcement learning by integrating agent-specific features into exploration using APMs and LLMs, while reducing reliance on human-in-the-loop intervention. © 2013 IEEE.

키워드