ERFSL: An Efficient Reward Function Searcher via Language Models for Custom-Environment Multi-Objective Optimization (Student Abstract)
This work addresses the challenge of manual reward engineering for multi-objective RL in custom environments, offering an automated LLM-based approach that reduces human effort.
ERFSL uses LLMs to automatically design and optimize reward functions for multi-objective reinforcement learning in custom environments, achieving correction of reward codes with one feedback iteration per requirement and requiring only 5.2 iterations on average to meet user requirements even when initial weights are off by a factor of 500.
We propose ERFSL, an efficient reward function searcher using large language models (LLMs) for custom-environment, multi-objective learning-based methods (LB). ERFSL generates reward components based on explicit user requirements, rectifies them using a reward critic, and iteratively optimizes the weights of these components based on textual context generated by the training log analyzer. Applied to a simulation-based benchmark task, the reward critic corrects reward codes with only one feedback iteration per requirement, and the reward weight initializer acquires diverse reward functions within the Pareto set. Even when a weight is off by a factor of 500, an average of only 5.2 iterations is needed to meet user requirements. The approach works adequately with GPT-4o mini and does not require advanced understanding capabilities.