Assessing LLM code generation quality through path planning tasks
This work addresses the problem of assessing LLM code generation risks for safety-critical path planning tasks, highlighting a critical gap in existing benchmarks.
The study evaluated six LLMs on generating code for three path-planning algorithms across maps of varying difficulty, finding that LLM-generated code poses serious hazards for safety-critical applications and should not be used without rigorous testing.
As LLM-generated code grows in popularity, more evaluation is needed to assess the risks of using such tools, especially for safety-critical applications such as path planning. Existing coding benchmarks are insufficient as they do not reflect the context and complexity of safety-critical applications. To this end, we assessed six LLMs' abilities to generate the code for three different path-planning algorithms and tested them on three maps of various difficulties. Our results suggest that LLM-generated code presents serious hazards for path planning applications and should not be applied in safety-critical contexts without rigorous testing.