REAL: Benchmarking Abilities of Large Language Models for Housing Transactions and Services
This work addresses the need to evaluate LLMs as agents in real estate, but it is incremental as it applies existing benchmarking methods to a new domain.
The authors introduced REAL, the first evaluation suite with 5,316 entries to assess large language models' abilities in housing transactions and services, finding that LLMs still have significant room for improvement in this domain.
The development of large language models (LLMs) has greatly promoted the progress of chatbot in multiple fields. There is an urgent need to evaluate whether LLMs can play the role of agent in housing transactions and services as well as humans. We present Real Estate Agent Large Language Model Evaluation (REAL), the first evaluation suite designed to assess the abilities of LLMs in the field of housing transactions and services. REAL comprises 5,316 high-quality evaluation entries across 4 topics: memory, comprehension, reasoning and hallucination. All these entries are organized as 14 categories to assess whether LLMs have the knowledge and ability in housing transactions and services scenario. Additionally, the REAL is used to evaluate the performance of most advanced LLMs. The experiment results indicate that LLMs still have significant room for improvement to be applied in the real estate field.