Advancing Academic Chatbots: Evaluation of Non Traditional Outputs
This work addresses the need for better evaluation methods for academic chatbots, focusing on emerging outputs, but it is incremental as it builds on existing retrieval and generation techniques.
This research tackled the problem of evaluating large language models for non-traditional academic outputs like slide decks and podcast scripts, finding that GPT 4o mini with Advanced RAG produced the most accurate responses, while Graph RAG led to more hallucinations and limited improvements.
Most evaluations of large language models focus on standard tasks such as factual question answering or short summarization. This research expands that scope in two directions: first, by comparing two retrieval strategies, Graph RAG, structured knowledge-graph based, and Advanced RAG, hybrid keyword-semantic search, for QA; and second, by evaluating whether LLMs can generate high quality non-traditional academic outputs, specifically slide decks and podcast scripts. We implemented a prototype combining Meta's LLaMA 3 70B open weight and OpenAI's GPT 4o mini API based. QA performance was evaluated using both human ratings across eleven quality dimensions and large language model judges for scalable cross validation. GPT 4o mini with Advanced RAG produced the most accurate responses. Graph RAG offered limited improvements and led to more hallucinations, partly due to its structural complexity and manual setup. Slide and podcast generation was tested with document grounded retrieval. GPT 4o mini again performed best, though LLaMA 3 showed promise in narrative coherence. Human reviewers were crucial for detecting layout and stylistic flaws, highlighting the need for combined human LLM evaluation in assessing emerging academic outputs.