Adobe Summit Concierge Evaluation with Human in the Loop
This work addresses the problem of building enterprise AI assistants for event management, though it is incremental as it applies known methods like human-in-the-loop to a specific domain.
The paper tackled the challenge of developing a domain-specific AI assistant for Adobe Summit under real-world constraints like data sparsity and rapid deployment, resulting in a system that handles event-related queries through a human-in-the-loop workflow enabling scalable and reliable performance in cold-start scenarios.
Generative AI assistants offer significant potential to enhance productivity, streamline information access, and improve user experience in enterprise contexts. In this work, we present Summit Concierge, a domain-specific AI assistant developed for Adobe Summit. The assistant handles a wide range of event-related queries and operates under real-world constraints such as data sparsity, quality assurance, and rapid deployment. To address these challenges, we adopt a human-in-the-loop development workflow that combines prompt engineering, retrieval grounding, and lightweight human validation. We describe the system architecture, development process, and real-world deployment outcomes. Our experience shows that agile, feedback-driven development enables scalable and reliable AI assistants, even in cold-start scenarios.