James L. Chen

S M Sadrul Islam Asif, James Chen, Kennett Puerto Diaz et al.

Modern REST API testing relies on brittle sequences of calls to build system state. These multi-step tests suffer from non-determinism, poor scalability, and a "reachability tax" where a single failed setup step invalidates the entire test. We introduce TECTON, which breaks this cycle by replacing implicit state construction with explicit state synthesis of both the request payload and mock data it depends on. TECTON achieves this through two complementary mechanisms: it generates diverse, valid payloads directly, and it augments existing test mocks with realistic data so those payloads have valid system state to reference. Both mechanisms apply combinatorial testing to a new domain: the nested property space of Abstract Data Types (ADTs). TECTON decomposes complex API requests into primitive components to unleash LLMs on the more tractable subtasks of identifying equivalence classes of these primitives and generating representative values for them. It then uses LLMs to extract and inject state values via test mocks, enabling payloads to reference valid state. It recomposes these values into covering combinations to directly produce high-coverage test payloads. On standard RESTful benchmarks, TECTON achieves 70% average line coverage - a 20% absolute increase over sequence-based generators. It exposes 2x more runtime errors than any prior tool, including assertions and data constraint failures. TECTON's shift from sequencing API calls to synthesized payloads advances the state of the art in automated API validation.

Harshit Joshi, Shicheng Liu, James Chen et al. · stanford

Large Language Models can carry out human-like conversations in diverse settings, responding to user requests for tasks and knowledge. However, existing conversational agents implemented with LLMs often struggle with hallucination, following instructions with conditional logic, and integrating knowledge from different sources. These shortcomings compromise the agents' effectiveness, rendering them unsuitable for deployment. To address these challenges, we introduce Genie, a programmable framework for creating knowledge-intensive task-oriented conversational agents. Genie can handle involved interactions and answer complex queries. Unlike LLMs, it delivers reliable, grounded responses through advanced dialogue state management and supports controllable agent policies via its declarative specification -- Genie Worksheet. This is achieved through an algorithmic runtime system that implements the developer-supplied policy, limiting LLMs to (1) parse user input using a succinct conversational history, and (2) generate responses according to supplied context. Agents built with Genie outperform SOTA methods on complex logic dialogue datasets. We conducted a user study with 62 participants on three real-life applications: restaurant reservations with Yelp, as well as ticket submission and course enrollment for university students. Genie agents with GPT-4 Turbo outperformed the GPT-4 Turbo agents with function calling, improving goal completion rates from 21.8% to 82.8% across three real-world tasks.

Preethi Raghavan, James L. Chen, Eric Fosler-Lussier et al.

Electronic health records capture patient information using structured controlled vocabularies and unstructured narrative text. While structured data typically encodes lab values, encounters and medication lists, unstructured data captures the physician's interpretation of the patient's condition, prognosis, and response to therapeutic intervention. In this paper, we demonstrate that information extraction from unstructured clinical narratives is essential to most clinical applications. We perform an empirical study to validate the argument and show that structured data alone is insufficient in resolving eligibility criteria for recruiting patients onto clinical trials for chronic lymphocytic leukemia (CLL) and prostate cancer. Unstructured data is essential to solving 59% of the CLL trial criteria and 77% of the prostate cancer trial criteria. More specifically, for resolving eligibility criteria with temporal constraints, we show the need for temporal reasoning and information integration with medical events within and across unstructured clinical narratives and structured data.