Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos
Provides a scalable, clinician-grade behavioral assessment tool for earlier autism diagnosis from home videos, addressing diagnostic delays.
Fine-tuned Gemini 2.5 Pro on 400 clinician-rated home videos improved inter-rater reliability with clinicians by 40% (p<0.001) and achieved 77% accuracy (95% CI: 68-85%) and AUC of 86% (95% CI: 78-92%) for ASD diagnosis, matching or exceeding clinician performance.
Autism spectrum disorder (ASD) affects 1 in 31 US children, yet median age at diagnosis exceeds four years. Artificial intelligence pipelines that provide quantified diagnosis using easy to access observational data (e.g., home videos) could help with earlier diagnosis, and timely delivery of early treatments. We fine-tuned Gemini 2.5 Pro on 400 clinician-rated home videos with low-rank adaptation, training only on 30 behavioral features previously validated to produce reliable predictions when passed to various ML models. On 99 held-out children (49 ASD, 50 neurotypical), inter-rater reliability with clinicians (per-feature weighted Cohen's kappa) improved by 40% (p<0.001), with 27 of 28 evaluable features improving. As an emergent zero-shot capability, direct ASD diagnosis F1 improved by 53% (p<0.001), matching or exceeding clinician outcomes. Classifier-assisted pipelines using fine-tuned LLM-derived behavioral features matched clinician-scored inputs across all tested pathways and achieved 77% accuracy (95% CI: 68-85%) and an AUC of 86% (95% CI: 78-92%). Fine-tuned multimodal LLMs can serve as scalable behavioral feature extractors for use in autism assessment and diagnosis.