A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study
Pythia is a multi-agent system for autonomous, fine-tuning-free clinical symptom detection from clinical notes. It runs on a locally hosted open-weights model and selects prompts using development-set sensitivity and specificity. In a study with 72 signs and symptoms from 400 clinical notes (387 patients), Pythia achieved mean sensitivity 0.76 and specificity 0.95, compared to a lexicon's 0.82 and 0.76. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity 0.97.
Development
- First ReportA Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation StudyarXiv cs.AI
- Current AssessmentThis work demonstrates that multi-agent systems can effectively automate clinical information extraction without supervised fine-tuning, potentially reducing the barrier for deploying NLP in healthcare settings where labeled data is scarce.Hacker Linner · analysis
Pythia is a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning, achieving high specificity (0.95) compared to a lexicon (0.76) on clinical symptom detection.
Pythia uses a multi-agent architecture to autonomously generate and optimize prompts for clinical concept extraction, eliminating the need for fine-tuning. It runs on a locally hosted open-weights model, ensuring data privacy. The system selects prompts based on development-set sensitivity and specificity, achieving a balance that outperforms a lexicon in specificity while maintaining competitive sensitivity.
This work demonstrates that multi-agent systems can effectively automate clinical information extraction without supervised fine-tuning, potentially reducing the barrier for deploying NLP in healthcare settings where labeled data is scarce.
Pythia offers a cost-effective and privacy-preserving solution for extracting structured clinical data from notes, which can improve downstream analytics, decision support, and population health management without the need for extensive manual annotation or model fine-tuning.
Future work could extend Pythia to more clinical concepts and larger datasets, and compare against fine-tuned models. The approach may also be adapted to other domains requiring context-sensitive extraction from unstructured text.