In the study, which utilized a validated dataset of nearly 79,000 labeled medical concepts, the best-performing frontier LLM reached only a 55% F1 score, fabricating nearly one in three medical codes. Researchers found that relying on retrieval-augmented generation (RAG) failed to solve the problem; instead of improving reliability, RAG-assisted models saw their accuracy plummet to a 22.64% F1 score, identifying fewer than one in four relevant clinical concepts.
Medical AI Engine Outperforms LLMs in Clinical Coding Accuracy
A head-to-head evaluation of 272 hospital discharge summaries reveals that standard large language models struggle with medical coding, frequently hallucinating data. Minneapolis-based emtelligent reports its Medical Language Engine achieved an 89.85% F1 score, leaving general-purpose models far behind in clinical reliability.
Tim O'Connell, M.D., CEO and co-founder of emtelligent, noted that general-purpose models lack the ability to check answers against accepted medical ontologies. Because these systems do not signal uncertainty, they often generate incorrect codes, creating significant risks for patient care, hospital revenue, and regulatory compliance. The emtelligent platform bridges this gap by mapping clinical text to standardized terminologies like SNOMED CT, ICD-10, and LOINC, providing structured data that allows LLMs to function with greater precision.
Comments (0)
No comments yet. Be the first!