Paper Pairs LLaMA-3 With Gemini Vision For Medical Reports

The paper, titled “MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis,” was submitted by Rajdeep Pandhare, Rohit Kundaikar, Yakub Kojar and Shreyansh Tiwari and later published in the International Journal of Scientific Research in Engineering and Management. It sets out to address a specific bottleneck: clinicians reading increasingly long and varied medical reports under time pressure, with manual review described as slow, error-prone and hard to scale.

MeddeX’s architecture layers several pieces on top of a QLoRA fine-tuned LLaMA-3 8B language model. Retrieval-augmented generation grounds answers in the source document rather than letting the model invent detail, while the Gemini Vision layer handles anything that is not clean, text-native text: scanned image-based PDFs and direct radiological images. A hybrid extraction pipeline sits in front of both, including what the authors call a garbage-text detection mechanism built to catch corrupted and UTF-16-encoded documents, a practical problem anyone who has processed real hospital archives will recognize.

Beyond answering questions about a single report, the system adds report-scoped retrieval for tighter contextual grounding, a longitudinal comparison engine that tracks changes across a patient’s reports over time, and a self-healing backfill pipeline that processes older, legacy documents only when they are actually requested rather than all at once. The whole system runs on a FastAPI backend with a Supabase vector store behind it.

The numbers

The authors benchmarked MeddeX against the base LLaMA3-Med42-8B model and reported gains across every metric they tracked: ROUGE-1 rose by 9.2%, ROUGE-L by 11.8%, and BLEU by 71.4%, alongside smaller improvements on ROUGE-2 and BERTScore. The authors present these gains as evidence that domain-specific fine-tuning, rather than the base medical model alone, is doing meaningful work.

The paper frames MeddeX as “a practical and robust step toward AI-assisted clinical decision support,” language that signals an engineering prototype rather than a deployed clinical tool. No hospital or health system is named as a user, and the evaluation is against a single base model rather than a field trial. Still, the design pattern, an open fine-tuned language model with Gemini doing the heavy lifting on images, is one other teams building similar document pipelines are likely to copy, since it avoids the cost of training a vision model in-house while still handling the messiest input: a scanned, decades-old chart nobody digitized properly the first time.