Cancer Registry Chatbot Study Tests Gemini Against GPT Baselines

A team from the University of Missouri and collaborators has published a paper describing CRISS, the Cancer Registry Intelligent Support System, a chatbot meant to help cancer registrars and Oncology Data Specialists work through coding and staging rules that change often and are hard to interpret quickly by hand. The system is built as a retrieval-augmented generation, or RAG, assistant: it draws from a knowledge base built out of national cancer registry standards, splits that material into tagged passages, indexes them as dense embeddings, and then generates answers grounded in the passages it retrieves, with citations attached so a registrar can check the source before acting on it.

The researchers tested several kinds of models against this setup, including open-weight systems, proprietary systems, and plain non-RAG baselines, spanning both the Gemini and GPT model families. Every model answered easy, medium and hard registry questions, and an LLM-as-a-Judge protocol scored the answers on grounding and semantic similarity. Across every difficulty tier, the RAG-equipped versions of the models outperformed the same models running without retrieval, and the gap widened as the questions got harder, which the paper reads as evidence that retrieval matters more precisely when a registrar is facing the trickiest coding edge cases.

The paper also draws a split between how proprietary and open-weight models handled difficulty. Proprietary RAG models did best on easy and medium questions, but local, open-weight RAG models actually ranked highest on the hardest questions, and the paper notes proprietary models tended to answer more cautiously overall. The authors frame CRISS as a way to support training and helpdesk use for registrars while keeping a human making the final abstraction call, rather than as a system meant to replace that judgment.

The numbers

• RAG configurations scored a mean of 0.62, 0.56 and 0.59 across the easy, medium and hard tiers.
• Non-RAG baselines scored a mean of just 0.29, 0.26 and 0.29 on the same three tiers.
• The paper runs 21 pages, with 13 figures and 7 tables backing the evaluation.

The work is framed as a proof of concept rather than a finished clinical tool, and the authors are explicit that final coding decisions should stay with a human registrar. For a field where documentation standards shift year over year, a citation-supported assistant that shows its sources, rather than one that simply answers, is the design choice the paper argues actually moved the accuracy numbers.