Artificial Intelligence for Patient Support: Assessing Retrieval-Augmented Generation for Answering Postoperative Rhinoplasty Questions

Overview

Inaccurate or incomplete output from general purpose language models poses a safety risk. Retrieval-augmented generation addresses that by having the model draw from a curated knowledge base rather than its training alone. The authors tested four such models on 30 common postoperative rhinoplasty questions, with responses sourced from authoritative rhinoplasty texts.

Sections of note

• Design: comparative evaluation of four retrieval-augmented models: Gemini-1.0-Pro-002, Gemini-1.5-Flash-001, Gemini-1.5-Pro-001 and PaLM 2.
• 30 common patient inquiries were used.
• Responses were scored for accuracy on a 1 to 5 scale, comprehensiveness on a 1 to 3 scale, readability by Flesch scores, and understandability and actionability by the Patient Education Materials Assessment Tool.
• 41.7 percent of responses were completely accurate.
• The models failed to respond at all in 30.8 percent of cases, which the authors attribute to problems interpreting the question and retrieving material.
• Gemini-1.0-Pro-002 was most comprehensive, p less than 0.001.
• Readability, Flesch Reading Ease 40 to 49, and understandability, mean 0.7, fell below patient education standards.
• PaLM 2 scored lowest on actionability, p less than 0.007.

What it means for a patient

• Even when linked to authoritative textbooks, fewer than half the answers were completely accurate.
• Nearly a third of questions produced no answer at all, which is safer than a wrong answer but makes the tool unreliable.
• All the models wrote at a level too difficult for general patient education.
• Limits: 30 questions, four models tested at one point in time, and no comparison against answers from a surgeon.

Why this paper matters

Patients recovering from surgery ask questions when their surgeon is not available, and grounding a model in real textbooks is the main proposed fix for invented answers. Testing it in this setting shows the approach improves accuracy but introduces a high failure to respond rate. Whether these tools are safe for patient facing use is not yet established.

Terms

• Retrieval-augmented generation: a method where a model draws answers from a curated document set rather than memory alone.
• Large language model: software trained on text that generates human like written responses.
• Comprehensiveness: whether an answer covers all relevant aspects of a question.
• Actionability: whether the material tells the reader what to do.
• Flesch Reading Ease: a score estimating how easy a text is to read, higher being easier.