Advancement of Generative Pre-trained Transformer Chatbots in Answering Clinical Questions in the Practical Rhinoplasty Guideline

Overview

The authors evaluated how well artificial intelligence chatbots answer clinical questions drawn from practical rhinoplasty guidelines. Ten questions from the guidelines served as the source. For each, GPT-4 and GPT-3.5 were asked to supply an answer plus the policy level, the aggregate evidence quality, the level of confidence in the evidence, and the supporting references.

Sections of note

• Design: comparison of two language model versions on guideline questions. Level of evidence V.
• 10 clinical questions were included in the final analysis.
• The chatbots correctly answered 90.0 percent of the questions overall.
• GPT-4 was less accurate than GPT-3.5 on the questions themselves, 86.0 versus 94.0 percent, but the difference was not statistically significant, p equal to 0.05.
• GPT-4 was significantly more accurate on level of confidence in the evidence, 52.0 versus 28.0 percent, p less than 0.01.
• No statistical difference was found for policy level, aggregate evidence quality or reference match.
• GPT-4 presented existing rather than fabricated references significantly more often, 36.9 versus 24.1 percent, p equal to 0.01.

What it means for a patient

• The systems answered most guideline questions correctly, but their supporting references were real less than 40 percent of the time even in the better version.
• Fabricated references are a known failure mode of these systems, and this study quantifies it in a medical setting.
• Newer does not automatically mean more accurate. The newer version scored lower on the questions themselves, though not significantly.
• Limits: 10 questions, models tested at one point in time, and no comparison against a clinician answering the same questions.

Why this paper matters

Clinical guidelines are the reference standard for what should be done, so testing chatbots against them measures something more objective than patient satisfaction with answers. The reference fabrication finding is the practically important one, since a plausible but invented citation is harder to detect than a wrong answer. Ten questions is a small test set.

Terms

• Generative Pre-trained Transformer: the model architecture behind ChatGPT.
• Clinical question: a structured question a guideline is written to answer.
• Policy level: how strongly a guideline recommends or discourages an action.
• Aggregate evidence quality: the overall strength of the evidence supporting a recommendation.
• Reference match: whether a cited source actually exists and says what is claimed.