Comparative Performance of the Leading Large Language Models in Answering Complex Rhinoplasty Consultation Questions

Overview

Several language models can produce medical discussion at a human level, but they had not been compared on rhinoplasty knowledge. The authors put ten open ended rhinoplasty consultation questions to ChatGPT-4o, Google Gemini, Claude and Meta-AI. Responses were randomized and ranked by seven plastic surgeons specializing in rhinoplasty, from 1 for worst to 4 for best, and readability was scored separately.

Sections of note

• Design: blinded comparative ranking of four language models by seven expert surgeons.
• Ten open ended rhinoplasty consultation questions were used.
• Claude gave the top answer for seven questions; ChatGPT for three.
• Collective scores: Claude 224, ChatGPT 200, Meta 138, Gemini 138.
• Claude's mean score per question was 3.20, significantly outperforming all other models, p less than 0.05.
• ChatGPT's mean was 2.86, outperforming Meta and Gemini, which performed similarly to each other.
• Readability was assessed with Flesch Reading Ease and Flesch-Kincaid Grade scores.
• Meta had a significantly lower Flesch-Kincaid grade than Claude and ChatGPT, and a lower Flesch Reading Ease than ChatGPT.

What it means for a patient

• The models differ substantially in quality on the same questions, so which tool a person uses affects the information they get.
• The model rated best for content was not the easiest to read, and the model with the simplest text ranked lowest for quality.
• Ratings came from surgeons judging quality, not from patients judging usefulness.
• Limits: ten questions, seven raters, one point in time for models that change frequently, and no comparison against information from a surgeon.

Why this paper matters

People consult these tools before booking a consultation, and until now there was no comparison of which handles rhinoplasty questions well. Ranking them by blinded expert judgment establishes a baseline. The rankings date quickly as models are updated, and the study does not assess whether any answer is safe to act on.

Terms

• Large language model: software trained on text that generates human like written responses.
• Open-ended question: a question that cannot be answered yes or no.
• Flesch Reading Ease: a score estimating how easy a text is to read, higher being easier.
• Flesch-Kincaid Grade: an estimate of the school grade level needed to understand a text.
• Blinded ranking: rating by evaluators who do not know the source of what they rate.