Introduction

The internet remains a vital data source for patients seeking health information. Recent estimates suggest that approximately 80% of Americans utilize the internet for health searches.¹ Google is the most common search engine used by patients when seeking health information – as nearly 7% of Google’s more than 8.5 billion daily searches are health-related.² Given Google’s role as the predominant search engine used by patients, it is imperative that responses to patient questions are accurate and understandable. Yet, prior research has demonstrated that Google search information related to ophthalmologic subspecialties including glaucoma, retina, and oculoplastics is often inaccurate and written at a reading level that may be difficult to understand by patients.^3–7

In the past year, artificial intelligence (AI) large language models (LLMs) have risen in popularity amongst the general public and have emerged as an alternative information source for patients.⁸ ChatGPT has become the most popular LLM in the US, amassing more than 200 million users since 2022.^9,10 Prior studies have shown that the ability of ChatGPT to respond to patient queries has been mixed in ophthalmology and in medicine as a whole – with some studies reporting that responses provided by ChatGPT were accurate while others reported that ChatGPT responses contained inaccurate and potentially harmful information.^11–13

In addition to serving as an information source for patients, LLMs like ChatGPT also have the potential to greatly enhance patient education capabilities for physicians. LLMs which demonstrate an ability to easily break down complex medical phenomenon into terms that patients can better understand may contribute to more informed shared decision making between physicians and patients, which can ultimately improve patient outcomes.^14–17

While the ability of both Google and LLMs like ChatGPT to respond to patient queries has been explored in several ophthalmology subspecialties, to date, there is limited information on this topic in oculoplastics.^17–20 Furthermore, it is unclear whether ChatGPT has the ability to adjust patient education materials to appropriate reading levels while maintaining accuracy of information.

As such, the purpose of our study is twofold. First, a panel of fellowship-trained oculoplastics surgeons will compare the accuracy and readability of responses to common oculoplastics patient questions provided by two information sources: Google and ChatGPT. Second, we assess the ability of ChatGPT to assist physicians in creating customized patient education materials to better explain common oculoplastics procedures to patients with varying health literacy.

Materials and Methods

Generation and Responses of FAQs

We used a clean-installed Google Chrome (Menlo Park, CA) browser on Incognito Mode to execute all Google searches used in data collection. We disabled location filters and sponsored results to avoid bias from previous searches and targeted geographic search results. The 10 search terms used in this study were identified from a prior study evaluating patient education materials for oculoplastics diagnoses, which was based on the American Society of Ophthalmic Plastic and Reconstructive Surgery (ASOPRS) website.⁴ The search terms included: “thyroid eye disease”, “orbital cellulitis”, “orbital tumor”, “proptosis”, “ptosis”, “entropion”, “blepharospasm”, “chalazion”, “nasolacrimal duct obstruction”, and “epiphora”.

After each of the 10 Google searches were executed, the first three “Frequently Asked Questions” (FAQs) associated with each of the search terms were recorded. These 30 FAQs were entered into both the Google search engine and the ChatGPT V 3.5 tool and responses were recorded. For FAQs entered into Google, the response that was recorded was the response processed by the Google search engine that populated on the Google search page itself. However, the Google response which populates was derived from a website, and the website used to generate the response was also noted. For all ChatGPT queries in our study, the chatbot was instructed to respond with a temperature setting of 0.3 (scale 0–1, with 0 more focused responses and 1 more creative responses) to ensure more focused responses intended for use by physicians and patients.

Readability Analysis

Responses to patient FAQs provided by both Google and ChatGPT were evaluated for readability using 5 validated, objective readability assessments: Flesch Reading Ease (FRE), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and Coleman Liau Index (CLI). The FRE scale measures readability by generating a score from 0 (very difficult) to 100 (very easy). Each of the four remaining readability scales provide a “grade-level” at which the article was written. For example, a score of 6 indicates the text was written at a 6th grade reading level. An average grade level for each article was calculated from the GFI, FKGL, SMOG, and CLI indices.

JAMA Accountability Analysis

All 30 websites used by Google to generate responses to patient FAQs were evaluated for accountability (score of 0–4) using JAMA benchmarks. According to JAMA guidelines, a website containing patient education materials should (1) include all authors and their relevant credentials (2) list references (3) provide disclosures and (4) provide date of last update.

ChatGPT Procedure Explanations

ChatGPT was instructed to generate patient education materials for 8 common oculoplastics procedures to help patients better understand each respective procedure. The 8 procedures included: blepharoplasty, browplasty, dacryocystorhinostomy, rhinoplasty, orbital decompression, levator advancement, ectropion repair, and botox for blepharospasm. The phrase, “ChatGPT, please explain what happens in a [procedure name]” was entered into the ChatGPT tool. The responses provided were recorded and evaluated for readability using the five aforementioned readability indices. To assess ChatGPT’s ability to generate patient education materials at a specific reading level, the tool was then asked to generate procedure-specific patient education materials at a 7^th grade reading level. The phrase, “ChatGPT, please explain what happens in a [procedure name] at a 7^th grade reading level” was entered into the ChatGPT tool. The responses provided were recorded and, again, evaluated for readability.

Expert Panel Evaluation - FAQs

A panel of 6 fellowship-trained oculoplastics surgeons employed by six different academic institutions were asked to independently review responses to each of the 30 FAQs generated by both Google and AI for several different criteria. For all 30 FAQs, the responses provided by Google and AI were listed side-by-side in a randomized order and three questions were asked. Experts were first asked to select the better (more accurate and comprehensible) response to the patient question. They were then asked to identify which response was generated by artificial intelligence. Finally, they were asked if either of the responses contained inappropriate or inaccurate information.

Expert Panel Evaluation – Procedures

For each of the 8 oculoplastics procedures, the explanations provided by ChatGPT at various reading levels (unspecified and 7^th grade level request) were listed side-by-side in a randomized order. Experts were asked to choose which block of text they would select for informational pamphlets designed to better explain the procedure. They were then asked to identify the block of text written at a lower reading level. Finally, they were asked if either of the responses contained inappropriate or inaccurate information.

Statistical Analysis

Data was analyzed in R version 4.3.2. Two-sample t-tests were utilized to compare 1) the readability of responses provided by Google and ChatGPT and 2) the readability of procedure explanations provided by ChatGPT at various requested reading levels. Two-sided χ2 tests of independence and two-sided z-tests were used to assess for associations between categorical variables as appropriate. A P value < 0.05 was considered statistically significant.

Results

Frequently Asked Questions Related to Common Oculoplastics Conditions

Table 1 shows the 30 FAQs that populated after executing a Google search for 10 common oculoplastics conditions. Both open-ended and closed-ended questions populated, with qualitative and quantitative patient inquiries.

Table 1 Most Frequently Asked Questions for 10 Common Oculoplastics Conditions

Readability of Responses to Patient FAQs - Google vs ChatGPT

Responses to patient FAQs provided by ChatGPT were written at a significantly higher average grade level than responses provided by Google (15.6 vs 10.0, respectively, p < 0.001) (Table 2). ChatGPT responses also averaged a lower (more difficult to understand) FRE than Google responses (27.4 vs 48.5, respectively, p < 0.001).

Table 2 Readability of Responses to Oculoplastics FAQs – Google Vs ChatGPT

JAMA Accountability of Webpages Providing Answers to FAQs

Most webpages providing answers to Google FAQs related to the oculoplastics conditions were from educational institutions, with fewer webpages from national organizations and private practices. The average JAMA accountability score of the 30 webpages analyzed was 1.07/4. The most frequent JAMA accountability metrics on webpages analyzed included date of last update (50%), author list (20%), and references (17%).

Readability of ChatGPT Generated Patient Education Materials for Common Oculoplastics Procedures

Table 3 displays the readability metrics of the ChatGPT-generated patient education materials for common oculoplastics procedures. Without specifying a grade level at which patient education materials should be written, the average grade level of the patient education materials produced was 15.7 (college level), with a FRE of 29.8 (“very difficult”). When asking ChatGPT to generate patient education materials at a 7^th grade reading level, readability significantly improved to an average grade level of 11.7 (high school level) and a FRE of 53.6 (“fairly difficult”) (p < 0.001).

Table 3 Readability of Oculplastics Procedure Explanations – Unspecified Grade Level vs 7th Grade Level Requested

Expert Panel Evaluation – Google and ChatGPT Responses to FAQs

When asked to identify the better (more accurate and comprehensible) response to the 30 FAQs included in our study, responses provided by ChatGPT were selected 79% (142/180) of the time by expert reviewers while Google responses were selected less frequently (12%, 21/180). Table 4 shows the distribution of how expert reviewers compared AI and Google responses to patient FAQs by topic.

Table 4 Expert Review of the Better (More Accurate and Comprehensible) Responses to Patients’ Frequently Asked Questions by Topic (3 Questions for Each Topic Reviewed by All 6 Panelists)

When asked to identify which of the provided responses to the 30 FAQs was generated by AI, panelists were largely unsuccessful. Approximately 33% of responses to this question (59/180) were, “I cannot tell which response was generated by an artificial intelligence large language model”. Of the panelists who elected to make a selection, 31% (38/121) correctly identified the AI-generated response (Table 5).

Table 5 Expert Panelist Ability to Correctly Identify Responses to Patients’ Frequently Asked Questions Generated by Artificial Intelligence Large Language Models by Topic

When asked whether the AI-generated or Google-generated responses to patient FAQs contained inaccurate or inappropriate information, the majority of responses did not contain inaccurate or inappropriate information (136/180, 76%). Responses provided by Google were rated to contain inaccurate or inappropriate information 22% of the time while responses provided by ChatGPT were rated to contain inaccurate or inappropriate information 7% of the time (p < 0.001).

Expert Panel Evaluation – ChatGPT Generated Patient Education Materials for Common Oculoplastics Procedures

When asked to select which block of text (unspecified grade level vs 7^th grade reading level requested) they would use to incorporate into patient education materials about common oculoplastics procedures, responses were distributed between the text written at the unspecified grade level (19/48, 40%), 7^th grade reading level (12/48, 25%), and both equally (17/48, 35%) (Table 6). Panelists were accurately able to select which of the responses was written at the lower reading level for most procedures (30/48, 63%). When asked to evaluate whether each block of text contained inaccurate or inappropriate information, “neither” was selected 94% of the time. Both the responses provided at the unspecified grade level and the responses generated after the 7^th grade reading level was requested were labeled as inaccurate on 3% of total panelist reviews.

Table 6 Reading Level of Patient Education Information Preferred by Oculoplastics Surgeons if They Were Creating Informational Pamphlets for Patients by Topic

Discussion

Our study sought to compare the responses generated by Google and an artificial intelligence large language model (ChatGPT) to FAQs regarding common oculoplastics conditions. Additionally, our study assessed the efficacy of Chat GPT in generating patient-oriented educational materials tailored to varying levels of health literacy for common oculoplastics procedures. Ideally, patient education materials are written a low grade level with high accuracy, thereby achieving maximum reach. We found that when responding to patient FAQs, responses generated by ChatGPT were rated as more accurate and were preferred by a panel of fellowship-trained oculoplastics surgeons. However, responses provided by ChatGPT were written at a significantly higher grade level than responses generated by Google, which could impair understandability among patients with lower health literacy. ChatGPT was able to effectively create accurate patient education materials containing information about several common oculoplastics procedures. Furthermore, if prompted to generate patient education materials at a lower reading level, ChatGPT was able to effectively reduce the grade level at which procedure explanations were written from a college grade level to a high school grade level without sacrificing accuracy. Large language models such as ChatGPT have the potential to revolutionize both the way patients learn about their health and the way physicians educate patients. Oculoplastic surgeons cognizant of the many potential applications of LLMs, such as ChatGPT, may utilize these tools to ensure patients receive accurate information online while also tailoring education interventions to patients with varying health literacy.

Our study adds to the existing body of literature regarding the use of LLMs in ophthalmology. While a recent study determined that ChatGPT produces incomplete and potentially harmful information about common ophthalmic conditions,¹³ our results support two prior studies which found that ChatGPT was largely effective in responding to patients’ eye care questions.^18,21 The authors believe that LLMs represent an accurate information source for patients and can be utilized by providers as an effective patient educational tool.

With patients increasingly turning to the internet to learn more about their health, our study’s findings have important implications for the future of patient education. Currently, Google is the most common website patients turn to for health information.²² Our results demonstrate that 6 expert oculoplastic surgeons considered ChatGPT responses to patient queries more accurate than Google search responses. These results are interesting as most Google Search responses came directly from a physician website. Furthermore, oculoplastics surgeons had difficulty distinguishing AI-generated from Google-generated responses, with less than 50% accuracy overall, likely indicating preconceived notions about the types of responses likely to be generated by LLMs vs Google. LLMs have the potential to simplify and improve the way oculoplastic surgeons and their teams provide patient information. Physicians can use LLMs to update patient facing educational materials in clinic and on their websites by making them more understandable for patients with varying health literacy. Additionally, electronic health record systems can integrate AI tools as a first response option to common patient questions in an effort to minimize physician extender, registered nurse, and physician phone calls and charting time. Further, patients who utilize LLMs for health information also benefit from receiving a direct response to their inquiry, rather than having to sift through the vast amount of misinformation that patients often encounter after executing a Google search with a medical question.²³

While ChatGPT may represent an accurate alternative for patient inquiries regarding common oculoplastics procedures, readability of responses by ChatGPT were far higher than the 6^th grade level recommended by the American Medical Association. Average responses to patient queries in our study were written at a 16^th grade level (college level), which may be difficult to understand for patients without a college degree (62% of Americans).²⁴ Oculoplastics surgeons must therefore exercise caution when referring patients to ChatGPT as a potential information source. Patients should be directed to specifically note that they prefer a lower reading level when executing their ChatGPT inquiry in order to increase the chance that the response provided by the tool is at an appropriate reading level. There are, however, limitations with this approach. When instructed to provide responses at a 7^th grade reading level, the responses provided by ChatGPT were written at a 12^th grade level – an improvement compared to when no grade level was specified but still higher than the 7^th grade reading level requested. Further advances in LLMs are required to more accurately provide information at a requested reading level.

When using LLM for the purpose of creating patient facing educational materials, oculoplastics surgeons should understand ChatGPT’s ability to tailor patient education materials to lower reading levels without sacrificing accuracy – and should therefore specify their target reading level. In our study, the current version of ChatGPT was effective in improving understandability of text from a college graduate level to a high school graduate level, which would make patient education materials far more inclusive, as approximately 87% of Americans have obtained a high school degree as opposed to only 38% with a college degree.^24,25 Furthermore, survey respondents indicated that they were comfortable sharing patient education materials written at a lower reading level with patients. This is important because prior research in ophthalmology shows that tailoring patient educational materials to a patient’s health literacy level can empower patients to make informed eye care decisions and ultimately result in improved patient outcomes.^26–28 Additionally, the use of LLMs to translate patient education materials to other languages could help to address language barriers in healthcare that often result in worse patient outcomes.^29,30 Oculoplastics surgeons who are familiar with the ability of ChatGPT to customize patient education materials for patients with varying health literacy levels can use the tool to help ensure that all patients understand the basics of their eye condition when engaging in shared decision making regarding treatment options.

There are limitations to our study. First, the 10 oculoplastics conditions and 8 oculoplastics procedures utilized in this study represent a small fraction of all oculoplastics conditions and procedures, respectively. As such, our findings may not be generalizable to patient education regarding other conditions and procedures not included in this study. However, both the conditions and procedures selected for inclusion in this study were chosen based on prior research, public interest based on Google search volume, and author consensus regarding the most frequent conditions and procedures, respectively, they encounter in everyday practice.^4,31 Next, while expert reviewers in our study are employed by six different academic institutions, these reviewers represent a small sample of all oculoplastics surgeons in the country. As such, subjective survey responses may not be representative of the opinions of all oculoplastics surgeons, given the wide variability in responses provided by expert reviewers. Next, details regarding the data sets used to train the ChatGPT LLM as well as its intended target audience are not publicly available. Therefore, it is plausible that some patient education websites that populate after Google search could have been used to “teach” ChatGPT how to respond to patient queries. The impact that Google search results have on learned ChatGPT responses is unknown at this time. Furthermore, our study utilized ChatGPT version 3.5 – it would be beneficial to further investigate future versions of ChatGPT to determine if improvements to the interface can continue to optimize patient education. Additionally, we utilized the Google Chrome browser to execute the Google search to provide answers to the 30 FAQs generated for this study. It is possible that other browsers may have populated different answers to the 30 FAQs; however, a repeat analysis of the answers which populated after Google search on the Microsoft Edge browser showed similar website links. Finally, while we used readability indices as a proxy for understandability of answers to patient queries and procedural explanations, the terms “readability” and “understandability” are not synonymous. It is possible that a patient may consider a response that is more “readable” to be more difficult to understand when compared with another response that is less “readable”; however, the validated readability indices used in this study are frequently utilized as a barometer of understandability of patient education materials for patients in both ophthalmology and other medical fields.^32–35

Conclusions

In conclusion, expert reviewers in our study found ChatGPT responses to oculoplastics FAQs to be more accurate when compared with Google responses, suggesting that ChatGPT has the potential to become a useful resource for patients seeking information about their oculoplastics conditions. Furthermore, we demonstrate the ability for ChatGPT to improve patient education materials by reducing the grade level at which procedural explanations are written, which can help oculoplastics surgeons tailor patient education efforts to patients with varying health literacy. A better understanding of oculoplastics conditions and procedures amongst patients can help lead to informed eye care decisions and may enhance patient satisfaction and outcomes.