Introduction

Large language models (LLM) are systems that leverage artificial intelligence to generate human-like responses to various queries.¹ LLM chatbots have gained significant popularity and are increasingly used in healthcare education.² There are a variety of freely available LLMs for patients to use, including ChatGPT, Gemini, and Copilot. As LLMs are trained using material on the internet, LLM-generated material may reflect racial and/or gender bias inherent to current internet content.^3,4 Concerns about the risk of codifying racism and sexism into machine learning are shared by patients and the scientific community alike.⁴ As reflected in a study by the Pew Research Center, 51% of Americans who perceive racial and ethnic bias in healthcare believe artificial intelligence, upon which LLMs are based, will make inequities worse.⁵ Investigating these biases is crucial to avoid perpetuating further inequity in medicine.

The information patients obtain through these resources may significantly influence care outcomes. Potential biases in the generated responses based on patient demographics may introduce outcome disparities.⁶ However, the effect of gender, race, and ethnicity variables on LLM-generated patient education material has not been previously evaluated.

In this study, we evaluated the effect of gender, race, and ethnicity modifiers in the prompts used to generate patient education materials about myopia with LLMs.

Materials and Methods

This study protocol did not involve human subjects and was exempt from Institutional Review Board approval as the generated data did not interact with patients or their private information. Study protocols that do not involve human subjects are exempt from the requirements of Institutional Review Board approval.⁷ All the data was generated using freely available online resources.

Data Collection

ChatGPT, Gemini (formerly Google Bard), and Copilot (formerly Bing Chat) were provided a standardized prompt incorporating demographic data modifiers (gender, race, and ethnicity) to inquire about myopia: “I am a [race or ethnicity] [gender]. My doctor told me I have myopia. Can you give me more information about that?” The racial categories tested were Asian, Black, American Indian, and White, and the only ethnic category tested was Hispanic. These categories were selected because they represented the most populous racial and ethnic groups, as detailed by the 2023 US Census Bureau.⁸ Gender was limited to male or female, and patient age was omitted from the prompts. The prompt was inserted five times into new chat boxes each time. At the time of this study, OpenAI’s GPT-3.5 used offline training data up to September 2021.⁹ Though GPT-4 was available at the time of this experiment, the decision was made to use GPT-3.5 as it is freely accessible. This experiment was designed with the average user in mind, assuming they would rely on freely available tools and may not have access to premium plans. Gemini does not disclose up to what date the software was trained. Copilot was the only language model in this study with access to real-time information via Bing Search, which could impact the responses generated. However, Microsoft states that chats with the model do not contribute to the model’s training.¹⁰ We used the default “more balanced” conversational model with Copilot. This conversational mode is a balance of the Creative mode, which generates longer and more descriptive responses, and the Precise mode, which generates shorter, more direct answers.¹¹

Readability Measures

The generated responses were collected and analyzed for readability by word count, Simple Measure of Gobbledygook (SMOG) index, Flesch-Kincaid Grade Level, and Flesch Reading Ease by copying the output into https://charactercalculator.com/smog-readability/ and https://charactercalculator.com/flesch-reading-ease/ (Table 1).¹² These readability formulas rely on the standards set by the McCall-Crabbs Standard Test Lessons in Reading serves as the gold standard for validating comprehension which assesses the understanding of varying reading difficulties by various known grade level students, so cut off scores would indicate expected comprehension for the assigned grade level.

Table 1 Calculation and Interpretation of Validated Readability Metrics

These measures are validated to assess the readability of a variety of sources and are among some of the most used readability formulas in healthcare literature, with the Flesch-Kincaid (57.42%) and Flesch Reading Ease (44.52%) being the most commonly used.¹³ Though these formulas are validated to assess the readability of a variety of sources, the SMOG index has been suggested to be more appropriate for healthcare information due to its use of more recent validation criteria, its validation against 100% comprehension, and its results’ consistency.¹³

Statistical Analysis

Descriptive statistics were generated using IBM SPSS Statistics 23.0. To assess the assumptions required for the application of analysis of variance (ANOVA), the homogeneity of variances across groups for all variables was tested. The assumptions of independence, normality, and homogeneity of variances were met when examining one large language model at a time. A multivariate ANOVA was conducted, and statistical significance was assessed for each factor and interaction based on the tests of Pillai’s Trace, Wilk’s Lambda, Hotelling's Trace, and Roy’s Largest Root. Post-hoc testing to evaluate pairwise differences was performed for other elicited differences.

When comparing the responses of large language models to one another, the assumption of homogeneity of variances was not met, as evaluated through Levene’s Test of Equality. For comparison between large language model groups, Welch’s ANOVA was conducted and reported.

Results

A total of 150 responses were generated (a combination of 3 LLMs, 2 genders, 5 race or ethnicity categories, and 5 trials each). 50 responses were analyzed for each LLM (n=50). Higher scores on the Flesch-Kincaid Grade Level and SMOG Index indicate harder-to-read material. Higher Flesch Reading Ease scores denote easier-to-read material. There was no significant difference in the SMOG index of responses generated by the three chatbots (p=0.55) (Table 2). Word count, Flesch-Kincaid Grade Level, and Flesch Reading Ease were significantly different between responses generated by the three chatbots (p<0.05) (Table 2). ChatGPT generated the shortest responses, followed by Copilot and Gemini (Table 2). Based on the Flesch-Kincaid Grade Level and Flesch Reading Ease score, Copilot generated the least readable responses, and ChatGPT generated the most readable responses (Table 2).

Table 2 Descriptive Statistics of Readability Divided for Each Large Language (ChatGPT, Gemini, Copilot)

When evaluating ChatGPT- and Copilot-generated responses, the gender, race, or ethnicity mentioned in the prompt did not significantly affect the word count, SMOG index, Flesch-Kincaid Grade Level, or Flesch Reading Ease score (Table 3a and b). In contrast, these readability measures were significantly different among Gemini-generated responses based on the race modifier of the prompt (p<0.05). Gender modifiers did not significantly affect readability measures among Gemini-generated responses (Figures 1–3). Gemini showed a significant effect of race/ethnicity on all tested dependent variables as well as a combined interaction effect of gender and race/ethnicity (p<0.05). When comparing the SMOG indices between race modifiers in a pairwise fashion, the SMOG index was significantly higher for responses generated with prompts mentioning a Black race than those mentioning a Hispanic ethnicity (p=0.03) (Figure 1 and Table 3c). Flesch-Kincaid Grade Level was significantly higher for responses generated with prompts mentioning an Asian race than those mentioning a Hispanic ethnicity (p=0.004) (Figure 2 and Table 3c). Flesch Reading Ease score was significantly higher for responses generated with prompts mentioning a Hispanic ethnicity than those mentioning an Asian race (p=0.04) (Figure 3).

Table 3 Sub-Analysis of Descriptive Statistics of Readability Divided for Each Large Language (ChatGPT, Gemini, Copilot)

Figure 1 SMOG Index of Gemini-generated responses with race or ethnicity modifiers, independent of gender modifiers. Error bars represent 95% confidence intervals. **indicates a significance value less than 0.05.

Figure 2 Flesch-Kincaid Grade Level of Gemini-generated responses with race or ethnicity modifiers, independent of gender modifiers. Error bars represent 95% confidence intervals. **indicates a significance value less than 0.05.

Figure 3 Flesch Reading Ease of Gemini-generated responses with race or ethnicity modifiers. Error bars represent 95% confidence intervals. ** indicates a significance value less than 0.05.

Discussion

Disparities in healthcare outcomes stem from a variety of factors. The increasing ubiquity of LLMs in the healthcare environment introduces a conduit by which gender- and race-related biases from the internet can affect the quality of patient care. In our study, we found that the length and readability of generated patient education materials about myopia were unaffected by the patient demographic modifiers of gender, race, or ethnicity when using ChatGPT or Copilot, while Gemini generated less readable materials for prompts mentioning a Hispanic ethnicity compared to prompts mentioning a Black or Asian race. Gender alone did not significantly impact the readability of generated materials. Variation of race and ethnicity modifiers, as well as the interaction between gender, race, and ethnicity, demonstrated significant differences in the readability of Gemini-generated responses. This pattern may reflect historically rooted stereotypes about racial and ethnic educational disparities that have been narrowing for decades.¹⁴

The racial, ethnic, and gender biases elucidated in this study can hinder optimal patient care by generating confusing or inaccurate material that may conflict with provider care, erode patient trust and reinforce existing health disparities. The readability of patient education material significantly impacts knowledge and behavioral changes that are crucial to patient compliance and satisfaction.¹⁵ Notably, among all three LLMs, the readability of the materials generated exceeded the sixth-grade level specified as the gold standard by the American Medical Association.¹⁶ Despite this guideline, the readability of patient-oriented educational materials sourced from high-impact journals remains above recommended levels and has not significantly improved from 1998 to 2018.¹⁷

Comparing LLMs, Copilot produced materials with significantly higher difficulty levels according to Flesch-Kincaid Grade Level and Flesch-Reading Ease metrics compared to ChatGPT and Gemini. Notably, there were no significant differences observed among the platforms when assessed using the SMOG index. Previous research shows that readability formulas like Flesch-Kincaid Grade Level and Flesch Reading Ease scores may underestimate readability for materials in which comprehension is crucial. Therefore, these measures may be less suitable for health education materials compared to the SMOG index, which includes criteria for expected complete comprehension.¹³

While previous studies have shown racial, ethnic, and gender biases in material produced by machine learning models, our study suggests these biases may not always manifest consistently across platforms.¹⁸ As each algorithm is developed on different selected materials, these can create variability in the biases expressed by different LLMs. Particularly noteworthy is the lack of detectable gender bias in the readability of materials from any platform. Prior literature identifies GPT-4 as perpetuating racial and gender stereotypes in other contexts by not appropriately modeling the demographic diversity of medical conditions.¹⁹ Our findings highlight the diverse biases inherent in different LLMs, emphasizing the need for caution when using these tools in clinical settings. Moreover, these findings underscore the need for continued efforts in training, bias detection, and further research to address the nuanced biases propagated by various language models as they continue to proliferate.

This study did not include age as a demographic variable, which may have generated more readable material based on suspected reading levels for different age groups. Beyond these findings, the LLM-generated responses varied by demographics, and future studies should examine the accuracy of responses. This study is also limited by the inherent unpredictability (stochasticity) of LLMs, and results may vary depending on when the LLM is prompted. All queries were conducted on the same day, and multiple trials were performed to reduce confounding factors. Another limitation of this study is inherent in the readability formulas used, which do not account for the imaging and videos included in the generated responses. This study followed a narrow line of inquiry about myopia, and the type of medical questions patients would ask remains unknown. This work could be replicated in a variety of contexts. Niche and rare diagnoses could further affect the readability of the generated material if information without medical jargon is less available.

Conclusion

In summary, race and ethnicity modifiers but not gender affected the length and readability of patient education materials about myopia generated by Gemini. ChatGPT and Copilot demonstrated no significant differences in readability based on demographic prompts. The breadth and accuracy of the information included based on LLM and demographic variables should be further investigated. Future studies could assess the accuracy of generated information and the readability of other disease processes to identify potential sources of misinformation. The potential impact of readability on LLM responses when prompting for certain ages could also be examined.