Introduction

Patients, as healthcare recipients, play an essential role in evaluating the quality of care. Gathering, understanding and responding to patients’ voices is therefore a popular means of creating a humane healthcare system. Quantitative surveys have been widely adopted to capture patient feedback, and the survey results could serve as a cost-effective method to drive service improvement. For example, the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) surveys and Picker Patient Experience Questionnaire-15 (PPE-15) are widely used to measure and improve the quality of hospitalization. However, a major limitation of quantitative surveys is that positive replies are generally given for the closed questions,¹ which leaves little room for quality improvement. Moreover, previous studies have shown that quantitative data provide insufficient detail on the issues that are salient to patients and fail to drive service improvements.^2,3

To complement quantitative measures, open-ended questions with free-text comments are commonly included in patient experience surveys.^1,3,4 Evidence shows that when patients are presented with both patient narratives and quantitative data, they tend to pay more attention to the narratives.⁵ Open-ended questions add value to quantitative measures by offering information that matters most to patients and by providing detailed descriptions of the service issues that closed-ended items may not cover.²

Open-ended questions offer the opportunity to obtain substantial actionable information for quality improvement. However, feedback alone is far from sufficient. Free-text comments remain largely unexplored and underutilized,³ which may be related to the unstructured nature of these replies. Traditionally, extracting meaningful information from raw free-text data requires substantial effort. Cunningham and Wells³ manually conducted thematic analysis of 6961 free-text comments to identify the proportion of different sentiment and patient experience themes included in the comments. Although manual analysis has produced valuable outcomes, the surge in data volume resulting from information- based feedback mechanisms has given rise to an urgent requirement for scalable methodologies.^4,6

Natural language processing (NLP) techniques offer promising solutions for efficiently analyzing large free-text datasets. NLP can extract meaningful information and determine the discussion topics that occur in the text. Other industries, such as banks,⁷ the tourism industry⁸ and marketing,⁹ have been quick to embrace this technology to analyze users’ needs and preferences. The application of NLP to mine data in patient feedback has also emerged over the last decade.¹⁰ For example, Bovonratwet et al¹¹ used machine learning-based NLP to conduct sentiment analysis and topic modeling for 1048 patient comments and reported that 25% of comments were negative and 58% were positive, and the negative comments most frequently addressed room conditions and communication. However, analyzing the content of free-text comments remains a nascent technology to the health care industry, and there is little empirical evidence on the relationship between patients’ free-text feedback and their overall hospital rating. Moreover, studies on the use of free-text comments to capture patients’ needs and preferences have focused mainly on English-speaking users. Little is known about what the Chinese public talks about during their encounters with hospitals.

This study therefore applied a NLP approach to answer the following key questions: What aspects of care do patients discuss? How do patients perceive their hospital journey? And how do commonly expressed patient experience topics, particularly negative comments, correlate with variations in patient satisfaction and loyalty?

Methods

Design

This was a retrospective observational study that used routinely collected patient satisfaction data from June 2022 to June 2023 from a large national medical center in China.

Data Sources

The data were extracted from an electronic system used for collecting patient feedback. One day after discharge, patients were sent a mobile phone text message that included a link to a questionnaire on their care experience during hospitalization. In the text message, patients were informed that their responses would be utilized for analyzing care quality and pursuing a research objective, by completing the questionnaire the participants consented to publication of anonymized responses and direct quotes. If patients were willing to provide feedback, they clicked the embedded link and completed the questionnaire. Three days after the first sent, nonrespondents were sent a reminder. All data were extracted from the database and exported to a text file. The data consisted of the following:

• Patient demographic and diagnostic information, including age, sex, home address, health insurance, primary diagnosis, and length of hospital stay. These data were extracted from the hospital information system.
• Patients’ satisfaction and loyalty in relation to the healthcare service. Patients provided feedback on the questions “In general, how satisfied are you with your medical care?”, and “To what extent would you recommend your family members and friends to visit this hospital if needed?”. The satisfaction score was ranged from 0 to 10, and the recommendation score ranged from 0 to 5.
• Free-text comments (optional) captured patients’ feedback on the question “is there any comment you would like to make regarding the service you received?”

In total, 208065 patients were sent the SMS invitation and 38606 responded, with a response rate of 18.54%. Among the respondents, 72.67% (28054/38606) provided free-text comments. These 28054 feedback answers were analyzed, and this sample size was adequately powered to support both qualitative and quantitative analyses.

Data Pre-Processing

The content of the free-text comments was unstructured. For further processing and analysis, we cleaned these comments by removing incorrect punctuation, non-Chinese characters and additional spaces. It is challenging to perform accurate topic modeling for a large corpus, so stop-word removal was used to simplify the dataset.

Qualitative Content Analysis

This study randomly selected 20% of the sample data as learning samples to construct a prediction model. In order to build the model with the best prediction performance, the learning samples were divided into training set, validation set, and testing set with a 60%/20%/20% splitting ratio.¹² The training dataset was used as the learning template to estimate the model parameters. The validation dataset was used to estimate prediction error and avoid overfitting. The test dataset was used to estimate the final model performance. The learning sample was manually coded, while the remaining data would be automatically coded by a machine learning and deep learning approach.

Manual-Coding Approach

To increase the credibility of topic modeling, two researchers with expertise in patient experience independently coded 10% of the total comments from the inpatient survey to develop and refine the coding framework based on the 2012 NHS patient experience framework. The coding framework is shown in Table S1. Each comment was categorized into one or more patient experience themes from the framework. In addition, two researchers used the same dataset to perform sentiment analysis. According to the emotional attributes of the content, the researchers determined and labeled each comment as positive sentiment, neutral sentiment, negative sentiment or mixed sentiment. If the content was a complaint, it was labeled a negative sentiment, while if the content was praise, it was labeled a positive sentiment. If the content was entirely factual, it was labeled neutral (such as “The hospital staff should continue to work hard to create a wonderful service”). A portion of the comments carried two or more sentiments, such as, “When I asked questions, the doctors explained to me carefully and nicely. I like doctors! But the nurses were very impatient, and I felt that they didn’t want to spend much time with me”. This type of comment was classified as a mixed sentiment.

Manually coded comments were used as the learning template to categorize the remaining comments using machine learning or deep learning algorithms. The interrater agreement for each theme was calculated to limit personal bias. The interrater agreement (Cohen’s Kappa) between the two annotators ranged from 0.81 to 0.93, indicating a substantial agreement.¹³ During the process of coding, new codes were developed if new content appeared in the comments. Any disagreements were discussed by the team to reach a consensus on the appropriate theme and sentiment of the comment.

Machine Learning Model-Based Coding Approach

The patient experience topic modeling is cast into a multilabel classification problem. We applied six machine learning (ML) approaches to categorize the remaining comments and evaluated the performance of these approaches using the training dataset. The ML approaches included decision tree, support vector machine, logistic regression, XGBoost, multinomial naïve Bayes and random forest. According to the assessment results, the decision tree, support vector machine, and random forest had better performance, with high accuracy, precision, recall, and F-measures (Table 1). To obtain the best-performing model, we used a multiclassifier voting strategy to combine these three high-performing machine learning models to obtain the final classification result in the processing step. As shown in Table 1, the performance metrics of multiclassifier collaborative tagging were excellent. We therefore integrated a decision tree, support vector machine, and random forest using hard voting to construct a classifier to predict patient experience topic. We classified the remaining comments into one or more predefined categories and categorized their sentiment attributes.

Table 1 The Performance of Sentiment Prediction Models

Deep Learning Model-Based Coding Approach

Human emotions are complex, and open-closed comments include many mixed sentiments that require contextual analysis. When comments are reviewed manually, contextual information can be accurately analyzed, but this approach is challenging for machine learning method. A new language representation model-BERT, which stands for Bidirectional Encoder Representations from Transformers, has high performance in text-based emotion detection.¹⁴ Therefore, we used a BERT-based model to extract the sentiments in patients’ comments. Preclassified data were used as the training set. As shown in Table 2, the performance metrics of the BERT-based model were far better than those of the machine learning models. Furthermore, patient experience comments were classified into five distinct emotion categories, namely happy, angry, sad, surprised, and afraid.

Table 2 The Performance of Patient Experience Themes Prediction Models

Quantitative Analysis

The statistical analysis was conducted using Python and IBM SPSS Statistics 26. To efficiently extract and count each topic, all qualitative data were binarized to address multilabel classification using one-hot encoding. Then, the machine-coded data were imported into SPSS software to describe the characteristics of the discussion topics and sentiments of the comments and to calculate interrater agreement using Cohen’s kappa values. To identify which aspects of care patients complained would have an impact on their overall rating of the hospital, we used logistic regression to analyze the relationship between patient satisfaction/loyalty and the occurrence/nonoccurrence of patient experience topics within individual negative patient comments. Because the responses for patient satisfaction and patient loyalty were highly skewed with most scores clustered at the high values, we primarily used the top box scoring method,¹⁵ whereby scores were dichotomized as 5 (maximum score) vs less than 5. Odds ratios were calculated. Independent variables were selected based on evidence from previous studies showing a significant relation to patient experience, such as sex, age, and length of hospital stay. All significance tests were two-sided, and the probability was considered significant when p was < 0.05. No missing data imputation methods were used. Participants who made comments that were with mixed, neutral, or positive sentiments were excluded from the analysis. The Logistic regression was also used to analyze differences in clinical or sociodemographic characteristics between those respondents who made comments, and those respondents who made no comments. We displayed frequently appearing words as “word clouds” to assess the frequency represented by the font size of each word in the comments.

Results

Characteristics of Free-Text Patient Experience Comments

In total, 208065 patients were sent an invitation message; 38606 responded and completed the survey, with an 18.54% response rate. Of those respondents, 72.67% (28054/38606) provided free-text comments. The largest numbers of comments were about nurses (20.15%, 5654/28054), followed by doctors (11.02%, 3092/28054), health care assistants (1.42%, 398/28,054), and nonhealthcare workers (0.10%, 27/28054). Furthermore, 2.17% (609/28054) of the comments that used the term “medical staff” without a particular object. The remaining comments pertained to the environment, medical equipment, or lacked an object.

There were differences in clinical and sociodemographic characteristics between respondents who made comments and those who did not (Table 3). Women, elderly patients, surgical patients, patients without spouses, patients without medical insurance, and patients with lower satisfaction levels and with longer lengths of hospitalization were more likely to comment, while respondents diagnosed with cancer were less likely to comment.

Table 3 Results of Multivariate Logistic Regression Analysis of Factors Affecting Patients’ Behavior of Leaving a Comment

Performance Metrics of Machine Learning Models and Deep Learning Models

Table 2 illustrates the performance metrics of the machine learning models and deep learning models. The accuracy, precision, recall, and F-measures of the integration of the decision tree, support vector machine, and random forest methods for patient experience themes were 0.98, 0.77, 0.78, and 0.78 respectively. For patient experience sentiment, the accuracy, precision, recall, and F-measures for the deep learning models were 0.91.

Sentiment Analysis

Of the 28054 respondents, 69.0% (19356/28054) provided positive comments, 18.0% (5042/28054) provided negative comments, 9.7% (2731/28054) provided neutral comments, and 3.3% (925/28054) provided mixed comments. Positive comments (average 9 words) tended to be shorter, more generic and less detailed than negative comments (average 28 words) and mixed comments (average 47 words). Participants who were older, local or single or not diagnosed with cancer were more likely to leave negative comments (Table S2). Findings from the zero - shot emotion identification indicated that the happy emotion had the highest prevalence, amounting to 48.2% (13522/28054) of the total. Subsequently, the surprised emotion accounted for 16.2% (4544/28054), while the angry, sad, and afraid emotions comprised 15.4% (4321/28054), 13.4% (3759/28054), and 6.8% (1908/28054) respectively.

Patient Experience Themes

Of the 28054 respondents, 16410 provided general comments, such as, “very good” or “very satisfied”, while the remaining 11644 commented on certain aspects of care. There were 22 patient experience themes discussed in the open-ended comments (Table 4), and 26.7% (3114/11644) comments discussed more than one theme. Box S1 includes some specific examples of each theme.

Table 4 The Sentiment Distribution of Patient Experience Themes

As shown in the Table 4, among the respondents who commented on certain aspects of care, the five most commonly mentioned themes were about the “humanity of care” (28.28%, 3293/11644), followed by “information, communication and education” (14.25%, 1659/11644), “food” (13.17%, 1534/11644), “technical competence” (11.04%, 1286/11644), and “ward environment” (10.43%, 1214/11644). The five most common themes in the positive comments were the “humanity of care”, “efficacy of treatment”, “sense of responsibility of staff”, “technical competence”, and “food”, while the five most common themes in the negative comments were “humanity of care”, “information, communication and education”, “ward environment”, “food”, and “access to care”. Word clouds were created to present a visual representation of the text data (Figure 1). Whether in all comments or in positive and negative comments, the proportion of “humanity of care” is the highest. The sentiment distribution of each topic is reported in Table 4.

Figure 1 Word clouds of patient experience themes in free-text comments.

The Relationship Between the Occurrence of Negative Comments and Patient Loyalty

As shown as Table 5, the regression analysis indicated that the occurrence of negative comments about “humanity of care”, “information, communication, and education”, “sense of responsibility of staff”, “technical competence”, “responding to requests”, “continuity of care”, and “standardization of the care procedure” was significantly associated with worse patient loyalty (OR = 0.424, 95% CI = 0.359~0.501; OR = 0.449, 95% CI = 0.367~0.550; OR = 0.474, 95% CI = 0.246~0.912; OR = 0.484, 95% CI = 0.339~0.691; OR = 0.495, 95% CI = 0.326~0.752; OR = 0.589, 95% CI = 0.403~0.860; OR = 0.598, 95% CI = 0.375~0.952). The occurrence of negative comments about “humanity of care”, “information, communication, and education”, “sense of responsibility of staff”, “technical competence”, “responding to requests”, and “continuity of care” was significantly associated with a worse overall satisfaction (OR = 0.400, 95% CI = 0.340~0.470; OR = 0.537, 95% CI = 0.444~0.649; OR = 0.456, 95% CI = 0.241~0.863; OR = 0.688, 95% CI = 0.497~0.952; OR = 0.443, 95% CI = 0.293~0.670; OR = 0.635, 95% CI = 0.444~0.909). The occurrence of negative comments about other patient experience themes had no impact on patient satisfaction and their loyalty.

Table 5 The Relationship Between the Occurrence of Negative Comments and Patient Satisfaction and Their Loyalty

Discussion

Providing free-text comment boxes enables patients to freely discuss particular aspects of the health care service that are important to them or that have an impact on their overall experience. In this study, 72.67% of the participants provided free-text comments, indicating that patients are active in providing their feedback. Natural language process technology was used to process large amounts of free-text patient experience responses efficiently and to mine meaningful and actionable information for improvement.

Free-text patient experience feedback is unstructured and the texts are always multilabeled, which means that patients discuss more than one topic and that their narrative can be assigned to two or more labels. In this study, 11.10% (3114/28054) of patients commented on two or more specific aspects of care. Therefore, patient experience topic modeling is a multilabel classification task. This type of task is often considered more challenging than single-label text classification.¹⁶ A traditional method for handling the multilabel classification problem is to decompose it into multiple independent binary classification tasks. To address this issue, this study integrated the decision tree, support vector machine, and random forest methods and used a hard voting method to propose multilabel learning algorithms. The machine learning-based multimodel voting ensemble strategy achieved an accuracy of 0.98 for multiclassification tasks. This performance was higher than that of any individual machine learning model, indicating its robust classification performance on the label-imbalanced dataset.¹⁷ Compared with individual models, ensemble models exhibit overall better performance in many other industries.¹⁸

To address the complex sentiment elements included in the patients’ comments, a new language representation model, BERT, which stands for Bidirectional Encoder Representations from Transformers, was used. The BERT approach achieved accuracy of 0.91 for sentiment analysis, indicating its excellent classification performance. Its performance was better than the machine learning model. The sentiment analysis showed that 69.0% (19356/28054) participants provided positive comments, other existing NLP research¹¹ in healthcare also found the positive emotion accounted for the largest share.

The analysis of the comments revealed that the majority were about nurses and doctors, indicating that interpersonal interactions are patients’ main issue of interest during their encounters with hospitals. The positive-to-negative comment ratio was 1:0.26, demonstrating that although most participants experienced positive care, a noteworthy minority reported a negative hospital experience. Moreover, previous studies have shown that negative comments have a greater value for driving changes than positive comments.^19,20 Our study also found that negative comments tended to be longer and more detailed than positive comments and had much richer information. The health care system should monitor the rise of negative voices against services.

Among the patients’ discourses included in this content analysis, a wide range of themes were discussed. These themes were divided into interpersonal and functional aspects. The interpersonal aspects included humanity of care, information, education and education, privacy protection, involvement of family members, and responding to requests, which constituted the greatest proportion of patient experience topics. Larson et al²¹ also stated that patient experience mainly reflects the interpersonal aspects of health care services. This is especially true for the humanity of care, which has been universally discussed and demonstrated to be the critical attribute of patient experience and satisfaction in previous research.^22,23 This study found that themes associated with positive and negative emotional feedback both focus on the humanity of care (eg, “rude”, “friendly”). Rude behavior encountered by patients may trigger dissatisfaction, while friendly behavior may receive praise. Maramba et al²⁴ conducted a textual analysis of free-text comments from patients and found that “rude” was significantly associated with a worse experience. Wofford et al²⁵ also reported that patients consistently complained about the interpersonal aspects of care. Improving the interpersonal aspects of care therefore plays a critical role in managing patient experience.

Functional aspects included access to care, food, ward environment, technical competence, efficacy of treatment, physical comfort, error in treatment and coordination of care. Similar findings were presented in previous studies.²⁶ Many topics appear frequently in traditional hospital-initiated surveys (eg, access to care and physical comfort). There are also some topics that are not typically addressed, such as after-discharge care and coordination of care, indicating that health care organizations are able to use open-ended responses to identify unexpected aspects of care that may not be apparent to hospitals.⁹

Although a wide range of patient experience themes were discussed, this study found that patients had different preferences for specific aspects of care. We analyzed the relationship between the occurrence of negative comments and overall satisfaction and patient loyalty and found that the occurrence of negative comments about “humanity of care”, “information, communication, and education”, “sense of responsibility of staff”, “technical competence”, “responding to requests”, “continuity of care”, and “standardization of the care procedure” was significantly associated with worse patient satisfaction and loyalty, while the occurrence of negative comments about other aspects of care, such as “ward environment”, “equipment”, and “food” had no impact on patient satisfaction and loyalty. This suggests that different aspects of healthcare service have varying impacts on patient experience. For example, although the ward environment and equipment leave a bad impression on patients, it may not affect their overall evaluation of service quality. On the contrary, when there is a lack of humanistic care in services, insufficient timely and proactive response to needs, or unmet health knowledge needs, patients will have a poor hospital experience. Most of the discussion topics that have a significant impact on patient satisfaction and loyalty pertain to the interpersonal aspects of care. Therefore, this study underscores that the interpersonal aspects of care typically represent the “moment of truth” in a service interaction. When patients make a critical evaluation of these interpersonal aspects of care, they are less likely to recommend their family members or friends to visit the hospital, demonstrating that the interpersonal aspects of care are particularly important to patients. Efforts to enhance interpersonal aspects of care—such as communication skills, empathy training, and care coordination—remain crucial for delivering truly patient-centered care, as these aspects directly influence how patients perceive and engage with their healthcare experiences.

Limitations

This was a single-center study, and our findings therefore may not be generalizable. However, our hospital is a national hospital, and this study is the first to analyze free-text patient experience comments in China. Thus, we suggest that this research provides a starting point for Chinese hospital administrators and clinicians to consider how free-text patient experience comments can assist with health care improvement. In addition, hospital-originated survey may influence the content of patient feedback, while online platforms allow individuals to openly gather, communicate, and share information about their interactions with healthcare services, becoming an essential means of understanding patient experience. Future research should consider the value of online platform. Moreover, the sentiment labelling is a subjective process, and further research should explore other approaches to further validate the findings in this study.

Conclusions

The five most frequent patient experience discussion topics were “humanity of care”, followed by “information, communication and education”, “food”, “technical competence”, and “ward environment”, highlighting the interpersonal and functional aspects of care. The occurrence of negative comments about “humanity of care”, “information, communication, and education”, “sense of responsibility of staff”, “technical competence”, “responding to requests”, and “continuity of care” was significantly associated with worse patient satisfaction and loyalty, demonstrating that the interpersonal aspects of care may hold particular significance for patients.