Introduction

Real-world evidence (RWE) studies are typically used to investigate efficacy and safety of prescription drugs in routine clinical practice settings; the term real-world evidence in relation to healthcare appeared in the published literature at least as early as 1998.^1,2 These studies help to expand the examined population from that limited by clinical trial inclusion/exclusion criteria to a wide range of participants, allowing confirmation of safety and efficacy in the broader population.^1,3 RWE studies may also allow inclusion of distinct patient groups not included in clinical trials to be investigated. In addition, drugs can be studied over longer periods of time than are feasible with a randomized controlled trial (RCT). Perhaps more importantly, drug efficacy and safety can be assessed in a less supervised context than that seen in a clinical trial.

Variations in definitions of RWE frequently cause confusion in the interpretation and use of the real-world data used to generate RWE, especially in the regulatory setting. For the purposes of this review, we have followed the US Food and Drug Administration (FDA) definition of RWE as data collected during a noninterventional study of a marketed drug administered during routine medical practice according to a healthcare provider’s (HCP’s) clinical judgment.⁴ These studies can provide a range of important data and insights, including analysis of use patterns of marketed drugs as well as questionnaires, laboratory tests, or imaging studies. They can also be used to study off-label usage and to provide support or hypothesis generation for expanded indications and additional claims,⁵ as well as to determine important information about the clinical effects of drugs for rare diseases.

Typical RWE study designs include prospective and retrospective examination of registries, electronic medical records (EMR), and pharmacy and claims databases, as well as cross-sectional surveys.^1,4,6 EMR and claims databases, however, all require HCPs to actively follow patients or, at a minimum, to input relevant health data into the database. To help navigate development of RWE study data, regulatory agencies, including FDA, European Medicines Agency (EMA), Health Canada, the UK National Institute for Health and Care Excellence (NICE), Japan’s Pharmaceuticals and Medical Devices Agency (PMDA), and regulatory agencies in China have developed guidance for use of RWE studies to support regulatory applications.^4,7–11

Although Phase IV postapproval commitment studies, spontaneous reports registries (eg, V-safe), and other safety studies required by regulatory agencies report data collected in a real-world setting, they are not the focus of this review.^12,13 Interventional studies involving the use of drugs are usually defined as clinical investigations requiring a new drug application, so they do not represent RWE studies. However, observational real-world data may be of value for identifying potential participants for an interventional trial, to identify clinical endpoints (ie, hypothesis generating), or even as an external historical control for a single-arm study.

While most RWE focuses on prescription products and nonpharmacological interventions, study of nonprescription products (ie, medications and supplements) in this setting is of particular importance. Globally, with the greater awareness, emphasis, and adaption of self-care among the general population, as well as the associated switch of a number of products from prescription to nonprescription,¹⁴ there is an increasing need to generate RWE to understand how these products are experienced in a self-care setting. Recent guidance from the FDA¹⁵ and Medicines and Healthcare products Regulatory Agency¹⁶ has expanded RWE definitions to allow for RCTs when executed in real-world settings.

Collection of RWE for nonprescription products presents significant challenges. By definition, use of nonprescription products usually does not require involvement of HCPs, which can result in difficulties with patient recruitment, accurate data collection, and follow-up. As a result, methodological gaps exist in the research of nonprescription products in a real-world setting. The objectives of this narrative review are to identify RWE studies implemented for nonprescription products around the world and to explore the methodologies used to address some of these challenges, with a view to informing design of studies in this emerging area.

Materials and Methods

Literature searches in PubMed were conducted using the following terms: dietary supplement(s), nonprescription or over the counter, and real world or pharmacy to identify relevant studies. A second search was conducted using Google Scholar with terms over-the-counter or nonprescription, and real world or pharmacy. Additional articles were identified during full-text review. Included articles described studies of nonprescription drugs conducted in a community setting outside the context of a formal clinical study. For this investigation, only RWE studies that reported on analysis of patient effectiveness and/or safety data of nonprescription drugs were included.

Results

The initial PubMed and Google Scholar searches yielded a total of 1744 articles that were screened (Figure 1).¹⁷ After review of titles and abstracts, most of these were eliminated because they did not describe actual studies or analysis of patient data; were limited to assessment of a disease state rather than investigation of the effectiveness and/or safety of a drug or supplement; or focused on symptom patterns before treatment or attitudes toward disease and treatment. Finally, a total of 49 articles described different methods of generating RWE data in the nonprescription setting and were included in the qualitative synthesis. These methods included RCTs (n=1), prescription-to-nonprescription switches (n=3), cross-sectional surveys (n=27), database studies (n=12), case reports (n=1), and novel approaches (n=5).

Figure 1 PRISMA flow diagram.17 Abbreviations: PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RWE, real-world evidence.

Randomized Controlled Trials

One RCT of a nonprescription cough remedy using RWE data was identified.¹⁸ Patients enrolled in the study were adults aged 18 years and older who self-referred to a general practitioner or pharmacist owing to a cough of less than 7 days’ duration. The study had numerous inclusion and exclusion criteria. Briefly, patients with cough severity of at least 60 on a 100-mm visual analog scale were included, and patients with chronic cough and who smoked or used angiotensin-converting enzyme inhibitors were excluded. The primary endpoint was change in cough severity as recorded in a daily diary at 4 days of follow-up. This controlled study of a nonprescription product highlights one of the most effective ways of recruiting participants to such a study, ie, by a primary care physician or pharmacist.

Other reports, such as one examining the effects of a preparation containing two marine algae for treatment of metabolic syndrome, describe retrospective longitudinal studies.¹⁹ While such studies maintain many of the limitations of RCTs, in that inclusion/exclusion criteria limit the enrolled population (eg, by age, history of certain symptoms, use of specific prescription medications, and concomitant disease), they do not deliver the same quality of data as a formal trial because of their retrospective nature and lack of detailed follow-up. A number of other studies, both single-arm and comparative, that described themselves as RWE studies were excluded from the current review for these reasons.²⁰

Pragmatic trials are a variation of RCTs that incorporate elements of RWE data collection.²¹ These interventional studies retain many features of RCTs, while allowing for a broader patient population, and permit study of clinically meaningful outcomes beyond the narrow endpoints of RCTs. None were identified in our review.

Prescription-to-Nonprescription Switch

Label comprehension and actual use studies are critical steps in the switching of a pharmaceutical product from prescription to nonprescription. These studies help demonstrate that patients can appropriately self-identify that a particular drug is correct for their condition, and that they can safely follow directions for use without clinician oversight.²² Benefits of these studies include helping to identify the typical demographics of the patient population likely to take a nonprescription product and describing actual use patterns. One study showed that individuals (n=1999) approached in a shopping mall could determine with 90% accuracy that omeprazole for management of frequent heartburn was an appropriate treatment for them.²³ The study further demonstrated that subjects were able to follow directions with 79% accuracy. In a similar study, 249 participants with a history of nonprescription analgesic use of 5 or more doses per month for the previous 3 months were recruited via newspaper advertisements, on-site advertising at the pharmacy, online advertising, or direct mail.²⁴ After a telephone screening, participants were randomized to a self-selection study or a compliance study of ibuprofen immediate release (IR)/extended release (ER) 600-mg caplets. Overall, 69.1% correctly self-selected the ibuprofen IR/ER formulation, and 5 participants (1.2%) used the drug for more than 10 consecutive days, indicating a low degree of inappropriate use. No differences in usage were noted in select demographic comparisons. However, in terms of RWE, a key limitation of these studies is that, by design, they collect few if any data on the safety and effectiveness of the product being studied.

A recent extension of this type of study utilized outcomes modeling data based on the typical patient.²⁵ The benefit of a nonprescription progestin-only contraceptive was modeled based on an actual-use study of women who chose to buy the drug, including data on the method of contraception used before study enrollment. Effectiveness was determined by calculating the expected number of pregnancies in two hypothetical cohorts of 100,000 women using different contraceptive methods over 1 year, basing expected failure rates on published outcomes. The analysis determined that as many as 30,624 unintended pregnancies could be prevented through use of the nonprescription progestin-only contraceptive.

Cross-Sectional Surveys

The most common RWE studies of nonprescription products found in our literature search were cross-sectional surveys, which are observational studies that collect and analyze data at a single time point.²⁶ These surveys have been conducted in numerous settings including online, by phone, point-of-sale (pharmacy), outpatient clinics, and shopping malls. The studies cover a range of therapeutic areas including headache and osteoarthritis pain, smoking cessation, allergy/asthma, common cold, dermatology, gastrointestinal symptoms, and vulvovaginal candidiasis. Table 1 summarizes the cross-sectional surveys of nonprescription products identified in our review.^27–53

Table 1 Summary of Real-World Cross-Sectional Surveys

The sample sizes of the identified studies ranged from approximately 200 to 19,000 participants. It is beyond the scope of this review to determine the appropriateness of the sample size in each of these studies given the range of endpoints measured. However, as with any study, statistical power is predicated on the sensitivity of the endpoints in the studied population. Sample-size calculations should be based on the sensitivity and specificity of the endpoints and the anticipated drop-out rate, as with an RCT.

Most of the studies summarized in Table 1 report on drug/supplement usage patterns. Indeed, some studies, notably the smoking cessation studies, were designed to determine what products were being used by consumers to address their condition, while others were designed to assess how specific agents were being used, primarily to ensure proper use. Many of the studies, however, also reported on effectiveness (usually as some measure of symptom relief), patient-reported outcomes (including quality of life [QOL] and treatment satisfaction), and safety and tolerability. Prospective cross-sectional surveys, therefore, can provide data on how nonprescription products are being used and how consumers perceive their effectiveness in a real-world setting.

One study that reported the output from in-depth, semistructured telephone interviews with just 21 patients in the setting of smoking cessation was excluded from this review.⁵⁴ The data collected in this study were qualitative in nature and therefore did not meet our criteria for quantitative evidence that could be applied in a regulatory setting.

Database Studies

One of the most common forms of RWE studies for prescription products is database research.⁶ This approach can also be used for study of nonprescription products. However, the value of such databases depends on the quality and completeness of data collected, eg, whether complete information on nonprescription products or routine pain assessments was collected and entered into the database.⁵⁵ While some database studies are based on retrospective analysis of EMR, claims databases, and pharmacy databases, others are based on prospective, longitudinal, observational studies in which detailed information on a particular condition is collected. An example of a retrospective analysis of patient records is a study of metamizole in Brazil, where the drug is available on a nonprescription basis, to determine how it is used.⁵⁶ A total of 455,834 patients were recorded as having taken metamizole during the study period. The study found that the most common reason for use was pain (81%), followed by fever (19%); headache (19%), sore throat (8%), muscular pain (7%), and abdominal pain (5%). In another retrospective study of patient records, the effect of various nutraceuticals was studied in patients with dyslipidemia who were following a Mediterranean diet. In this study of 487 patients, the nutraceuticals were found to enhance the improvement of lipid profiles, although most patients did not achieve low-density lipoprotein–cholesterol goal.⁵⁷

A benefit of EMR analysis is the ability to query a very large population. An example of this is an analysis of 60,212 unique patients who had taken either ibuprofen 200 mg, naproxen 220 mg, or a combination of the two.⁵⁸ This study used International Classification of Diseases, Ninth Revision (ICD-9) codes to identify the proportion of these patients who experienced gastrointestinal complications. When the rate of complications on a nonsteroidal anti-inflammatory drug (NSAID) was compared with the time period 365 days before taking the drug, the odds ratio for gastrointestinal complications with naproxen was 1.54 (95% confidence interval [CI], 1.04–2.28; p=0.03) and with ibuprofen was 1.38 (95% CI, 1.07–1.78; p=0.01).

Pharmacy transaction data can also be used to generate RWE for nonprescription drugs. Three reports from Australia used the NostraData database, which provides a demographically representative dataset, to study allergic rhinitis.^59–61 The first of these studies was limited to examining the seasonality of oral antihistamine and intranasal corticosteroid purchases.⁶⁰ A second study was able to determine the extent of multitherapy use compared with recommended intranasal corticosteroids alone.⁵⁹ The final study showed that many patients are purchasing nonprescription oral antihistamines that are not clinically effective.⁶¹ The study further showed that patients with prescriptions for comorbid respiratory conditions were more likely to purchase intranasal corticosteroids.

A common approach to analysis of medical databases is propensity score matching, in which patients in the database taking a particular drug or supplement are matched with otherwise similar patients not receiving the intervention, giving the ability to control for confounding participant characteristics without reducing the power of the model by scoring multiple confounders into one variable.⁶² A number of such analyses have been conducted with nonprescription supplements. Two of these involved patients who did or did not take vitamin D. In one analysis, 10,974 patients hospitalized with heart failure and a history of vitamin D supplementation had a significantly lower risk of in-hospital mortality and mortality within 7 and 30 days of hospitalization than those who had not taken vitamin D.⁶³ In a separate analysis of a prospective, multicenter, longitudinal, observational cohort, 236 propensity score–matched patients with knee osteoarthritis who took vitamin D had no significant changes in pain or physical function during 2 years of follow-up compared with controls, consistent with RCTs.⁶⁴ A broader non-propensity-matched study conducted in the UK measured overall mortality among vitamin D and multivitamin users and nonusers.⁶⁵ Propensity score matching has also been used to study calcium supplementation in a Korean health claims database.⁶⁶ This approach has also been used to explore gender differences in smoking cessation effectiveness among users of varenicline or nicotine patches.⁶⁷

Case Reports

In some situations, no information on the role of certain products is available for specific populations. Case series reports can provide a limited amount of data and can be hypothesis generating. One such case series investigated the efficacy of a nonprescription dietary supplement of quebracho, conker tree, and Mentha balsamea Willd extracts over a 2-week period for treatment of irritable bowel syndrome in 24 patients.⁶⁸ Retrospectively compiled data found that symptoms of abdominal pain, constipation, and bloating improved in 88% of patients.

Novel Approaches

The use of smartphone apps to gather cross-sectional data has offered a new method for collection of RWE data. The Mobile Airways Sentinel Network (MASK) employed the mobile phone app Allergy Diary to collect information on allergic rhinitis symptoms in 9122 participants using 4 different intranasal corticosteroids and 8 oral antihistamines.²⁸ Data from 112,054 days were recorded. Although use of an app allows for ongoing follow-up that is not achievable with in-person telephone or online surveys, users were erratic in reporting symptom and treatment data. The study found that patients manage allergic rhinitis contrary to physician guidelines, increasing the number of treatments to control worsening symptoms, rather than adjusting dosage to achieve control. Data collected using the app showed that levocetirizine was the most effective oral antihistamine, and the study was able to differentiate between fluticasone furoate, mometasone furoate, and mometasone furoate plus azelastine. While this study demonstrates some of the weaknesses of online data collection, in that users did not enter data consistently and reliably, the volume of data collected allowed conclusions to be drawn; a pilot study of only 2871 users that preceded the above described study was likely too small to generate definitive conclusions, highlighting the need for a large sample size.^28,69 A strength of this approach, however, is that it may help to reduce recall bias associated with cross-sectional surveys since data are inputted by the subject in real time. This points toward models for future real-world cross-sectional surveys of nonprescription products, meeting consumers where they are in the modern world. In addition, the ability to collect a wide array of data with this approach may allow for study of novel concepts. While the introduction of such new technologies may present some regulatory hurdles, the challenges associated with smartphone apps, such as patient confidentiality and data security, are not different from more established survey techniques.

One study highlighted the possibilities of using Google Trends to model allergic rhinitis symptoms and use of oral antihistamines. While this preliminary publication is limited to demonstrating the seasonality of disease symptoms and associated drug uptake, it shows a possible novel approach to mine the vast quantity of data collected by Google.⁷⁰ The use of social media presents a similar opportunity for health monitoring and signal detection by mining online content, owing to the vast quantity of data collected.⁷¹ Although so far limited to prescription drugs, social media can be used to monitor not only for adverse events, but also for off-label drug use and drug-drug interactions. The artificial intelligence (AI) revolution has led to the development of natural language processing (NLP). NLP allows words and phrases to be mined and has the potential to identify information about specific drugs, diseases, and adverse effects. However, a limitation of social media is the inability to verify cause-effect relationships between a drug and a specific adverse event reported by an individual patient.⁷¹

An area where NLP has already been applied is in the analysis of EMR data.⁷² While structured EMR data have proven useful, until now, analysis of unstructured clinical notes, which often contain information about supplement and nonprescription drug use, has not been possible. A recent report conducted detailed surveys among 377 patients and compared the results with clinical notes using an NLP model. The NLP model achieved an F1 score (a measure of predictive performance) of 0.914, indicating a good performance of the model, although results varied for individual dietary supplements. The ability to conduct NLP analysis on EMR records may allow for more detailed research on the use of supplements and their potential effects.

Discussion

While RCTs may be the gold standard for generating robust clinical data with limited potential bias, RWE studies can provide information that informs day-to-day clinical practice in a less controlled environment that is closer to reality.³ This may especially be the case for nonprescription products that are taken without medical supervision and therefore may be taken under conditions different from those in which the product was originally studied and approved. RWE studies can help improve our understanding of effectiveness, safety, and usage patterns in populations larger and more diverse than can be studied in RCTs.^1,3 In addition, they can help provide an early assessment of effectiveness and tolerability, as well as generating long-term data. These studies can also provide a view of unmet needs by identifying future needs and data gaps. A weakness of RCTs is that the inclusion/exclusion criteria make for a homogeneous study population that does not represent all possible users. RWE data can circumvent this weakness by allowing a wider array of participants and a larger sample size, owing to possible variability in effectiveness across different groups.

One argument against using RWE database and registry studies to inform clinical practice is that they do not provide the same robustness of data as RCTs, especially in the nonprescription setting.^3,73 However, an appropriate study hypothesis supported by a protocol and statistical analysis plan that adjusts for the differences in population and data collection methods can account for these disparities in data integrity, accuracy, and quality. Another challenge of RWE data is the fact that EMR and claims databases are frequently incomplete when it comes to nonprescription drug use.⁷⁴ Although there is an awareness among experts regarding this deficiency, there is a need to educate HCPs on the importance of maintaining complete records. Statistical approaches toward handling any missing data in this context are required. It is recommended that propensity score matching be implemented in database studies to help reduce any confounding factors.

Cross-sectional surveys are the most common RWE studies of nonprescription products. These studies are not comparable to database and registry studies, which are based on clinical data. Indeed, participants surveyed outside of a medical setting do not represent a clinical population. As a result, these studies are limited by recall bias, ie, the accuracy and/or honesty of responses to survey questions by respondents, as well as potentially biased answers in response to leading questions, and social desirability bias.^26,55 Survey questions should therefore be designed to minimize possible misunderstanding or inaccurate responses, as well as to probe aspects of the disease and its treatment that are important to the patient, including health-related QOL. Implementation of validated patient-reported outcome instruments would be one way to achieve this.

AI approaches, including NLP, that mine existing data to develop hypotheses have an advantage over RCTs in that they, by definition, query in a retrospective fashion. RCTs by contrast require that all endpoints are prespecified before any patient receives a drug. All such research, whether mining social media or EMR, requires that individual patient identities remain anonymous in accordance with standard ethics laws and practices. More research is required to determine the full potential and limitations of AI in this setting. A limitation of this review is that because the term real-world is often ill defined and frequently not used in potentially relevant studies, the results of the literature search may not be all-inclusive.²⁰ As a result, additional methodologies for RWE study of nonprescription products may not have been identified.

Although this review did identify a number of RWE studies for nonprescription products, the question of who will conduct future RWE studies remains. Since nonprescription drugs and supplements are usually taken without the advice of an HCP, the people who are typically the drivers of medical research are not best placed to conduct the research. Furthermore, HCPs cite time constraints as a barrier to collection of complete patient medical histories for input to EMR and other databases.⁷⁵ This means that many real-world studies of nonprescription products are destined to be funded and carried out by companies that market the product, raising questions about conflicts of interest. To mitigate this potential bias, there is a need to create standards of RWE in terms of acceptable clinical endpoints and guidance for development of clinically robust patient-reported outcome surveys, as well as metrics for determining study quality. Implementation of these standards may allow for implementation of meta-analyses, which could further improve the credibility of RWE studies.

FDA, EMA, Health Canada, and Japan PMDA have all shown willingness to consider RWE data for inclusion in new drug approval applications or for line extensions of prescription drugs.⁵ Paying close attention to regulatory guidance for future design of RWE studies and patient recruitment may help improve study of nonprescription products and allow results to have a greater impact, especially in terms of driving expanded labels.^4,7–11 However, this guidance is still evolving and primarily directed at development of prescription drugs. There are few examples of nonprescription products gaining new or expanded indications using real-world data; some examples include Vicks cold medicine, Sudocrem cream, Bepanthen ointment, and flurbiprofen lozenges. It is important for real-world data to be of good quality and known provenance (eg, patient-level data) so they can be evaluated.^4,9

One of the key recommendations from EMA to expand the data available for different outcomes and exposure across regions is to improve the number, size, and type of data sources.⁷ These include EMRs from secondary care settings, biobanks, large claims databases, and disease or patient registries. EMA also recommended developing collaborations between regulatory agencies and external stakeholders. With respect to nonprescription products, a limitation of these sources is that data on these products are not universally included and/or accurately recorded. Mechanisms for facilitating and incentivizing collection of data on nonprescription products are required to fulfill the EMA recommendation.

Conclusion

The recognition of RWE studies by regulatory bodies as being important sources of data on the effectiveness and safety of drugs has led to increased interest in this type of data generation. Development of more complete databases that capture nonprescription drug and supplement use would facilitate real-world study of these products. Proper design of study protocols, whether the study is prospective or retrospective, will help to improve the quality and utility of data. Newer approaches, including use of smartphone apps, and AI, including NLP, may help to facilitate RWE studies of nonprescription products.