Carelon Research impact study: Using background incidence rates to inform vaccine safety
July 2022 | Written by Christopher Crowe, MPH, CPH and Daniel C. Beachler, PhD, MHS
In celebration of the 25th anniversary of Carelon Research Founder’s Day, we’re rolling out a series of articles highlighting some of our most impactful work. These ‘impact studies’ have contributed to the growing evidence base that influences healthcare decision makers and healthcare policy and practice. They also showcase the diverse service offerings at Carelon Research. Though all our work aims to improve the quality of healthcare, we’re especially proud of this work from research teams across our organization.
Post-market vaccine safety research and background incidence rates
Scientists and medical experts develop preventative vaccines, like the COVID-19 vaccines, which go through several stages of research including randomized controlled trials (RCTs) to evaluate their safety and efficacy.
RCTs are comparative studies that use randomization and scientifically rigorous methods that are sufficient for approval-related decision making. However, these studies may not have sufficient sample sizes to detect rare adverse events (for example, severe allergic reactions) or may exclude people with certain characteristics (for example, pregnant women).
Researchers then turned to real-world data (RWD) to gather more information on vaccines and monitor their safety and effectiveness in practice. Regulators, such as the US Food & Drug Administration, collaborate with Carelon Research and other organizations using data from insurance claims and medical records to support post-marketing vaccine safety or surveillance studies. This research helps regulators monitor the safety of new and existing vaccines following their initial approval and informs subsequent recommendations and approvals.
One milestone study by Carelon Research (formerly HealthCore) and collaborators, published in Vaccine in 2018 , described background incidence rates (IRs) of numerous safety events of interest estimated using claims data on 20 million individuals. These IRs can be used to estimate the rate of events (for example, heart attack, allergic reaction, Guillain-Barré Syndrome) we expect to see in a given population in the absence of a vaccine. These rates can then be compared to the event rates reported in newly vaccinated individuals to evaluate for potential safety signals..
Challenges of developing accurate background rates
It is recognized that background rates can vary widely across subgroups of the overall population (for example, pediatric versus adolescent versus adult), and standardizing demographics for background rates is essential when comparing them to post-vaccination safety event rates.
A less-recognized source of variation is from differing case definitions for a safety event. When using claims or medical record data, cases are usually identified using clinically informed algorithms (case definitions). These case definitions are a combination of codes for diagnoses, procedures, and treatments, but often lack validation and can be prone to false positive and false negative errors.
In our recent study, we examined the level of impact different case definitions may have on background IRs. We defined broader (sensitive) case definitions to identify as many cases as possible, and narrower (specific) case definitions to minimize the number of false positive cases. While it is known that background incidence rates can vary by case definition, the amount of variation across definitions has not been widely explored .
Background rates of safety outcomes relevant to pneumococcal vaccination in a US commercially insured population
The Safety and Epidemiology Research team at Carelon Research and researcher partners at Pfizer Inc. conducted a study exploring the background IRs of potential safety outcomes relevant to pneumococcal vaccination in a US commercially insured population, which was published in 2018 .
Researchers identified 32 safety events comprising cardiac, metabolic, allergic, autoimmune, neurologic, hematologic, and nephrologic outcomes and estimated the IRs for each of these outcomes using two case definitions – one prioritizing sensitivity and one prioritizing specificity among unvaccinated individuals who were in the target population for pneumococcal vaccination.
While some safety events had similar rates regardless of the case definition, the rates of other safety events varied markedly depending on the case definition. The rates of neurologic, hematologic, and nephrologic outcomes were particularly dependent on the case definition used. On average, the sensitive definition identified over three times as many cases compared to the specific definition for these outcomes.
This study provides context for potential safety events that may arise among children and adults vaccinated with a pneumococcal vaccine. More broadly, this study highlights outcomes where the case definition used can severely alter study findings and needs to be standardized when comparing vaccinated and unvaccinated (background) rates. This also provides motivation for further development of revised case definitions for these outcomes to be further explored in validation studies.
Utility of background rates for identifying new safety signals for new vaccines
This study used RWD to estimate background IRs to enable general population comparisons for newly vaccinated populations. As the use of RWD increases, it’s important to understand its strengths as well as its limitations to ensure the accuracy and reliability of results so that decisions made by regulators, healthcare providers, and patients are well informed to best serve public health.
This study highlights the utility of background rates and how they can be useful in identifying new safety signals for new vaccines. By using multiple case definitions and identifying outcomes that vary greatly depending on case definition, researchers can better target outcomes that would benefit from further refined definitions (algorithm development) and validation studies to provide more accurate results.
In future research, whether it be for the pneumococcal vaccines, COVID-19 vaccines, or other vaccines, researchers can consider the historical background IRs found in this study to contextualize further findings.
…
For more information about our safety & epidemiology capabilities, including vaccine safety, pregnancy outcome studies, single and multi-database post-authorization safety studies, risk evaluation and mitigation (REMS), validation studies and machine learning, medical record review and abstraction, and survey based research, visit our Safety & Epidemiology Services page.
