Using Real-World Data to Measure Vaccine Effectiveness (VE)

Why Real-World Data for VE—and What Regulators Expect

Randomized trials establish efficacy under controlled conditions; real-world data (RWD) tell us how vaccines perform across ages, comorbidities, variants, and care systems over months or years. Post-authorization, decision makers want to know: Does protection wane? Do boosters restore it? Which subgroups (e.g., adults ≥65 years, the immunocompromised) need earlier re-dosing? RWD—immunization registries, EHR/claims, laboratory systems, and vital records—lets us answer these questions at scale. But credibility hinges on methods and documentation: explicit protocols and SAPs; auditable data pipelines; bias diagnostics (propensity scores, negative controls); and transparency about laboratory performance and manufacturing quality context. When lab results define outcomes, include analytical capability—e.g., RT-PCR LOD 25 copies/mL and LOQ 50 copies/mL (illustrative), or ELISA IgG LOD 3 BAU/mL and LOQ 10 BAU/mL—so case adjudication is reproducible. To pre-empt “non-biological” confounders in reviewer discussions, keep a short appendix with representative PDE (e.g., 3 mg/day for a residual solvent) and cleaning MACO limits (e.g., 1.0–1.2 µg/25 cm²) demonstrating stable manufacturing hygiene.

Regulators also expect ALCOA (attributable, legible, contemporaneous, original, accurate) for data transformations and outputs, and computerized-system controls (21 CFR Part 11 and EU Annex 11): role-based access, audit trails, validated backups, and time synchronization between sources. Build governance that connects clinical, epidemiology, statistics, safety, and quality—monthly boards reviewing KPIs, pre-declared decision thresholds, and version-locked code. For practical checklists to align SOPs and analysis artifacts, see PharmaRegulatory.in, and mirror terminology used by the European Medicines Agency in post-authorization guidance.

Core VE Designs with RWD: Cohort, Test-Negative, and Case-Control

Cohort designs. Follow vaccinated and comparator groups over time using Cox or Poisson models. Represent time since vaccination (TSV) via restricted cubic splines or pre-specified intervals (0–3, 3–6, 6–9, 9–12 months). Estimate hazard ratios (HR) or incidence-rate ratios (IRR) and convert to VE = (1−HR)×100% or (1−IRR)×100%. Adjust for calendar time, geography, and variant periods; include prior infection and booster status as time-varying covariates. Example (dummy): Adjusted HR for hospitalization 0.35 at 0–3 months → VE 65%; 0.58 at 6–9 months → VE 42%.

Test-Negative Design (TND). Restrict to symptomatic testers; cases are test-positives, controls test-negatives. TND reduces healthcare-seeking bias but assumes similar exposure/testing propensities. Always stratify by symptom criteria and testing policy periods, and run falsification checks (e.g., pre-rollout “VE” ≈ 0%).

Case-control. Useful for rare outcomes (ICU, death). Sample controls densely in time (risk-set sampling) and match on age, sex, geography, and calendar time; analyze with conditional logistic regression. Whatever the design, pre-declare subgroup analyses (≥65, immunocompromised), outcome tiers (ED visit, hospitalization, ICU, death), and decision thresholds that trigger communications or label updates.

Data Linkage & Quality: Turning Heterogeneous Sources into Analysis-Ready Sets

VE lives or dies on linkage. Combine immunization registries (dose dates, products, lots) with EHR/claims (encounters, comorbidities), laboratories (PCR/antigen/serology), and vital statistics (deaths). Use privacy-preserving linkage (hashing, third-party matching) and log deterministic/probabilistic match keys. Build an ETL with validation gates: impossible intervals (dose 2 before dose 1), duplicate vaccinations, outcome-date sanity checks, and cross-source concordance (admit/discharge vs diagnosis timestamps). Version-lock code and containerize (e.g., Docker) so re-runs reproduce hashes. Maintain a data dictionary and MedDRA/ICD-10 mapping under change control. Archive raw snapshots with checksums to satisfy ALCOA’s “original.”

Outcome adjudication must be explicit. Define laboratory thresholds and specimen rules (e.g., accept PCR Ct ≤ 35; resolve discordant antigen/PCR with repeat testing). If using biomarkers in severity tiers, declare the assay performance in the SAP: potency or infection assays with LOD/LOQ values. Keep a short “quality context” memo in the TMF with representative PDE and MACO examples to document that manufacturing and cleaning controls stayed in-spec while clinical effectiveness varied.

Governance, KPIs, and Decision Rules

Stand up a monthly Safety/Effectiveness Board to review dashboards and decide actions. Pre-define KPIs: cohort coverage (% registry-linked to EHR), lag from data cut to dashboard, capture of prior infection, VE at key TSV intervals, and subgroup VE. Quality KPIs include ETL error rate, linkage success, audit-trail review completion, and reproducibility checks (code hash). Establish decision rules such as: “If hospitalization VE in ≥65 years drops >10 points over a quarter with overlapping variant periods and no quality confounder, then recommend booster timing update and prepare HCP comms.” File minutes and decisions with supporting outputs in the TMF.

For hands-on SOP templates covering protocols, ETL validation, and inspection-ready reports, see pharmaValidation.in. Public terminology for post-authorization evidence can be cross-checked on the EMA website.

Modeling Waning & Boosters: Time-Since-Vaccination Done Right

Waning is not a single slope—it varies by age, risk, variant, and outcome. Treat time since vaccination (TSV) as a primary exposure. In Cox models, use restricted cubic splines (3–5 knots) or stepped intervals (0–3, 3–6, 6–9, 9–12 months). Interact TSV with age bands and immunocompromised status. For boosters, apply a biologically plausible grace period (e.g., 7–14 days post-booster) and model booster status as a time-varying covariate. Adjust for calendar time via strata or splines to absorb variant waves and policy changes; include prior infection as a time-varying variable. Report absolute risks (per 100,000 person-months) alongside VE to support policy decisions.

Bias control. Guard against immortal-time bias by aligning person-time precisely around dose dates and grace periods. Use propensity-score weighting/matching with calendar-time strata and geography to reduce confounding by indication. Deploy negative control outcomes (e.g., ankle sprain) and exposures (future vaccination date) to detect residual bias. In TND, vary symptom definitions and exclude occupational screens to test robustness. Where outcomes depend on assays, keep method transparency visible—e.g., RT-PCR LOD 25 copies/mL; LOQ 50 copies/mL—and preserve chain-of-custody. Tie everything back to ALCOA: version-locked code, timestamped cuts, and immutable raw snapshots.

Case Study (Hypothetical): A National VE Program that Drove a Booster Decision

Context. A country links registries, EHR, labs, and vital stats for 2.5 M adults. Findings (dummy). Hospitalization VE in ≥65 years: 68% at 0–3 months post-primary, 52% at 3–6 months, 36% at 6–9 months. Booster lowers HR to 0.28 (VE 72%) in months 0–3 post-booster, stabilizing at VE 60% by months 3–6. TND sensitivity analyses show VE within ±3 points; cohort and case-control designs converge on similar estimates. Negative controls are null; falsification in pre-rollout months ≈0% VE. Labs document analytical capability; adjudication rules are transparent. Quality appendix shows representative PDE 3 mg/day and MACO 1.0–1.2 µg/25 cm²; no manufacturing or cold-chain anomalies are linked to outcome spikes.

Action. The board applies pre-declared rules: “>10-point drop in ≥65s over a quarter with consistent bias checks → recommend booster at 6 months.” HCP materials are updated; an eCTD supplement compiles protocol/SAP, dashboards, and a reproducibility package (container hash, code, parameter files). Public comms explain denominators, absolute risks, and limits. The system continues monthly, ready to detect further waning or variant-specific changes.

Deliverables & Inspection Readiness: Make ALCOA Obvious

Create a simple crosswalk in the TMF: SOP → data cuts → code → outputs → decisions → labels/comms. For each cycle, file (1) protocol/SAP (and addenda), (2) data-cut memo (sources, versions, date), (3) analysis report with TSV curves and subgroup tables, (4) bias diagnostics (balance plots, negative controls), (5) reproducibility pack (code, containers, hashes), and (6) board minutes with decisions. Keep one internal link handy for your teams’ SOPs and validation templates—practitioners often adapt patterns from PharmaSOP.in—and cite a single external reference for public expectations; the ICH Quality Guidelines page is a concise touchstone to align vocabulary on validation and data integrity across functions.

Overcoming Data Quality and Standardization Challenges in Rare Disease Natural History Studies

Introduction: Why Data Quality Matters in Rare Disease Registries

Natural history studies are foundational in rare disease clinical development, particularly when traditional randomized trials are not feasible. However, the scientific and regulatory value of these studies heavily depends on the quality and consistency of the data collected. Unfortunately, due to heterogeneous disease presentation, multi-center variability, and resource constraints, maintaining data integrity in these registries is a substantial challenge.

High-quality data is essential for informing external control arms, selecting clinical endpoints, and gaining regulatory acceptance. Poor data quality or inconsistent data standards can compromise the interpretability of study outcomes and delay drug development timelines. Thus, sponsors and researchers must proactively address issues of data quality and standardization across every phase of natural history study design and execution.

Common Sources of Data Quality Issues in Natural History Studies

Natural history studies are typically observational, multi-site, and often global in nature. This introduces several challenges related to data consistency and quality:

• Variability in Data Entry: Different sites may interpret data fields differently without standardized CRFs
• Inconsistent Terminology: Disease phenotype descriptions often vary by clinician or country
• Missing or Incomplete Data: Due to long follow-up periods, participant dropouts, or loss to follow-up
• Lack of Real-Time Monitoring: Registries may not use centralized monitoring or data reconciliation processes
• Retrospective Data Integration: Retrospective chart reviews may introduce recall bias or incomplete datasets

Addressing these issues requires a combination of standard data frameworks, robust training, and system-level data governance.

Data Standardization: Role of CDISC and Common Data Elements (CDEs)

Standardization across sites and studies is a cornerstone for regulatory-usable data. Two critical components in this area are:

• CDISC Standards: The Clinical Data Interchange Standards Consortium (CDISC) offers the Study Data Tabulation Model (SDTM) and CDASH for standardized data capture and submission.
• Common Data Elements (CDEs): NIH, NORD, and other bodies define standard variables and definitions across therapeutic areas to harmonize data capture.

Using these standards ensures compatibility with clinical trial datasets, facilitates data pooling, and aligns with FDA and EMA submission expectations. For example, a neuromuscular disorder registry using CDISC CDASH standards demonstrated easier integration with an interventional study for regulatory submission.

Site Training and Protocol Adherence

One of the biggest drivers of data inconsistency is variation in how study sites interpret and apply protocols. Standardized training programs and manuals of operations (MOOs) can address this issue:

• Use centralized training sessions and site initiation visits (SIVs)
• Provide annotated eCRFs with definitions and data entry examples
• Create FAQs and real-time query resolution support for data entry teams
• Perform routine refresher training for long-term registry studies

These steps help align data capture across geographies and staff turnover, particularly in long-term registries that span years or decades.

Real-World Case Example: Registry for Fabry Disease

The Fabry Registry, one of the largest rare disease natural history studies globally, initially suffered from high variability in endpoint recording (e.g., GFR and cardiac metrics). By introducing standardized lab parameters, centralized echocardiogram readings, and CDISC compliance, data uniformity improved significantly.

This transformation enabled the registry data to be used successfully in support of label expansions and publications. Lessons from this case highlight the value of early planning and data harmonization.

Electronic Data Capture (EDC) and Source Data Verification (SDV)

Technology plays a central role in improving registry data quality. Use of purpose-built EDC systems enables:

• Real-time edit checks and logic validation (e.g., disallowing impossible age or lab values)
• Audit trails to track modifications and data queries
• Central data repositories with role-based access control

Source Data Verification (SDV) in observational studies, though less rigorous than trials, is still important. A sampling-based SDV strategy (e.g., 10% of patient records) can identify systemic errors and provide confidence in dataset quality.

“`html

Handling Missing Data and Outliers

Missing data is common in real-world observational research. Ignoring this problem can introduce bias and reduce the scientific value of the dataset. Strategies include:

• Imputation Methods: Use statistical techniques like multiple imputation or last observation carried forward (LOCF) based on context
• Clear Data Entry Rules: Establish consistent conventions for unknown or not applicable responses
• Monitoring Trends: Identify sites or data fields with high missingness rates

For example, in a rare pediatric lysosomal disorder registry, >20% missing values in a primary outcome measure led to exclusion from FDA consideration. After protocol revision and improved training, missingness dropped below 5% within a year.

Global Harmonization in Multinational Registries

Rare disease registries often span multiple countries and languages, creating additional complexity. Harmonizing data across regulatory regions requires:

• Translation of eCRFs and training documents using back-translation methodology
• Unit conversion tools (e.g., mg/dL to mmol/L for lab data)
• Standardizing outcome measurement tools across cultures (e.g., pain scales)
• Incorporating ICH E6(R2) GCP principles for observational studies

Platforms like EU Clinical Trials Register offer examples of harmonized study protocols across the European Economic Area (EEA).

Quality Assurance (QA) and Data Monitoring Strategies

Even in non-interventional registries, ongoing QA processes are essential. Key components of a QA plan include:

• Risk-Based Monitoring (RBM): Focus on critical variables and high-risk sites
• Central Statistical Monitoring: Use algorithms to detect unusual patterns or outliers
• Automated Queries: Generated by EDC systems based on predefined rules
• Data Review Meetings: Regular interdisciplinary discussions on data trends

These approaches reduce errors, enhance data integrity, and improve readiness for regulatory inspection or data reuse.

Metadata Management and Documentation

Every data element in a registry must be well-defined, traceable, and auditable. Metadata documentation helps ensure transparency and reproducibility:

• Define variable names, formats, and coding dictionaries (e.g., MedDRA, WHO-DD)
• Maintain version-controlled data dictionaries
• Log any CRF or eCRF changes with impact analysis
• Align metadata with data standards used in trial submissions

Metadata compliance facilitates smoother integration with clinical trial datasets and aligns with eCTD Module 5 expectations for real-world evidence inclusion.

Conclusion: Elevating Natural History Data to Regulatory Standards

Data quality and standardization are not optional in natural history studies—they are prerequisites for scientific credibility and regulatory utility. By adopting common data standards, leveraging technology, and investing in training and QA, sponsors can generate robust datasets that support clinical development and approval pathways.

With rare diseases at the forefront of innovation, high-quality observational data can accelerate breakthroughs, reduce time to market, and bring much-needed therapies to underserved populations worldwide.