Using Real-World Data to Measure Vaccine Effectiveness (VE)

Why Real-World Data for VE—and What Regulators Expect

Randomized trials establish efficacy under controlled conditions; real-world data (RWD) tell us how vaccines perform across ages, comorbidities, variants, and care systems over months or years. Post-authorization, decision makers want to know: Does protection wane? Do boosters restore it? Which subgroups (e.g., adults ≥65 years, the immunocompromised) need earlier re-dosing? RWD—immunization registries, EHR/claims, laboratory systems, and vital records—lets us answer these questions at scale. But credibility hinges on methods and documentation: explicit protocols and SAPs; auditable data pipelines; bias diagnostics (propensity scores, negative controls); and transparency about laboratory performance and manufacturing quality context. When lab results define outcomes, include analytical capability—e.g., RT-PCR LOD 25 copies/mL and LOQ 50 copies/mL (illustrative), or ELISA IgG LOD 3 BAU/mL and LOQ 10 BAU/mL—so case adjudication is reproducible. To pre-empt “non-biological” confounders in reviewer discussions, keep a short appendix with representative PDE (e.g., 3 mg/day for a residual solvent) and cleaning MACO limits (e.g., 1.0–1.2 µg/25 cm²) demonstrating stable manufacturing hygiene.

Regulators also expect ALCOA (attributable, legible, contemporaneous, original, accurate) for data transformations and outputs, and computerized-system controls (21 CFR Part 11 and EU Annex 11): role-based access, audit trails, validated backups, and time synchronization between sources. Build governance that connects clinical, epidemiology, statistics, safety, and quality—monthly boards reviewing KPIs, pre-declared decision thresholds, and version-locked code. For practical checklists to align SOPs and analysis artifacts, see PharmaRegulatory.in, and mirror terminology used by the European Medicines Agency in post-authorization guidance.

Core VE Designs with RWD: Cohort, Test-Negative, and Case-Control

Cohort designs. Follow vaccinated and comparator groups over time using Cox or Poisson models. Represent time since vaccination (TSV) via restricted cubic splines or pre-specified intervals (0–3, 3–6, 6–9, 9–12 months). Estimate hazard ratios (HR) or incidence-rate ratios (IRR) and convert to VE = (1−HR)×100% or (1−IRR)×100%. Adjust for calendar time, geography, and variant periods; include prior infection and booster status as time-varying covariates. Example (dummy): Adjusted HR for hospitalization 0.35 at 0–3 months → VE 65%; 0.58 at 6–9 months → VE 42%.

Test-Negative Design (TND). Restrict to symptomatic testers; cases are test-positives, controls test-negatives. TND reduces healthcare-seeking bias but assumes similar exposure/testing propensities. Always stratify by symptom criteria and testing policy periods, and run falsification checks (e.g., pre-rollout “VE” ≈ 0%).

Case-control. Useful for rare outcomes (ICU, death). Sample controls densely in time (risk-set sampling) and match on age, sex, geography, and calendar time; analyze with conditional logistic regression. Whatever the design, pre-declare subgroup analyses (≥65, immunocompromised), outcome tiers (ED visit, hospitalization, ICU, death), and decision thresholds that trigger communications or label updates.

Data Linkage & Quality: Turning Heterogeneous Sources into Analysis-Ready Sets

VE lives or dies on linkage. Combine immunization registries (dose dates, products, lots) with EHR/claims (encounters, comorbidities), laboratories (PCR/antigen/serology), and vital statistics (deaths). Use privacy-preserving linkage (hashing, third-party matching) and log deterministic/probabilistic match keys. Build an ETL with validation gates: impossible intervals (dose 2 before dose 1), duplicate vaccinations, outcome-date sanity checks, and cross-source concordance (admit/discharge vs diagnosis timestamps). Version-lock code and containerize (e.g., Docker) so re-runs reproduce hashes. Maintain a data dictionary and MedDRA/ICD-10 mapping under change control. Archive raw snapshots with checksums to satisfy ALCOA’s “original.”

Outcome adjudication must be explicit. Define laboratory thresholds and specimen rules (e.g., accept PCR Ct ≤ 35; resolve discordant antigen/PCR with repeat testing). If using biomarkers in severity tiers, declare the assay performance in the SAP: potency or infection assays with LOD/LOQ values. Keep a short “quality context” memo in the TMF with representative PDE and MACO examples to document that manufacturing and cleaning controls stayed in-spec while clinical effectiveness varied.

Governance, KPIs, and Decision Rules

Stand up a monthly Safety/Effectiveness Board to review dashboards and decide actions. Pre-define KPIs: cohort coverage (% registry-linked to EHR), lag from data cut to dashboard, capture of prior infection, VE at key TSV intervals, and subgroup VE. Quality KPIs include ETL error rate, linkage success, audit-trail review completion, and reproducibility checks (code hash). Establish decision rules such as: “If hospitalization VE in ≥65 years drops >10 points over a quarter with overlapping variant periods and no quality confounder, then recommend booster timing update and prepare HCP comms.” File minutes and decisions with supporting outputs in the TMF.

For hands-on SOP templates covering protocols, ETL validation, and inspection-ready reports, see pharmaValidation.in. Public terminology for post-authorization evidence can be cross-checked on the EMA website.

Modeling Waning & Boosters: Time-Since-Vaccination Done Right

Waning is not a single slope—it varies by age, risk, variant, and outcome. Treat time since vaccination (TSV) as a primary exposure. In Cox models, use restricted cubic splines (3–5 knots) or stepped intervals (0–3, 3–6, 6–9, 9–12 months). Interact TSV with age bands and immunocompromised status. For boosters, apply a biologically plausible grace period (e.g., 7–14 days post-booster) and model booster status as a time-varying covariate. Adjust for calendar time via strata or splines to absorb variant waves and policy changes; include prior infection as a time-varying variable. Report absolute risks (per 100,000 person-months) alongside VE to support policy decisions.

Bias control. Guard against immortal-time bias by aligning person-time precisely around dose dates and grace periods. Use propensity-score weighting/matching with calendar-time strata and geography to reduce confounding by indication. Deploy negative control outcomes (e.g., ankle sprain) and exposures (future vaccination date) to detect residual bias. In TND, vary symptom definitions and exclude occupational screens to test robustness. Where outcomes depend on assays, keep method transparency visible—e.g., RT-PCR LOD 25 copies/mL; LOQ 50 copies/mL—and preserve chain-of-custody. Tie everything back to ALCOA: version-locked code, timestamped cuts, and immutable raw snapshots.

Case Study (Hypothetical): A National VE Program that Drove a Booster Decision

Context. A country links registries, EHR, labs, and vital stats for 2.5 M adults. Findings (dummy). Hospitalization VE in ≥65 years: 68% at 0–3 months post-primary, 52% at 3–6 months, 36% at 6–9 months. Booster lowers HR to 0.28 (VE 72%) in months 0–3 post-booster, stabilizing at VE 60% by months 3–6. TND sensitivity analyses show VE within ±3 points; cohort and case-control designs converge on similar estimates. Negative controls are null; falsification in pre-rollout months ≈0% VE. Labs document analytical capability; adjudication rules are transparent. Quality appendix shows representative PDE 3 mg/day and MACO 1.0–1.2 µg/25 cm²; no manufacturing or cold-chain anomalies are linked to outcome spikes.

Action. The board applies pre-declared rules: “>10-point drop in ≥65s over a quarter with consistent bias checks → recommend booster at 6 months.” HCP materials are updated; an eCTD supplement compiles protocol/SAP, dashboards, and a reproducibility package (container hash, code, parameter files). Public comms explain denominators, absolute risks, and limits. The system continues monthly, ready to detect further waning or variant-specific changes.

Deliverables & Inspection Readiness: Make ALCOA Obvious

Create a simple crosswalk in the TMF: SOP → data cuts → code → outputs → decisions → labels/comms. For each cycle, file (1) protocol/SAP (and addenda), (2) data-cut memo (sources, versions, date), (3) analysis report with TSV curves and subgroup tables, (4) bias diagnostics (balance plots, negative controls), (5) reproducibility pack (code, containers, hashes), and (6) board minutes with decisions. Keep one internal link handy for your teams’ SOPs and validation templates—practitioners often adapt patterns from PharmaSOP.in—and cite a single external reference for public expectations; the ICH Quality Guidelines page is a concise touchstone to align vocabulary on validation and data integrity across functions.

Pharmacovigilance for COVID-19 and Future Vaccines

Build the Right Pharmacovigilance Architecture: From Intake to Evidence You Can Defend

Post-marketing pharmacovigilance (PV) for COVID-19 vaccines—and for whatever comes next—requires a layered system that converts raw reports into defensible evidence. Start with intake and case processing that can scale: Individual Case Safety Reports (ICSRs) arrive via portals, email, call centers, and partner regulators. Your safety database should enforce E2B(R3) structure, MedDRA version control, and role-based access. Minimum case validity (identifiable patient, reporter, suspect product, and event) must be checked within 24 hours for seriousness triage. De-duplication rules (e.g., match on age/sex/onset/lot) are essential when media attention drives duplicate submissions. All edits and code changes must carry time-stamped audit trails consistent with Part 11/Annex 11, with ALCOA discipline visible in exported PDFs and XML acknowledgments filed to the TMF.

Once intake is stable, stitch passive reports to active, denominated datasets (claims/EHR, immunization registries) via privacy-preserving linkage. This lets you move from “someone noticed” to “how often relative to background.” Set up a governance cadence that blends clinical, epidemiology, statistics, quality, and regulatory. Every candidate signal should have a reproducible path: disproportionality screen → observed-versus-expected (O/E) check → sequential monitoring if needed → confirmatory study design (e.g., SCCS). Keep a one-page system map in your PV System Master File (PSMF) that links SOPs, databases, code repositories, and decision logs. For practical, regulator-aligned templates that speed SOP drafting, many teams adapt examples from PharmaSOP.in. For high-level public expectations and terminology you should mirror, consult the U.S. FDA.

COVID-19–Specific Practices That Should Become Standard: Speed, Adjudication, and Transparent Numbers

COVID-19 compressed safety decision cycles from months to days. Three practices deserve to persist. First, rapid cycle analysis (RCA) that updates weekly allowed earlier detection of real imbalances while controlling false positives; your protocol should pre-declare cadence, risk windows (e.g., myocarditis 0–7 and 8–21 days), and alpha-spending rules. Second, adjudication panels using Brighton Collaboration definitions turned noisy narratives into graded diagnostic certainty; maintain specialty panels (e.g., cardiology/neurology/hematology) and train them on uniform checklists. Third, transparent numbers build trust: when case definitions depend on biomarkers, state analytical capability—e.g., high-sensitivity troponin I LOD 1.2 ng/L and LOQ 3.8 ng/L for myocarditis confirmation; D-dimer assay LOD/LOQ for thrombotic events if relevant.

Quality context also matters. Reviewers routinely ask if manufacturing or hygiene could confound a safety pattern. Keep a succinct appendix that cites representative PDE (e.g., 3 mg/day for a residual solvent) and cleaning validation MACO limits (e.g., 1.0–1.2 µg/25 cm²) for the products and sites involved. Even though these are not “safety signals,” they reassure assessors that non-biological explanations (e.g., contamination) are unlikely, letting the analysis focus on biology and epidemiology rather than speculation.

Data Integrity, Dashboards, and What to Trend Every Month

A PV system that cannot show its own health will struggle in inspection. Define data-quality checks at intake (missing seriousness, impossible onset dates), coding (MedDRA drift), and analytics (version-locked code, reproducible seeds). Trend KPIs monthly and present them at Safety Governance: case validity within 24 hours, follow-up rate at 14 days, de-duplication yield, PRR screens reviewed on schedule, RCA boundary crossings, and time-to-decision for label actions. Implement a “completeness score” for ICSRs and route outliers to retraining. Keep external context visible by tagging media spikes and policy changes so you can explain bursts of reports without over-reacting.

Finally, make traceability obvious. Archive database cuts with date/time, software versions, and checksums; store adjudication minutes and decision memos in the TMF with cross-links to datasets and code. Run quarterly audit-trail reviews for privileged actions (case merges, code changes). When inspectors arrive, they should see a living system, not a static binder.

From Signal to Causality: PRR/ROR/EBGM → O/E → RCA → SCCS

Screening starts in spontaneous reports with disproportionality metrics. Pre-declare thresholds such as PRR ≥ 2 with χ² ≥ 4 and n ≥ 3; ROR with 95% CI excluding 1; and EBGM with lower bound (e.g., EB05) >2. These are hypothesis generators, not verdicts. Next, check observed versus expected using stratified background rates. Example (dummy): in one week, 1,200,000 second doses are administered to males 12–29; background myocarditis is 2.1/100,000 person-years. Expected in a 7-day window ≈ 1,200,000 × (7/365) × (2.1/100,000) ≈ 0.48. If six adjudicated Level 1–2 cases occur, O/E ≈ 12.5—strongly suggestive. If the program requires near-real-time oversight, initiate rapid cycle analysis (RCA) with MaxSPRT boundaries that control type I error across weekly looks. Confirm with self-controlled case series (SCCS), which compares incidence during risk windows (e.g., 0–7, 8–21 days) with control time within the same person, inherently controlling for fixed confounders. Declare how results drive actions: label updates, Risk Management Plan amendments, targeted studies, or enhanced monitoring.

Where laboratory markers define a case, keep the analytics transparent: assay LOD/LOQ, calibration certificates, and chain-of-custody for any central retesting. Maintain batch/lot traceability linking cases to distribution records; when regulators ask whether handling or hygiene could explain patterns, show that lots were in shelf life and under state-of-control with representative PDE and MACO examples already documented.

Case Study (Hypothetical): A Six-Week Path From Rumor to Label Action

Week 1–2: Passive screen. A cluster of myocarditis reports emerges in males 12–29, typically 2–4 days after dose 2; PRR 3.1 (χ² 9.8) and EB05 2.4. Narratives show chest pain and elevated high-sensitivity troponin I (above LOQ 3.8 ng/L). Week 3: O/E. 1.2 M second doses administered to males 12–29; expected 0.48 cases in 7 days; observed 6 adjudicated Level 1–2 → O/E 12.5. Week 4–5: RCA boundary crossed. MaxSPRT flags Days 0–7; clinical adjudication panel confirms Brighton levels. Week 6: SCCS. IRR 4.6 (2.9–7.1) for Days 0–7; IRR 1.8 (1.1–3.0) for Days 8–21. Action: label and RMP updated; Dear HCP communication drafted with absolute risks (“~12 per million second doses in young males within 7 days”) and guidance. Quality cross-check: lots in specification; cold-chain logs in range; representative PDE 3 mg/day and MACO 1.0–1.2 µg/25 cm² unchanged; no non-biological confounders found.

Future-Proofing: Governance for Next-Gen Platforms and Pandemics

mRNA, protein-adjuvant, and vector platforms will evolve; your PV governance should be ready before the next emergency. Pre-register AESIs by platform (e.g., myocarditis for mRNA, TTS for adenovirus vectors), their risk windows, and diagnostic packages. Maintain standing adjudication panels and reserve contracts for data access (claims/EHR/registries) with pre-approved protocols, so RCA and SCCS can start on Day 1. Keep communication templates that explain signal logic in plain language, include denominators, and link to public resources. Codify how manufacturing and distribution context is checked for every signal so quality questions do not derail medical decision-making.

Most importantly, make the record easy to follow. In your TMF and PSMF, keep a crosswalk that shows SOPs → data cuts → code → outputs → decisions → labeling. Version-lock code, archive database snapshots with checksums, and run scheduled audit-trail reviews. For method calibration, run periodic “negative control” screens to ensure the system is not over-signaling. When a real signal emerges, the combination of transparent thresholds, rapid analytics, clean documentation, and clear quality context will let you act quickly without sacrificing rigor.