What a systematic review is (and is not)
The difference between systematic and narrative reviews, and why the method matters more than the conclusion
What a systematic review is
A systematic review is a research study. Its unit of analysis is other studies. It asks a defined question, searches for all relevant evidence using a reproducible strategy, appraises the quality of what it finds, and synthesises the results — quantitatively if possible, narratively if not. Everything it does is pre-specified, documented and transparent.
The word systematic is the operative one. What separates a systematic review from a narrative review is not length, authority or the number of references — it is method. A narrative review selects evidence to support an argument; a systematic review specifies its search, inclusion criteria, appraisal tools and synthesis method before it looks at the data, and reports what it finds whether or not the findings are convenient.
This distinction matters practically. A narrative review written by an eminent authority may be persuasive but is not reproducible — another reviewer, using different selection criteria, might reach a different conclusion. A systematic review can be reproduced, updated, and challenged at each step, because each step is documented.
What it is not
A narrative review is an expert summary. It is useful but not systematic — the author decides which studies to include, and the reader cannot verify the selection.
A scoping review maps the breadth of literature on a topic. It may use systematic search methods but does not appraise quality or synthesise effect estimates. Its purpose is to describe the landscape, not to answer a clinical question.
A meta-analysis is a statistical technique, not a study type. It combines the quantitative results of multiple studies into a pooled estimate. A systematic review may or may not include a meta-analysis — it depends on whether the included studies are sufficiently similar to pool. A meta-analysis without a systematic review (pooling a hand-picked selection of studies) is a methodological error, not a shortcut.
| Type | Method | Use |
|---|---|---|
| Systematic review | Pre-specified protocol, comprehensive search, quality appraisal, transparent synthesis | Answering a defined clinical or research question with all available evidence |
| Narrative review | Expert-selected literature, no required protocol | Providing an authoritative overview of a topic |
| Scoping review | Systematic search, charting of evidence, no quality appraisal | Mapping the literature landscape, identifying gaps |
| Meta-analysis | Statistical pooling of effect estimates | Quantitative synthesis within a systematic review (not a standalone study type) |
| Umbrella review | Systematic review of systematic reviews | Synthesising an evidence base that already has multiple reviews |
Why medical writers do this work
Systematic reviews and meta-analyses are among the most technically demanding documents a medical writer produces. They require methodological precision, statistical literacy, and the ability to write transparently about what the evidence does and does not show. They are also among the most influential — they sit at the top of the evidence hierarchy and directly inform clinical guidelines, regulatory decisions and health technology assessments.
A medical writer working on a systematic review typically contributes to protocol development, the writing of the manuscript, PRISMA compliance, and the presentation of results including forest plots and summary tables. Understanding the methodology — not just the reporting — is what separates competent writing from accurate writing.
Drills
A meta-analysis pools the results of multiple studies into a single estimate. If the studies it pools were selected by hand rather than identified through a comprehensive, pre-specified search, the pooled estimate reflects the selection rather than the totality of evidence. Studies with inconvenient results may have been omitted — deliberately or unconsciously — and the precision of the pooled estimate then gives a false impression of certainty. This is why meta-analysis is a technique used within a systematic review, not a substitute for one.
Formulating the question
PICO, eligibility criteria, the protocol and PRISMA-P — because the question determines everything that follows
The PICO framework
Every systematic review begins with a question, and the question determines everything that follows — the search strategy, the inclusion criteria, the comparisons, the outcomes and the analysis. A poorly defined question produces a review that either finds nothing relevant or finds everything and can synthesise none of it.
The standard framework for structuring a clinical question is PICO: Population, Intervention, Comparator, Outcome. For non-intervention reviews, the variant is PECO (Exposure replaces Intervention). For diagnostic accuracy reviews, it is PIRD (Index test, Reference standard, Diagnosis).
| Element | Definition | Example |
|---|---|---|
| Population | Who is studied — age, condition, setting | Adults (≥18 years) with non-valvular atrial fibrillation |
| Intervention | What is being evaluated | Direct oral anticoagulants (DOACs): apixaban, rivaroxaban, edoxaban, dabigatran |
| Comparator | What the intervention is compared against | Warfarin (dose-adjusted, target INR 2.0–3.0) |
| Outcome | What is measured — primary and secondary | Primary: stroke or systemic embolism. Secondary: major bleeding, all-cause mortality, intracranial haemorrhage |
Eligibility criteria
The PICO question translates directly into the eligibility criteria for the review. These specify which studies will be included and which excluded, and they must be defined before the search is run — not adjusted after seeing the results.
Eligibility criteria typically cover: study design (e.g., RCTs only, or RCTs and observational studies), population (as defined in P), intervention and comparator (as defined in I and C), outcomes (as defined in O), language restrictions (if any), and date restrictions (if any, with justification).
The decision about study design is consequential. Restricting to RCTs gives the highest internal validity but may miss important safety signals or real-world effectiveness data that only observational studies capture. Including observational studies increases the volume of evidence but introduces confounding. The choice depends on the question and should be justified in the protocol.
The protocol and PRISMA-P
PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) is a 17-item checklist for systematic review protocols. It ensures that the methods are documented before the review is conducted, reducing the risk of post-hoc changes driven by the results.
A protocol should specify: the rationale for the review, the PICO question, the eligibility criteria, the search strategy (including databases and search terms), the screening process, the data extraction plan, the risk of bias assessment tool, the planned synthesis (narrative, meta-analysis, or both), and the approach to assessing certainty of evidence (e.g., GRADE).
The protocol should be registered (PROSPERO for health-related reviews) or published before the review begins. Deviations from the protocol during conduct are permitted but must be documented and justified in the final report.
Drills
Population: Hospitalised adults (≥18 years) receiving systemic antibiotic therapy for any indication.
Intervention: Probiotics (any species, any formulation) administered concurrently with antibiotic therapy.
Comparator: Placebo or no probiotic co-administration.
Outcome: Primary: incidence of antibiotic-associated diarrhoea (defined as ≥3 loose stools per day for ≥2 consecutive days during or within 4 weeks of antibiotic therapy). Secondary: Clostridioides difficile infection, duration of diarrhoea, adverse events related to the probiotic.
Study design: Randomised controlled trials.
Key decisions documented: The population is restricted to hospitalised adults because the risk profile, antibiotic regimens, and infection control context differ from community settings. The intervention is deliberately broad (any probiotic species) to map the landscape; subgroup analysis by species and dose is planned. The comparator excludes active controls (e.g., one probiotic vs another) to answer the threshold question of whether probiotics work at all against no treatment.
The search strategy
Database selection, search syntax, Boolean logic, sensitivity and the documentation that makes a search reproducible
Database selection
No single database indexes all biomedical literature. A systematic search requires multiple databases, selected for their coverage of the topic. The minimum for a health-related systematic review is typically MEDLINE (via PubMed or Ovid), Embase (broader coverage of European and pharmacological literature), and the Cochrane Central Register of Controlled Trials (CENTRAL).
Depending on the topic, additional sources may be needed: PsycINFO for psychological interventions, CINAHL for nursing and allied health, ERIC for educational interventions, or regional databases such as LILACS (Latin America) or CNKI (China). Grey literature sources — conference abstracts, trial registries (ClinicalTrials.gov, ICTRP), dissertations, regulatory documents — reduce publication bias by capturing studies that were never published in journals.
Search syntax and Boolean logic
A systematic search translates the PICO question into a set of search terms combined with Boolean operators. Each PICO element becomes a concept block. Within each block, synonyms and related terms are combined with OR (broadening). Between blocks, concepts are combined with AND (narrowing).
Controlled vocabulary (MeSH in MEDLINE, Emtree in Embase) ensures that conceptually equivalent terms are captured regardless of how individual authors phrase them. Free-text searching with truncation (e.g., anticoagul*) catches terms not yet indexed or indexed differently. A robust search combines both.
A search that finds 5,000 records of which 35 are included is typical. A search that finds 200 records of which 35 are included is almost certainly not sensitive enough.
Documenting the search
The search must be fully documented and reproducible. PRISMA 2020 requires that the full search strategy for at least one database be reported, including all search terms, Boolean operators, field codes, limits and the date the search was run. The search strategies for all databases should be available (for example, as supplementary material).
An information specialist or librarian should ideally design or peer review the search strategy. The PRESS (Peer Review of Electronic Search Strategies) checklist provides a structured approach to search peer review.
Drills
("Anti-Bacterial Agents"[MeSH] OR antibiotic*[tiab] OR antimicrobial*[tiab] OR antibacterial*[tiab]) AND ("Diarrhea"[MeSH] OR diarrh*[tiab] OR diarh*[tiab] OR "loose stools"[tiab] OR "loose stool"[tiab]) AND ("Inpatients"[MeSH] OR hospitali*[tiab] OR "in-patient*"[tiab] OR inpatient*[tiab] OR "hospital setting"[tiab])Notes:
1. MeSH terms capture the controlled vocabulary; free-text terms (tiab = title/abstract) catch articles not yet indexed or using non-standard terminology.
2. Truncation (
diarrh*) captures both British and American spellings.3. This is the Population block. It would be combined with AND to an Intervention block (probiotics) and, if needed, a Study design filter for RCTs.
4. The comparator (placebo/no probiotic) and outcome (incidence of diarrhoea) are typically not searched as separate blocks, because adding them reduces sensitivity without improving precision at the search stage — they are applied at the screening stage instead.
Screening and selection
Title/abstract screening, full-text review, the PRISMA flow diagram and inter-rater reliability
The screening process
Screening is conducted in two stages: title/abstract screening (to exclude obviously irrelevant records) and full-text screening (to apply the full eligibility criteria). Both stages should be conducted independently by at least two reviewers. Disagreements are resolved by discussion, or by a third reviewer if consensus is not reached.
At the title/abstract stage, the default is to include. If there is any doubt, the record goes forward to full-text review. Exclusion at this stage should only be for records that are clearly outside the scope — wrong population, wrong study type, not a research study. A reason for exclusion is not required at this stage.
At the full-text stage, every excluded record must have a documented reason for exclusion. PRISMA 2020 requires that the reasons for exclusion at full-text stage be reported, typically in the flow diagram. Common reasons include: wrong population, wrong intervention, wrong comparator, wrong outcome, wrong study design, duplicate report of an included study.
The PRISMA flow diagram
The PRISMA flow diagram is a visual summary of the screening process. It shows the number of records identified from each source, the number screened, the number assessed for eligibility, the number included, and the reasons for exclusion at each stage.
The 2020 update of PRISMA introduced a revised flow diagram that distinguishes between records identified from databases and registers versus other sources (citation searching, grey literature, expert consultation). It also tracks the number of reports (publications) separately from the number of studies, because one study may produce multiple reports.
Inter-rater reliability
Agreement between reviewers at screening is measured using Cohen’s kappa (κ). A kappa of 0.6–0.8 is considered substantial agreement; above 0.8 is almost perfect. Low agreement at title/abstract stage suggests the eligibility criteria are ambiguous and need clarification before proceeding.
Disagreements at full-text stage should be documented, including which reviewer included and which excluded and the reason. This is the audit trail for the screening process.
Drills
Improved exclusion reasons:
• Wrong population (n = 23) — e.g., paediatric, or condition other than non-valvular AF
• Wrong intervention (n = 18) — e.g., antiplatelet therapy without anticoagulant, or heparin
• Wrong comparator (n = 12) — e.g., DOAC vs DOAC (not vs warfarin)
• Wrong outcome (n = 9) — e.g., pharmacokinetic outcomes only, no clinical endpoints
• Wrong study design (n = 22) — e.g., case reports, narrative reviews, editorials
• Duplicate report of included study (n = 4) — e.g., conference abstract of included RCT
• Full text not available (n = 4) — contacted authors, no response
What changed:
1. "Not relevant" is not a reason — it is a conclusion. The reason must say why the study is not relevant (wrong population, wrong intervention, etc.).
2. "Other" conceals the actual reasons and should be broken down.
3. Each reason maps directly to an eligibility criterion, making the exclusions auditable.
Risk of bias assessment
RoB 2 for randomised trials, ROBINS-I for observational studies, and the domain writers get wrong most often
Why risk of bias matters
Not all studies are created equal. Two RCTs on the same question may produce different results not because the treatment works differently, but because one trial was designed in a way that systematically favours a particular result. Risk of bias assessment identifies these design and conduct features so that the review can interpret the evidence accordingly.
Risk of bias is not the same as study quality. Quality is a broad concept that includes reporting, statistical power and clinical relevance. Risk of bias is narrower and more operational: it asks whether specific features of the study design or conduct could have distorted the result in a particular direction.
RoB 2 for randomised trials
The Revised Cochrane Risk of Bias tool (RoB 2) assesses bias across five domains, each corresponding to a stage of the trial where bias can be introduced:
| Domain | What can go wrong | What the reviewer looks for |
|---|---|---|
| Randomisation process | Non-random allocation; baseline imbalances | Method of sequence generation, allocation concealment, baseline comparisons |
| Deviations from intended interventions | Unblinding, crossover, co-interventions | Blinding of participants and personnel, deviations from protocol, intention-to-treat analysis |
| Missing outcome data | Differential attrition, informative censoring | Completeness of outcome data, reasons for missingness, handling of missing data |
| Measurement of the outcome | Assessor aware of allocation; subjective endpoints | Blinding of outcome assessment, objectivity of measurement, consistency of methods |
| Selection of the reported result | Multiple outcomes measured, only favourable ones reported | Protocol comparison, multiple outcome definitions, selective reporting indicators |
ROBINS-I for non-randomised studies
ROBINS-I (Risk Of Bias In Non-randomised Studies of Interventions) assesses seven domains for observational studies: confounding, selection of participants, classification of interventions, deviations, missing data, measurement of outcomes, and selection of the reported result. It benchmarks the study against a hypothetical, perfectly designed RCT — asking how the result might differ from the RCT result due to each domain.
The distinction matters for medical writers: a systematic review that includes both RCTs and observational studies must use both tools and present the results of each tool separately.
Drills
Randomisation process: Low risk. Computer-generated sequence with central allocation ensures adequate generation and concealment.
Deviations from intended interventions: Low risk. Double-blind design minimises the risk of unblinding-driven co-interventions or crossover.
Missing outcome data: High risk. Differential attrition (22% vs 8%) suggests informative censoring — participants in the active arm may have dropped out due to adverse effects. Without reasons for dropout, the direction of bias cannot be assessed but is likely to favour the active treatment (if those who fared worst left the analysis).
Measurement of the outcome: Low risk. All-cause mortality is an objective outcome that cannot be influenced by knowledge of allocation.
Selection of the reported result: Low risk. The protocol is available and all pre-specified outcomes are reported.
Overall: High risk of bias, driven by the missing outcome data domain. The differential attrition is the dominant concern.
Critical appraisal for diagnostic & observational studies
Evaluating diagnostic accuracy (QUADAS-2) and non-randomised observational cohorts (Newcastle-Ottawa Scale) with methodological precision
1. Why study design dictates appraisal architecture
A fatal methodological error in systematic reviews is applying an appraisal tool designed for randomised controlled trials (such as Cochrane RoB 2) to diagnostic test accuracy (DTA) or observational cohort studies. Each study design possesses unique structural vulnerabilities to bias:
- Diagnostic Accuracy Studies: Evaluate test performance (sensitivity, specificity, PPV, NPV, ROC/AUC) against a reference standard. Biases arise from spectrum effects (patient selection), test interpretation bias (lack of blinding between index test and reference standard), and differential verification (applying different reference standards based on index test results).
- Observational Cohorts & Case-Control Studies: Evaluate exposures and long-term outcomes without random allocation. Biases arise from confounding by indication, selection bias, exposure misclassification, and differential loss to follow-up.
Always pre-specify your appraisal instruments in the PROSPERO protocol: RoB 2 for RCTs, ROBINS-I for non-randomised interventional studies, QUADAS-2 for diagnostic accuracy studies, and Newcastle-Ottawa Scale (NOS) or JBI tools for observational cohorts/case-controls.
2. QUADAS-2: Quality Assessment of Diagnostic Accuracy Studies
QUADAS-2 is the international gold standard endorsed by Cochrane and PRISMA for appraising diagnostic accuracy studies. It evaluates four fundamental domains across two separate dimensions: Risk of Bias and Applicability Concerns.
| Domain | Key Signaling Questions | High Risk Scenario | Applicability Concerns |
|---|---|---|---|
| 1. Patient Selection | Was a consecutive or random sample enrolled? Was a case-control design avoided? Were inappropriate exclusions avoided? | Two-gate design enrolling known severe cases vs healthy young controls (inflates sensitivity and specificity). | Are enrolled patients representative of the clinical target population receiving the test in real practice? |
| 2. Index Test | Were index test results interpreted without knowledge of the reference standard? If a threshold was used, was it pre-specified? | Pathologist interprets liquid biopsy NGS results while knowing the confirmed tissue biopsy outcome. Post-hoc threshold optimization (ROC curve hunting). | Does the index test, its execution, or its interpretation differ from the review question? |
| 3. Reference Standard | Is the reference standard likely to correctly classify the condition? Were reference results interpreted without knowledge of the index test? | Imperfection in reference standard; unblinded histopathology review. | Does the target condition defined by the reference standard match the review question? |
| 4. Flow & Timing | Was there an appropriate interval between index test and reference standard? Did all patients receive the same reference standard? Were all patients included in the analysis? | Disease progression during 3-month gap between blood draw and tissue biopsy. Partial verification (only positive blood tests get biopsied). High uninterpretable rate excluded. | (Domain 4 evaluates Risk of Bias only; no applicability rating). |
3. Newcastle-Ottawa Scale (NOS) for Cohort & Case-Control Studies
The Newcastle-Ottawa Scale (NOS) uses a semi-quantitative "star system" (maximum 9 stars) to evaluate three broad perspectives of study quality in non-randomised observational studies:
- Representativeness of exposed cohort: Truly or somewhat representative of community/clinic (★).
- Selection of non-exposed cohort: Drawn from the same community/source (★).
- Ascertainment of exposure: Secure medical records or structured clinical interview (★).
- Demonstration that outcome was not present at start: Clear baseline verification (★).
- Control for main confounder: Study controls for primary pre-specified confounder (e.g., age or baseline disease stage) (★).
- Control for additional factor: Study controls for secondary critical confounders (e.g., prior lines of therapy, smoking status) (★).
- Assessment of outcome: Independent blind assessment or record linkage (★).
- Follow-up length: Long enough for outcomes to manifest (e.g., ≥12 months for PFS) (★).
- Adequacy of follow-up: Complete follow-up or <10% attrition with accounted loss (★).
Domain 1 (Patient Selection) — High Risk of Bias & High Applicability Concern: The study used a two-gate (case-control) design enrolling extreme phenotypes (severe ICU chest pain vs healthy young controls) rather than a consecutive spectrum of undifferentiated emergency department chest pain patients. This artificially inflates both sensitivity and specificity.
Domain 2 (Index Test) — High Risk of Bias: The index test operator was unblinded to the reference standard (final hospital discharge diagnosis), introducing severe diagnostic review bias.
Overall Assessment: The study exhibits high risk of bias across multiple domains and cannot provide trustworthy diagnostic accuracy metrics for emergency clinical decision-making.
Data extraction
Extraction forms, handling missing data, contacting authors, and why double extraction is not redundancy
What to extract
Data extraction captures the information needed to describe each study, assess its quality, and include it in the synthesis. A standardised extraction form ensures consistency across reviewers and across studies.
Standard fields include: study identification (first author, year, journal, country), study design, population characteristics (sample size, age, sex, disease severity, setting), intervention and comparator details (dose, duration, delivery), outcome definitions, outcome data (effect estimates, confidence intervals, p-values, event counts, sample sizes per arm), and risk of bias judgements.
Handling missing data
Missing data are the rule, not the exception. The most common situations and their solutions:
| Problem | Solution |
|---|---|
| Mean reported without SD | Calculate SD from SE, CI, p-value, or t-statistic if available. If not, impute from other studies in the review (using the median SD of similar studies). |
| Median and IQR reported instead of mean and SD | Estimate mean and SD using established formulae (Wan et al., 2014; Luo et al., 2018). Note the estimation in the data extraction form. |
| Only p-value reported, no effect estimate | Contact the authors. If no response, back-calculate an approximate effect size from the p-value and sample size, or exclude from quantitative synthesis with documentation. |
| Aggregate data only (no per-arm data) | Use the aggregate data directly if the analysis permits it (e.g., mean difference with SE). Otherwise, contact authors. |
| Multiple time points reported | Extract the time point specified in the protocol. If not specified, extract all and select during analysis with justification. |
Contacting authors
When essential data are missing from the publication, contacting the corresponding author is standard practice. The contact should be specific: state the review, cite the paper, and list exactly what data are needed. Allow two to four weeks for a response, with one follow-up. Document all contact attempts regardless of response, including dates, the request made, and whether data were received.
If data are obtained from the authors, this must be noted in the review — the data are not from the published record and cannot be verified by the reader against the publication.
Drills
1. Estimate the mean from the median and IQR using the method of Luo et al. (2018) or Wan et al. (2014). These methods use the median, quartiles and sample size to estimate the mean and SD. The formula accounts for skewness in the distribution.
2. Estimate the SD similarly. Wan et al. provide separate formulae for estimating SD from the IQR and sample size.
3. Document the estimation in the data extraction form, noting that the original data were reported as median and IQR and the mean and SD are estimates.
4. Conduct a sensitivity analysis excluding this study to assess whether the estimated values change the pooled result. If the study contributes substantially to the pooled estimate, the sensitivity of the result to the estimation should be reported.
5. Consider whether the skewed distribution (suggested by the asymmetric IQR) indicates that the mean difference is not the right summary measure for this outcome, and whether ratio of medians or a different approach would be more appropriate.
AI & software automation in evidence synthesis
Leveraging Covidence, Rayyan, DistillerSR, machine learning priority screening and LLMs under rigorous PRISMA 2020 validation standards
1. The Evidence Synthesis Tooling Ecosystem
Modern systematic literature reviews require dedicated evidence synthesis platforms designed for blinding, audit logging, and conflict resolution. Using generic spreadsheets (Excel) for large multi-reviewer screenings leads to catastrophic version-control failures, unblinded bias, and irreproducible selection.
| Platform | Core Strengths | AI / Automation Features | Best Use Case |
|---|---|---|---|
| Covidence | Seamless Cochrane integration, built-in PRISMA flow chart generator, standardized RoB 2 templates, dual-blind conflict resolution UI. | Machine learning deduplication, automatic keyword highlighting (inclusion green / exclusion red). | Standard academic & Cochrane intervention systematic reviews. |
| Rayyan QCRI | Ultra-fast mobile/web screening, free tier for independent researchers, robust tagging and blinding toggles. | 5-star prediction model: active learning algorithm ranks unscreened abstracts by relevance based on early reviewer decisions. | High-volume title/abstract screening on tight timelines. |
| DistillerSR | Enterprise-grade compliance (21 CFR Part 11), fully customizable modular forms, automated audit trails. | AI-powered screening (DAIS), automated data extraction verification, real-time re-ranking. | Pharma Global Value Dossiers (GVD), HTA submissions, regulatory SLRs. |
| EPPI-Reviewer | Advanced qualitative and mixed-methods synthesis capabilities, complex multi-arm coding hierarchies. | Text mining, priority screening, automated RCT classifier (Cochrane Machine Learning Model). | Complex public health, scoping, and mixed-method policy reviews. |
2. Machine Learning Priority Screening & Active Learning
In traditional screening, reviewers screen records in arbitrary chronological or alphabetical order. With Active Learning (Priority Screening), a machine learning model (e.g., Support Vector Machine or Transformer) continuously trains on the titles and abstracts that human reviewers have already coded as Include or Exclude.
- Human reviewers dual-screen an initial random training set (e.g., 500 records) to establish baseline inclusion criteria.
- The active learning engine builds a semantic vector representation of included vs excluded citations.
- The remaining unscreened citations are continuously re-ordered, presenting the highest-probability candidates first.
- Once 500–1,000 consecutive citations yield zero inclusions (the "stopping heuristic"), reviewers can deploy targeted verification protocols for the remaining tail.
3. Large Language Models (LLMs) in Data Extraction: Governance & Validation
While LLMs (such as GPT-4, Claude, or Gemini) can rapidly parse unstructured trial text, zero-shot unvalidated extraction produces hallucinations, numeric transcription errors, and missing denominator errors. Medical writers must enforce a Human-in-the-Loop (HITL) validation protocol:
Never ask open-ended questions. Provide JSON schemas, explicit PICO field definitions, and demand exact verbatim text quotes with page/table coordinate citations for every extracted numeric value.
Treat the LLM as Reviewer 1 (first-pass extractor). An experienced human medical writer serves as Reviewer 2, auditing all extracted values against the original publication figures and supplementary appendices.
Under PRISMA 2020 Item 5 and Item 8, explicitly disclose all software, AI tools, machine learning classifiers, prompt templates, and human validation workflows used in screening and extraction.
Deficiencies:
1. Fails to name the specific software, algorithmic model, version, or platform (e.g., Covidence, Rayyan, LLM API).
2. Fails to specify the training/prompting protocol, inclusion threshold, or stopping heuristic.
3. Fails to state human reviewer involvement, dual-checking protocols, conflict resolution mechanisms, and validation error rates.
Compliant PRISMA 2020 Rewrite:
"Title and abstract screening was facilitated using Rayyan QCRI. Following an initial calibration set of 500 citations screened independently in duplicate by two human reviewers (MW, SP), Rayyan's active learning prioritization algorithm was enabled to rank unscreened records by probability of inclusion. All potentially eligible records identified by the algorithm were subsequently verified by two independent reviewers, with disagreements resolved by consensus or third-party adjudication (AK). Data extraction was performed using a standardized electronic form with automated first-pass assistance; all numerical data points, hazard ratios, and 95% confidence intervals were 100% independently cross-checked against source publication tables and supplementary appendices by a senior medical writer."
Meta-analysis fundamentals
Effect measures, fixed-effect and random-effects models, forest plots, heterogeneity and prediction intervals
Effect measures
The choice of effect measure depends on the type of outcome. For binary outcomes (event/no event), the standard measures are the odds ratio (OR), risk ratio (RR), and risk difference (RD). For continuous outcomes (measured on a scale), the measures are the mean difference (MD) (when all studies use the same scale) and the standardised mean difference (SMD) (when studies use different scales to measure the same construct).
| Measure | When to use | Interpretation |
|---|---|---|
| Risk ratio (RR) | Binary outcomes; most common in clinical reviews | RR = 0.75 means 25% relative risk reduction |
| Odds ratio (OR) | Binary outcomes; case-control studies; logistic regression | Approximates RR when event rate is low (<10%) |
| Risk difference (RD) | Binary outcomes; communicating absolute effect | RD = -0.05 means 5 fewer events per 100 people |
| Mean difference (MD) | Continuous; same scale across studies | Difference in means in original units |
| Standardised mean difference (SMD) | Continuous; different scales measuring same construct | Effect in SD units (0.2 small, 0.5 medium, 0.8 large) |
Fixed-effect vs random-effects models
A fixed-effect model assumes that all studies estimate the same underlying effect, and the differences between study results are due to chance alone. The pooled estimate is a weighted average of the study effects, with weights proportional to the inverse of each study’s variance (larger, more precise studies contribute more).
A random-effects model assumes that the true effect varies between studies (because populations, interventions and settings differ) and that the observed effects are drawn from a distribution of true effects. The pooled estimate is the mean of this distribution. Weights are more equal across studies because the between-study variance (τ²) is added to each study’s within-study variance.
In practice, the choice matters most when heterogeneity is present. With low heterogeneity, the models give similar results. With high heterogeneity, the random-effects model gives wider confidence intervals (appropriately reflecting the uncertainty) and gives more weight to smaller studies (which may or may not be desirable).
The forest plot
A forest plot is the standard visual display of a meta-analysis. Each study is shown as a square (whose size reflects the study’s weight) with a horizontal line (the confidence interval). The pooled estimate is shown as a diamond at the bottom. A vertical line at 1 (for ratio measures) or 0 (for difference measures) represents no effect.
Reading a forest plot: if all confidence intervals cross the line of no effect, no individual study shows a statistically significant result — but the pooled estimate may still be significant because pooling increases precision. If the confidence intervals barely overlap, heterogeneity is likely high.
Heterogeneity
Heterogeneity is variation between study results beyond what would be expected by chance. It is quantified by three statistics:
Drills
Different clinical outcomes demand distinct pooled effect size metrics. Master the interpretation of pooled statistical models across major therapeutic areas:
| Therapeutic Domain | Outcome Type & Metric | Heterogeneity (I^2) & Model Choice | Pooled Result Phrasing (Publication Standard) |
|---|---|---|---|
| Cardiology (SGLT2i in Heart Failure) | Dichotomous (CV Death or HHF): Relative Risk (RR) / Hazard Ratio (HR) | I^2 = 14\% (Low): Fixed-Effects / Inverse-Variance Model | "Across 5 trials (N = 21,947), SGLT2 inhibitors significantly reduced the risk of CV death or hospitalisation for heart failure by 23% (pooled HR 0.77; 95% CI, 0.72 to 0.82; P < 0.0001; I^2 = 14\%)." |
| Pulmonology (Severe Asthma Biologics) | Count Data (Annualized Exacerbation Rate): Rate Ratio (RR) | I^2 = 58\% (Moderate): DerSimonian-Laird Random-Effects Model | "Random-effects meta-analysis of 8 RCTs (N = 4,120) demonstrated a 48% reduction in severe exacerbation rate (pooled RR 0.52; 95% CI, 0.43 to 0.63; P < 0.001; I^2 = 58\%, au^2 = 0.042)." |
| Neurology (Alzheimer's Disease) | Continuous Scale (ADAS-Cog Change from Baseline): Mean Difference (MD) | I^2 = 68\% (Substantial): Random-Effects with Subgroup Stratification | "Cholinesterase inhibitors produced a modest improvement in cognitive performance (pooled MD -1.87 points; 95% CI, -2.44 to -1.30; P < 0.0001; I^2 = 68\%). Subgroup analysis by baseline MMSE explained 74% of between-trial variance." |
“Twelve randomised controlled trials (n = 71,683) compared DOACs with dose-adjusted warfarin for the prevention of stroke or systemic embolism in patients with non-valvular atrial fibrillation. The pooled risk ratio was 0.76 (95% CI 0.65 to 0.89; p = 0.001), indicating a 24% relative reduction in stroke or systemic embolism with DOACs compared with warfarin. Heterogeneity was moderate (I² = 42%; Cochran Q p = 0.06), suggesting some variation between study effects beyond chance. The 95% prediction interval (0.51 to 1.13) indicates that while the average effect favours DOACs, a future study in a new setting could plausibly find no difference or even a slight increase in risk — reflecting the range of populations, DOAC agents and warfarin control quality across the included trials.”
Key points: The prediction interval is more informative than the confidence interval for clinical interpretation — it tells you the range of effects a clinician might see in practice, not just the precision of the average. The difference between the CI (excludes 1) and the prediction interval (crosses 1) is the difference between the mean effect and the range of true effects.
Subgroup analysis and publication bias
Pre-specified vs post-hoc subgroups, interaction tests, sensitivity analysis, funnel plots and Egger's test
Pre-specified vs post-hoc subgroups
A subgroup analysis asks whether the treatment effect differs between subsets of studies or participants. Pre-specified subgroup analyses — defined in the protocol before the results are known — are credible because the hypothesis preceded the data. Post-hoc subgroup analyses — suggested by the data — are exploratory and should be labelled as such.
Common pre-specified subgroups include: study design (RCTs vs observational), risk of bias (low vs high), intervention characteristics (dose, duration, specific agent), population characteristics (age, sex, disease severity, region), and outcome definition.
Sensitivity analysis
Sensitivity analysis tests whether the pooled result is robust to analytical decisions. Common sensitivity analyses include:
| Sensitivity analysis | What it tests |
|---|---|
| Leave-one-out | Recalculates the pooled estimate removing each study in turn. Identifies studies that drive the result disproportionately. |
| Fixed-effect vs random-effects | Whether the conclusion changes with the modelling assumption. |
| Excluding high risk-of-bias studies | Whether the result survives when only low risk-of-bias studies are pooled. |
| Excluding imputed data | Whether imputed SDs or converted medians change the result. |
| Excluding outliers | Whether an extreme result drives the pooled estimate. |
Publication bias
Publication bias occurs when studies with positive or significant results are more likely to be published than those with null or negative results. It inflates the pooled effect in a meta-analysis by selectively including favourable evidence.
The funnel plot is the primary visual tool for assessing publication bias. It plots each study’s effect estimate against its precision (usually standard error). In the absence of bias, the plot should be symmetrical — small studies scatter widely around the pooled estimate, and large studies cluster narrowly. Asymmetry suggests that small studies with unfavourable results are missing.
Egger’s test provides a statistical test for funnel plot asymmetry. A significant Egger’s test (p < 0.10, by convention) suggests small-study effects, which may be due to publication bias — but also to genuine differences between small and large trials.
Drills
“Visual inspection of the funnel plot suggested possible asymmetry, with an absence of small studies showing no benefit or harm. Egger's test for small-study effects gave p = 0.08, which approaches but does not reach conventional significance. These findings are consistent with possible publication bias, although the small number of studies (n = 12) limits the reliability of both the visual assessment and the statistical test. The review did not search for unpublished studies beyond trial registry records, which may have contributed to the asymmetry. The pooled estimate should be interpreted with caution, as publication bias, if present, would be expected to inflate the apparent treatment effect.”
Key points:
1. The language is appropriately cautious — "consistent with possible," not "proves."
2. The limitation of the search (no contact with authors for unpublished data) is acknowledged as a contributor.
3. The direction of potential bias is stated — inflation, not deflation.
4. The funnel plot's reliability with 12 studies is noted (Cochrane recommends at least 10 for funnel plot assessment).
Small-study effects, funnel plots & publication bias
Detecting reporting bias, mastering funnel plot interpretation, Egger's regression, Begg's test and trim-and-fill analysis
1. The Spectrum of Reporting Biases
Publication bias is only one component of a broader phenomenon known as dissemination bias or reporting bias. Statistically significant, positive, or sensational trial results are systematically favored throughout the scientific dissemination lifecycle:
Studies with positive results are published significantly faster (time-lag bias) and are much more likely to be submitted and accepted by high-impact peer-reviewed journals than null/negative studies.
Positive trials are preferentially published in English (language bias) and are cited at 2–3 times higher rates by subsequent authors (citation bias), making them far easier to retrieve in routine searches.
Within a published study, non-significant primary or secondary endpoints are omitted, buried in appendices, or converted into post-hoc positive composite outcomes.
2. Anatomy and Interpretation of Funnel Plots
A funnel plot is a scatter plot of study effect estimates (x-axis) against a measure of study precision or sample size (y-axis, typically standard error SE( heta) plotted inverted, with zero at the top).
- Top of the Funnel (Large Studies): Low standard error, narrow confidence intervals, tight clustering around the true population effect estimate.
- Bottom of the Funnel (Small Studies): High standard error, wide sampling variation, spreading symmetrically across both sides of the vertical summary line within the triangular 95% pseudo-confidence boundaries.
- Asymmetry Indicator: An empty or hollow quadrant (e.g., absent small trials showing lack of efficacy) indicates small-study effects, of which publication bias is a primary cause.
3. Statistical Tests for Funnel Plot Asymmetry
| Test Method | Mechanism & Formula | Clinical Interpretation & Limitations |
|---|---|---|
| Egger's Linear Regression | Regresses the standardized effect estimate (Z_i = heta_i / SE_i) against precision (1 / SE_i):Z_i = α + β × (1 / SE_i) |
Tests if intercept lpha = 0. If lpha eq 0 (p < 0.10), significant asymmetry exists. Most powerful test for continuous outcomes; prone to false-positive inflation in binary outcomes with rare events. |
| Begg & Mazumdar Rank Correlation | Calculates Kendall's tau ( au) between standardized effect sizes and sampling variances. | Non-parametric test; lower statistical power than Egger's test, but robust to extreme outliers. |
| Peters / Harbord Tests | Modified regression tests tailored specifically for binary outcomes (odds ratios / risk ratios). | Recommended by Cochrane when dealing with binary outcomes to eliminate artificial artifactual asymmetry caused by intrinsic correlation between OR and its variance. |
4. Duval & Tweedie's Trim-and-Fill Method
The Trim-and-Fill method is a non-parametric data augmentation technique that estimates the number of missing studies needed to restore funnel symmetry and recalculates the adjusted pooled effect estimate:
- Trim: Iteratively identifies and removes ("trims") asymmetric outlying small studies from the positive side to re-center the funnel around the true symmetrical core.
- Fill: Imputes mirror-image "fictional" studies on the opposite quadrant to balance the trimmed studies.
- Re-estimate: Recalculates the pooled effect estimate including the imputed studies to establish a conservative lower bound of efficacy.
"Although the initial pooled meta-analysis demonstrated a statistically significant increase in ACR50 response (RR 1.85, 95% CI: 1.45–2.36), visual inspection of the funnel plot and Egger's regression test (p = 0.008) indicated substantial small-study effects and potential publication bias. Sensitivity analysis using the Duval and Tweedie trim-and-fill method imputed five missing negative studies, attenuating the pooled treatment effect to RR 1.28 (95% CI: 0.98–1.67, p = 0.07). Consequently, the clinical benefit of this biologic must be interpreted with caution, as current published evidence likely overestimates true therapeutic efficacy due to selective dissemination of positive trials."
Advanced evidence synthesis: Network meta-analysis & SUCRA
Mastering indirect treatment comparisons (ITC), Bucher method, network geometry, transitivity, consistency and SUCRA rankings
1. From Pairwise Meta-Analysis to Network Meta-Analysis
Traditional pairwise meta-analysis can only synthesize evidence from head-to-head trials evaluating the exact same two interventions (Drug A vs Drug B). When multiple competing interventions exist within a disease space, clinical guidelines and HTA bodies require comparative efficacy and safety estimates across all available options simultaneously.
When Drug A is compared to Placebo (Drug A vs Drug C) and Drug B is compared to Placebo (Drug B vs Drug C), the indirect effect of A ext{ vs } B is calculated as:ln(RR_AB) = ln(RR_AC) - ln(RR_BC)
with pooled variance: Var(ln RR_AB) = Var(ln RR_AC) + Var(ln RR_BC).
Synthesizes direct evidence (direct head-to-head trials) and indirect evidence (via common comparators C, D, E) across an entire network of interventions simultaneously using Bayesian (Markov Chain Monte Carlo) or Frequentist graph models.
2. Network Geometry & Visualizing Evidence Networks
In a Network Plot:
- Nodes (Circles): Represent competing interventions. Node size is proportional to the total number of patients randomised to that treatment.
- Edges (Lines): Represent direct head-to-head RCT comparisons. Line thickness is proportional to the number of trials or precision of the direct comparison.
- Closed Loops: Formed when both direct and indirect paths connect treatments (e.g., triangle A-B-C). Closed loops allow formal statistical testing of consistency.
3. Core Methodological Assumptions: Transitivity & Consistency
An NMA is scientifically valid only if two foundational epidemiological assumptions hold true:
| Assumption | Epidemiological Definition | How to Verify & Test |
|---|---|---|
| 1. Transitivity (Clinical Similarity) | The common comparator (e.g., Placebo or Standard of Care) must be clinically and methodologically equivalent across all trial sets, and effect modifiers (e.g., age, disease severity, prior lines) must be balanced across comparisons. | Qualitative clinical cross-table auditing of trial baseline characteristics (patient demographics, dosing regimens, outcome definitions, trial eras). Cannot be tested purely with statistics. |
| 2. Consistency (Coherence) | Statistical agreement between direct evidence and indirect evidence within closed loops (Effect_{Direct} pprox Effect_{Indirect}). | Statistical testing using the Node-Splitting Method (separates direct from indirect evidence for each loop) and the global Design-by-Treatment Interaction Model (p > 0.05 indicates acceptable consistency). |
4. Ranking Interventions: SUCRA and Rankograms
Network meta-analysis enables probability-based hierarchy ranking across all evaluated treatments:
- Rankograms: Bar charts displaying the probability (0% to 100%) that each drug is Rank 1 (best), Rank 2, Rank 3, etc.
- SUCRA (Surface Under the Cumulative Ranking Curve): A single numerical metric between 0\% (certain to be the worst) and 100\% (certain to be the best):
SUCRA = [Σ CumProb(j)] / (T − 1)where T is the total number of treatments and CumProb(j) is the cumulative probability of being among the top j treatments.
Transitivity Violation: The assumption of transitivity is severely violated because the two trial sets have substantial clinical and temporal heterogeneity in fundamental effect modifiers (disease stage, prior treatment history, baseline performance status, and supportive care standards across 2005 vs 2022). Placebo in 2005 treatment-naïve patients is not biologically or prognostically comparable to Placebo in 2022 multi-refractory patients.
Consequence: The calculated indirect effect estimate of Drug A vs Drug B will be confounded by patient baseline prognosis rather than reflecting true pharmacodynamic superiority. Drug A would appear artificially superior simply because it was tested in healthier patients with longer expected survival.
GRADE and certainty of evidence
The GRADE framework, rating certainty, Summary of Findings tables and the language that matches each level
What GRADE is
The Grading of Recommendations, Assessment, Development and Evaluations (GRADE) framework provides a systematic approach to rating the certainty of evidence for each outcome in a systematic review. It moves beyond the pooled estimate to ask: how much should we trust this number?
GRADE starts with an initial rating based on study design: randomised trials start at High, observational studies at Low. The rating is then adjusted — downgraded for limitations, or upgraded for strengths.
Downgrading and upgrading
| Direction | Domain | What to look for |
|---|---|---|
| Downgrade (−1 or −2 each) | Risk of bias | High risk-of-bias studies contribute substantially to the pooled estimate |
| Inconsistency | Unexplained heterogeneity (high I², wide prediction interval, conflicting directions) | |
| Indirectness | The evidence does not directly answer the question (different population, surrogate outcome, indirect comparison) | |
| Imprecision | Wide CI that crosses the threshold of clinical significance; or too few events | |
| Publication bias | Funnel plot asymmetry; missing studies suspected | |
| Upgrade (+1 or +2, rare) | Large effect | RR < 0.5 or > 2 without plausible confounders |
| Dose-response | Clear gradient — higher dose, larger effect | |
| Residual confounding | All plausible confounders would reduce the observed effect (strengthening the conclusion) |
The four certainty levels
High: We are very confident that the true effect lies close to the estimate. Further research is very unlikely to change our confidence. Moderate: We are moderately confident. The true effect is likely close to the estimate, but there is a possibility that it is substantially different. Low: Our confidence is limited. The true effect may be substantially different. Very Low: We have very little confidence. The true effect is likely substantially different from the estimate.
High: “X reduces Y” (unqualified statement)
Moderate: “X probably reduces Y”
Low: “X may reduce Y”
Very Low: “We are very uncertain whether X reduces Y”
Medical writers must match the language to the rating. Writing “X reduces Y” when the GRADE rating is Low is a misrepresentation of the evidence, even if the pooled estimate is statistically significant.
Summary of Findings tables
The Summary of Findings (SoF) table is the standard format for presenting GRADE assessments. It shows, for each outcome: the number of studies and participants, the effect estimate with CI, the GRADE rating with footnotes explaining each downgrade, and an interpretation in absolute terms (e.g., “36 fewer per 1,000”).
SoF tables are required for Cochrane reviews and increasingly expected by journals publishing non-Cochrane systematic reviews. The GRADEpro tool generates standardised SoF tables.
Drills
Under GRADE (Grading of Recommendations Assessment, Development and Evaluation), RCT evidence starts at High Quality and observational evidence starts at Low Quality. Systematically apply the 5 downgrading and 3 upgrading criteria:
| GRADE Dimension | Trigger for Rating Modification | Impact on Certainty Level |
|---|---|---|
| 1. Risk of Bias (RoB) | Inadequate allocation concealment, lack of blinding with subjective outcome, or high differential attrition (> 20%). | Downgrade 1 or 2 levels (Serious / Very Serious). |
| 2. Inconsistency | Substantial unexplained heterogeneity (I^2 > 50\%, non-overlapping confidence intervals across major trials). | Downgrade 1 or 2 levels. |
| 3. Indirectness | Surrogate biomarker used instead of hard clinical endpoint, or trial population younger/healthier than real-world target. | Downgrade 1 or 2 levels. |
| 4. Imprecision | Total sample size falls below Optimal Information Size (OIS) or 95% CI spans both significant benefit and clinical harm. | Downgrade 1 or 2 levels. |
| 5. Publication Bias | Asymmetrical funnel plot, Egger's test p < 0.05, or evidence of withheld small negative industry trials. | Downgrade 1 level. |
| Upgrading (Observational Data Only) | Large magnitude of effect (RR > 2 or RR < 0.5 without confounders), dose-response gradient, or residual confounding would have reduced effect. | Upgrade 1 or 2 levels. |
GRADE rating: Moderate (downgraded 1 level from High)
Footnotes:
1. Risk of bias: downgraded by 1. Two of eight studies were at high risk of bias (one for differential attrition, one for outcome switching) and contributed 35% of the pooled weight. Sensitivity analysis excluding them gave RR 0.85 (95% CI 0.72–1.00), suggesting the pooled estimate may be inflated by these studies.
2. Inconsistency: not downgraded. I² = 42% indicates moderate heterogeneity. The prediction interval (0.51–1.13) crosses 1, but the heterogeneity is partially explained by DOAC agent in subgroup analysis, and the direction of effect is consistent across studies.
3. Indirectness: not downgraded. The population, intervention, comparator and outcome match the review question directly.
4. Imprecision: not downgraded. The 95% CI for the pooled estimate (0.65–0.89) does not cross 1 and excludes a trivial effect. The total number of events exceeds the optimal information size.
5. Publication bias: not downgraded. Funnel plot is symmetric; Egger’s test non-significant (p = 0.42).
Interpretation: DOACs probably reduce stroke compared with warfarin (Moderate certainty).
PRISMA 2020
The 27-item checklist, the abstract checklist, the flow diagram and the items writers get wrong
PRISMA 2020
PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is the current reporting guideline for systematic reviews. It replaced PRISMA 2009 with expanded and restructured items that reflect advances in methodology and the broader scope of modern reviews.
PRISMA 2020 has 27 items covering the title, abstract, introduction, methods, results, discussion and funding. Each item specifies what must be reported, and the PRISMA 2020 Explanation and Elaboration document gives examples of good and poor reporting for each item.
For medical writers, PRISMA is not a set of aspirational guidelines — it is the minimum reporting standard. Most journals that publish systematic reviews endorse PRISMA and require a completed checklist at submission.
Key items writers get wrong
| Item | What is required | Common failure |
|---|---|---|
| 6 — Eligibility criteria | Specify all criteria with rationale, including PICO, study design, date and language restrictions | Criteria listed without rationale; restrictions not justified |
| 7 — Information sources | All databases, registers, websites, organisations, reference lists searched, with dates | Only databases named; grey literature sources omitted; dates missing |
| 8 — Search strategy | Full search for at least one database; all searches available | Search terms listed but not the full strategy; Boolean logic unclear |
| 13a — Synthesis methods | Effect measure, model, software, heterogeneity assessment, sensitivity/subgroup analyses | "Data were pooled using meta-analysis" with no further detail |
| 16 — Flow diagram | PRISMA 2020 flow diagram with reasons for exclusion at full-text stage | Old-format diagram; studies/reports not distinguished; reasons vague |
| 22 — Certainty of evidence | Methods used to assess certainty (e.g., GRADE) for each outcome | GRADE mentioned but not applied; no Summary of Findings table |
Drills
“Risk ratios (RRs) with 95% confidence intervals were calculated for each study using the number of participants with events and the total number randomised in each arm. Studies were pooled using the DerSimonian-Laird random-effects model, selected a priori because clinical and methodological heterogeneity between studies was anticipated. Statistical heterogeneity was assessed using the I² statistic (interpreted as low [<25%], moderate [25–75%] or high [>75%]) and Cochran's Q test (significance threshold p < 0.10). The 95% prediction interval was calculated to estimate the range of true effects across settings. Pre-specified subgroup analyses examined the effect by DOAC agent, dose, and mean age (<65 vs ≥65 years), with interaction tests to assess whether effects differed between subgroups. Sensitivity analyses included leave-one-out analysis, restriction to studies at low risk of bias, and comparison of fixed-effect and random-effects models. Publication bias was assessed visually using funnel plots and statistically using Egger's regression test (for analyses with ≥10 studies). All analyses were conducted in R (version 4.3) using the meta package (version 6.5).”
What changed: Every analytical decision is specified — effect measure, model, heterogeneity statistics with interpretation thresholds, prediction interval, subgroup and sensitivity analyses, publication bias assessment method, and software. A reader can now reproduce the analysis.
Living systematic reviews & rapid reviews for HTA
Dynamic continuous evidence surveillance (LSR), accelerated synthesis methodologies, and HTA Global Value Dossier requirements
1. Living Systematic Reviews (LSR) Architecture
Traditional systematic reviews become out of date within 1–2 years of publication. A Living Systematic Review (LSR) is an evidence synthesis that is continually updated as new evidence becomes available, incorporating relevant new trial data as it emerges.
Automated monthly search queries across MEDLINE, Embase, Cochrane CENTRAL, and clinical trial registries (ClinicalTrials.gov, ICTRP) with auto-alerts routed directly into screening platforms.
Standardized extraction templates maintained in cloud-based repositories. Newly screened records are dual-extracted within 10 business days of publication.
Meta-analysis is dynamically re-run. If the new evidence alters the effect estimate, statistical significance, or GRADE certainty rating, a formal publication addendum or web update is released immediately.
2. Rapid Reviews vs Full Systematic Reviews
When policy makers, guideline panels, or pharma launch teams require urgent evidence syntheses in compressed timeframes (e.g., 2–8 weeks), a Rapid Review is conducted using rigorous, transparently justified methodological shortcuts:
| Review Stage | Standard Systematic Review | Rapid Review Streamlined Shortcut |
|---|---|---|
| Databases Searched | ≥ 3 major electronic databases + grey literature + trial registries + citation chasing. | 2 core databases (MEDLINE + Embase/CENTRAL); focused search strategies; limits on publication date (≤ 5–10 years) or English language. |
| Screening Protocol | 100% dual-independent screening with formal kappa calculation and adjudication. | Dual screening of initial 20% sample for calibration; single reviewer for remaining 80% with second-reviewer spot-checking of excluded records. |
| Data Extraction | 100% dual-independent extraction across all study variables. | Single extractor using standardized piloted form, with full verification by a second experienced reviewer. |
| Critical Appraisal | Full RoB 2 / ROBINS-I / QUADAS-2 appraisal by two independent raters. | Single rater appraisal with second-rater verification, or focusing on high-impact summary domains. |
3. Systematic Literature Reviews for HTA & Global Value Dossiers (GVD)
Systematic reviews designed for Health Technology Assessment (HTA) bodies (e.g., NICE in the UK, HAS in France, G-BA/IQWiG in Germany, CADTH in Canada, and the new EU-HTA Regulation) must comply with strict evidentiary mandates beyond standard journal publications:
- Complete Transparency: Search strategies must be provided verbatim with exact hit counts per line to permit 100% replication by HTA technical assessors.
- Comprehensive Adverse Event Matrices: Detailed extraction of Grade 3–5 treatment-emergent adverse events (TEAEs), serious adverse events (SAEs), and discontinuations due to toxicity across all treatment arms.
- Indirect Treatment Comparisons (ITC) & NMA Packages: Pre-specifying network geometry, baseline covariate distributions, and WinBUGS/R code to support economic Markov modeling and cost-effectiveness analyses.
"Omitting formal critical appraisal is unacceptable for regulatory and HTA dossiers, as agencies such as NICE, FDA, and EU-HTA will classify unappraised evidence syntheses as methodologically flawed and refuse to base clinical efficacy conclusions upon them. To meet the compressed 3-week deadline while maintaining methodological rigor, we recommend conducting an accelerated Rapid Review under Cochrane/WHO rapid synthesis standards, utilizing a single experienced appraiser with 100% verification by a second senior reviewer using validated RoB 2/ROBINS-I tools. This approach satisfies regulatory requirements for quality evaluation while reducing appraisal time by approximately 40%."
Capstone: from protocol to publication
An integrated exercise taking you through every stage of a systematic review, from PICO question to final conclusion
The end-to-end process
| Stage | Key output | Your role |
|---|---|---|
| 1. Question | PICO, eligibility criteria | Draft with clinical team; ensure specificity |
| 2. Protocol | PRISMA-P-compliant protocol; PROSPERO registration | Write and register |
| 3. Search | Documented search strategies for all databases | Collaborate with information specialist; document |
| 4. Screening | PRISMA flow diagram | May participate in screening; document process |
| 5. Extraction | Completed extraction forms; resolved discrepancies | Design form; may extract; document discrepancies |
| 6. Risk of bias | RoB tables and summary figures | May assess; prepare visual summaries |
| 7. Synthesis | Forest plots, narrative synthesis, SoF tables | Draft results; prepare figures |
| 8. Manuscript | PRISMA-compliant manuscript | Write; ensure all 27 items are addressed |
| 9. Submission | Cover letter, checklist, supplementary files | Prepare submission package; manage peer review responses |
The capstone drill
This module is a single integrated exercise. You will work through the complete process for a systematic review, writing each key output in sequence.
I: Structured exercise interventions (aerobic, resistance, or combined; supervised or unsupervised; ≥4 weeks duration).
C: Usual care, wait-list control, or attention control (no exercise intervention).
O: Primary: cancer-related fatigue (measured by any validated scale — FACIT-F, BFI, PFS). Secondary: quality of life, physical function, adverse events.
Design: Randomised controlled trials.
“The primary outcome (cancer-related fatigue) was measured using different validated scales across studies, so the standardised mean difference (SMD) with 95% confidence intervals was used as the summary measure. Studies were pooled using the restricted maximum likelihood (REML) random-effects model, selected a priori because clinical heterogeneity (exercise type, duration, cancer type) was anticipated. Statistical heterogeneity was assessed using I² and the 95% prediction interval. Pre-specified subgroup analyses examined the effect by exercise type (aerobic vs resistance vs combined), supervision (supervised vs unsupervised), cancer type, and intervention duration (<12 vs ≥12 weeks), with interaction tests to assess subgroup differences. Sensitivity analyses included leave-one-out analysis and restriction to studies at low risk of bias. Publication bias was assessed by funnel plot and Egger’s test (for analyses with ≥10 studies). Certainty of evidence was assessed using the GRADE framework, and results were presented in a Summary of Findings table. Analyses were conducted in R using the meta and metafor packages.”
“This systematic review found moderate-certainty evidence that structured exercise interventions probably reduce cancer-related fatigue in adult cancer survivors compared with usual care (SMD −0.38, 95% CI −0.55 to −0.21; 15 trials, 1,842 participants). The effect corresponds to a small-to-moderate clinical improvement. However, the 95% prediction interval (−0.92 to 0.16) indicates that the magnitude of benefit is likely to vary across settings, and in some contexts the effect may be negligible. Subgroup analysis suggested that supervised exercise programmes were more effective than unsupervised programmes (interaction p = 0.03), although this finding should be interpreted cautiously given the number of subgroup comparisons. The certainty of evidence was downgraded from High to Moderate due to inconsistency across trials, likely reflecting differences in exercise type, duration and cancer population. Further trials with standardised interventions and longer follow-up are needed to establish the optimal exercise prescription for fatigue reduction in this population.”
Checklist: GRADE language (probably) ✓ | CI and prediction interval ✓ | Subgroup with interaction test ✓ | Caution on subgroup ✓ | Certainty explanation ✓ | Future research need ✓