Week 1 · Module 1
Statistics diagnostic
Twenty-four questions covering everything a medical writer is expected to read, interpret and describe correctly — and a personal study order for the next four weeks
Your scenario
Second-round interview. The manager slides a forest plot across the table and asks: “What does this tell you, and what would you be careful about writing?” You know what a hazard ratio is. You are not certain what the interaction p value at the bottom means, whether the subgroup finding is reportable, or why the confidence interval on the smallest subgroup is so wide. You talk around it. She moves on. This module finds out exactly which of those gaps you have.
What this course assumes and what it does not
You are not going to become a statistician, and you do not need to. A medical writer needs to read statistical output correctly, describe it accurately, and recognise when a claim outruns the analysis. That is a different skill from generating the analysis, and it is the one that gets tested in interviews and enforced in QC.
No calculation by hand is required anywhere in this course. Every module works from output you would actually be handed — a table, a figure, a line from a statistical analysis plan.
Take it honestly
Do not look anything up. A wrong answer here is worth more than a right one, because it is the only way the diagnostic can find the gap. Guessing correctly and then skipping the module is how people arrive at an interview confident and wrong.
—
—
Your twelve areas — study the red ones first
Before you move on
Write down your three weakest areas. Return to this box after the capstone and note what changed.
Statistics is the area where writers most often overestimate themselves, because the vocabulary is familiar even when the concept is not.
Next module
1.2 — Data types and descriptive statistics
Week 1 · Module 2
Data types and descriptive statistics
Which summary belongs with which kind of data — and why reporting a mean for skewed survival data is the most common statistical error in clinical manuscripts
Your scenario
You draft a baseline characteristics table from the statistician's output. For time since diagnosis you write “mean 31.4 months (SD 28.7).” The statistician sends it back with one comment: “SD is almost as large as the mean — this is skewed. Use median and IQR.” The numbers were hers. The choice of which ones to print was yours, and that choice is a medical writing responsibility.
The four types of variable
Everything downstream — which summary, which test, which figure — follows from this classification. Get it wrong at the start and every choice after it is wrong too.
| Type | Clinical examples | Summarise with | Notes |
|---|---|---|---|
| Continuous | Age, weight, HbA1c, tumour diameter, blood pressure | Mean (SD) if symmetric Median (IQR) if skewed | Any value on a scale, with equal intervals. Differences are meaningful. |
| Ordinal | ECOG status, NYHA class, CTCAE grade, Likert scale | Median, or n (%) per category | Ordered, but gaps are not equal. Never report a mean CTCAE grade. |
| Nominal | Sex, tumour histology, country, treatment arm | n (%) | Categories with no order at all. |
| Binary | Responder, death, hospitalised, ≥ grade 3 event | n (%) with denominator | A special case of nominal with two levels. Always give the denominator. |
The error that gets flagged most
Reporting a mean for an ordinal scale. A mean ECOG of 1.7 implies the distance from 1 to 2 equals the distance from 3 to 4. It does not — the clinical difference between ECOG 2 and 3 is far larger than between 0 and 1.
Write instead: median ECOG 1 (range 0–2), or the count in each category.
Write instead: median ECOG 1 (range 0–2), or the count in each category.
Mean or median: the skew test
Definition
Skew
A distribution is skewed when it has a long tail on one side. In clinical data the tail is almost always on the right — most patients cluster low, a few have very large values. Time to event, duration of hospitalisation, biomarker concentrations and healthcare costs are all typically right-skewed.
The mean is dragged toward the tail; the median is not. When they diverge, the median is describing a typical patient and the mean is not.
The mean is dragged toward the tail; the median is not. When they diverge, the median is describing a typical patient and the mean is not.
Worked example — time since diagnosis, n = 40
Most patients
between 6 and 30 months
Three patients
at 120, 140 and 165 months
Mean
31.4 months — higher than roughly 70% of the patients in the study
SD
28.7 — almost as large as the mean
Median
18.5 months — genuinely typical
IQR
11.2 to 27.8
The tell: when the SD approaches or exceeds the mean, the data are skewed and you should be using the median. If the mean is far from the median, the same applies. Both checks take five seconds and you can run them on any output table you are handed.
Figure 1.2 — Why the mean misleads on skewed data. Most patients sit between 6 and 30 months. Three long-standing cases pull the mean to 31.4 months, higher than roughly 70% of the study population. The median of 18.5 months describes a typical patient; the mean does not. Two checks you can run on any table in five seconds: is the SD close to the mean, and is the mean far from the median?
Standard deviation is not standard error
| Standard deviation (SD) | Standard error (SE) | |
|---|---|---|
| Describes | Spread of individual patients around the mean | Precision of the estimated mean itself |
| As n grows | Stays roughly the same | Gets smaller |
| Use it for | Baseline tables, describing a population | Error bars on a mean, building confidence intervals |
| Answers | How variable are these patients? | How sure am I of this average? |
Why this one matters commercially
SE is always smaller than SD. Plotting error bars as SE instead of SD makes a figure look far tighter and more convincing than the data warrant. Reviewers check this. If a figure legend does not say which is plotted, that is a query you should raise before submission — and raising it is exactly the kind of thing that gets a junior writer noticed.
Drills
Drill 1 of 3
A baseline table reports duration of hospitalisation as mean 8.4 days (SD 9.1). What should you do?
AQuery it — the SD exceeds the mean, which indicates skew; request median and IQR
BNothing — the statistician provided these numbers
CConvert the SD to a standard error so the estimate looks more precise
DReport the mean with the range instead
Drill 2 of 3
How should CTCAE grade be summarised in a safety table?
AMean grade with standard deviation
Bn (%) at each grade, with a grade ≥3 row
CMean grade with a 95% confidence interval
DMedian grade only
Drill 3 of 3
A figure shows mean change from baseline with error bars. The legend does not state what the bars represent. Why does this matter?
AIt does not — error bars are standard
BIt is a minor formatting issue for the production team
CSD, SE and 95% CI bars have very different widths, so the reader cannot judge precision
DReaders assume SD by default, so it is fine
Interpretation drill
Write the baseline characteristics sentence. Choose the right summary for each variable and justify each choice.
Output handed to you (n = 240):
Age: mean 62.1, SD 9.4, median 62.0, IQR 55–69
Time since diagnosis (months): mean 31.4, SD 28.7, median 18.5, IQR 11.2–27.8
ECOG: 0 = 96 patients, 1 = 118, 2 = 26
Sex: male 141, female 99
Prior lines of therapy: median 2, IQR 1–3, range 0–7
Age: mean 62.1, SD 9.4, median 62.0, IQR 55–69
Time since diagnosis (months): mean 31.4, SD 28.7, median 18.5, IQR 11.2–27.8
ECOG: 0 = 96 patients, 1 = 118, 2 = 26
Sex: male 141, female 99
Prior lines of therapy: median 2, IQR 1–3, range 0–7
Model paragraph:
“The median age was 62.0 years (IQR 55–69) and 141/240 patients (58.8%) were male. Median time since diagnosis was 18.5 months (IQR 11.2–27.8). ECOG performance status was 0 in 96 patients (40.0%), 1 in 118 (49.2%) and 2 in 26 (10.8%). Patients had received a median of 2 prior lines of therapy (IQR 1–3).”
The five decisions:
1. Age — mean 62.1 and median 62.0 are nearly identical, so the distribution is symmetric and either is defensible. Reporting the median throughout keeps the table internally consistent, which is usually the better choice.
2. Time since diagnosis — SD 28.7 is close to the mean 31.4, and the mean sits well above the median. Clear right skew. Median and IQR, not mean and SD.
3. ECOG — ordinal, so counts and percentages per category. No mean.
4. Sex — nominal binary, so n (%) with the denominator shown.
5. Prior lines — count data, skewed by a tail out to 7. Median and IQR. Note the range is given in the output but adds little; the IQR is more informative and less driven by one patient.
What separates a good writer here: not the paragraph, but noticing that time since diagnosis was skewed and querying it if the statistician had supplied only the mean. That query is the job.
“The median age was 62.0 years (IQR 55–69) and 141/240 patients (58.8%) were male. Median time since diagnosis was 18.5 months (IQR 11.2–27.8). ECOG performance status was 0 in 96 patients (40.0%), 1 in 118 (49.2%) and 2 in 26 (10.8%). Patients had received a median of 2 prior lines of therapy (IQR 1–3).”
The five decisions:
1. Age — mean 62.1 and median 62.0 are nearly identical, so the distribution is symmetric and either is defensible. Reporting the median throughout keeps the table internally consistent, which is usually the better choice.
2. Time since diagnosis — SD 28.7 is close to the mean 31.4, and the mean sits well above the median. Clear right skew. Median and IQR, not mean and SD.
3. ECOG — ordinal, so counts and percentages per category. No mean.
4. Sex — nominal binary, so n (%) with the denominator shown.
5. Prior lines — count data, skewed by a tail out to 7. Median and IQR. Note the range is given in the output but adds little; the IQR is more informative and less driven by one patient.
What separates a good writer here: not the paragraph, but noticing that time since diagnosis was skewed and querying it if the statistician had supplied only the mean. That query is the job.
Reflection
Open the last baseline table you wrote or reviewed. For every continuous variable, is the SD close to the mean? Which ones should have been medians?
This five-second check on every table you touch will catch something roughly one time in three.
Next module
1.3 — Sampling, uncertainty and confidence intervals
Week 1 · Module 3
Sampling, uncertainty and confidence intervals
Why every number in a trial is an estimate, what a confidence interval actually claims, and how to describe precision without saying something false
Your scenario
Your Discussion says: “There is a 95% probability that the true hazard ratio lies between 0.55 and 0.99.” A reviewer strikes it out and writes: “This is not what a confidence interval means.” You have seen that sentence in dozens of published papers. It is still wrong, and the reviewer is still right.
Every trial number is an estimate
A trial enrols a sample. The number you report — a median, a response rate, a hazard ratio — is an estimate of what would be seen in the whole population, and a different sample would have given a slightly different number. Everything in statistics is machinery for quantifying that wobble.
Definition
Confidence interval
A range of values that is compatible with the observed data, at a stated confidence level.
The technical meaning: if you repeated the study many times and computed a 95% CI each time, about 95% of those intervals would contain the true value. The confidence attaches to the procedure, not to any single interval.
This is why
The technical meaning: if you repeated the study many times and computed a 95% CI each time, about 95% of those intervals would contain the true value. The confidence attaches to the procedure, not to any single interval.
This is why
there is a 95% probability the true value lies in this interval is wrong. The true value either is or is not in your particular interval — there is no probability about it once the study is done.How to describe a CI without being wrong
Wrong: “There is a 95% probability the true HR lies between 0.55 and 0.99.”
Wrong: “95% of patients had a hazard ratio between 0.55 and 0.99.”
Wrong: “The true HR is 0.74.”
Right: “The hazard ratio was 0.74 (95% CI 0.55–0.99).”
Right: “The data are compatible with a hazard ratio between 0.55 and 0.99.”
Right: “The estimate was imprecise, and the upper limit of 0.99 does not exclude a clinically negligible benefit.”
In most documents you simply report the interval and let it speak. The interpretive sentences belong in a Discussion, where you are commenting on precision.
Wrong: “95% of patients had a hazard ratio between 0.55 and 0.99.”
Wrong: “The true HR is 0.74.”
Right: “The hazard ratio was 0.74 (95% CI 0.55–0.99).”
Right: “The data are compatible with a hazard ratio between 0.55 and 0.99.”
Right: “The estimate was imprecise, and the upper limit of 0.99 does not exclude a clinically negligible benefit.”
In most documents you simply report the interval and let it speak. The interpretive sentences belong in a Discussion, where you are commenting on precision.
Reading width and position
A confidence interval carries two separate pieces of information, and writers usually read only one of them.
| What to look at | What it tells you | How to write about it |
|---|---|---|
| Does it cross the null? (1 for ratios, 0 for differences) | Whether no effect is compatible with the data | If it crosses, do not describe a benefit. Report the estimate and interval. |
| How wide is it? | How precisely the effect has been estimated | A wide interval warrants a precision comment even when it excludes the null. |
| Where is the far limit? | The most and least impressive effects still compatible with the data | An upper limit near the null means a trivial effect cannot be ruled out — say so. |
Four intervals, four different write-ups
HR 0.62 (0.51–0.75)
Narrow, well clear of 1. A clear and precisely estimated benefit.
HR 0.74 (0.55–0.99)
Excludes 1, but only just. Benefit, though the data do not exclude an effect close to negligible.
HR 0.31 (0.09–1.07)
Dramatic point estimate, crosses 1, very wide. Imprecise; no effect is compatible with these data. Classic small subgroup.
HR 0.98 (0.94–1.02)
Crosses 1 but is very narrow. A precisely estimated absence of effect — quite different from the row above, and often written up wrongly as the same thing.
The point: rows three and four both cross 1, so both are non-significant. They mean completely different things. Row three says we do not know; row four says we know there is no meaningful effect. Collapsing both into not statistically significant throws away the distinction that matters clinically.
Drills
Figure 1.3 — Four intervals, four different write-ups. Rows three and four both cross 1, so both are ‘not statistically significant’ — and they mean completely opposite things. Row three is an imprecise estimate from a small subgroup: we do not know. Row four is a precise estimate centred on no effect: we know there is no meaningful difference. Collapsing both into the same phrase throws away the distinction that matters clinically.
Drill 1 of 3
A response rate is 46% (95% CI 34–58). Which sentence is acceptable in a manuscript?
A“There is a 95% probability the true response rate is between 34% and 58%.”
B“95% of patients had a response rate between 34% and 58%.”
C“The response rate was 46%.”
D“The objective response rate was 46% (95% CI 34–58).”
Drill 2 of 3
Two secondary endpoints: OR 0.31 (95% CI 0.09–1.07) and OR 0.98 (95% CI 0.94–1.02). Both are non-significant. How should they be described?
AThe first as imprecise and inconclusive; the second as a precisely estimated absence of effect
BIdentically — both were not statistically significant
CThe first as showing benefit, the second as showing no benefit
DOmit both
Drill 3 of 3
A trial increases its sample size from 100 to 400 patients. What happens?
AThe standard deviation halves
BThe standard error roughly halves and the confidence interval narrows
CThe effect size increases
DThe p value increases
Interpretation drill
Write one Discussion sentence for each result. Each needs a different precision comment.
Primary endpoint: HR 0.71 (95% CI 0.55–0.92); p = 0.009
Key secondary (OS, immature): HR 0.82 (95% CI 0.61–1.10); p = 0.19
Prespecified subgroup, n = 28: HR 0.29 (95% CI 0.08–1.04)
Safety endpoint: risk difference 0.4 percentage points (95% CI −1.1 to 1.9)
Key secondary (OS, immature): HR 0.82 (95% CI 0.61–1.10); p = 0.19
Prespecified subgroup, n = 28: HR 0.29 (95% CI 0.08–1.04)
Safety endpoint: risk difference 0.4 percentage points (95% CI −1.1 to 1.9)
Model sentences:
Primary: “The study regimen reduced the risk of progression or death by 29% (HR 0.71, 95% CI 0.55–0.92; p = 0.009), with the interval excluding no effect across its full range.”
Key secondary: “Overall survival data were immature at this analysis; the hazard ratio of 0.82 (95% CI 0.61–1.10) is compatible with a moderate benefit but does not exclude no effect, and longer follow-up is required.”
Subgroup: “Although the point estimate in this subgroup was numerically favourable (HR 0.29), the confidence interval was very wide (95% CI 0.08–1.04) and includes no effect. With 28 patients, this analysis is hypothesis-generating only.”
Safety: “The risk difference was 0.4 percentage points (95% CI −1.1 to 1.9), a precise estimate that effectively excludes any clinically meaningful difference in this endpoint.”
The four different jobs:
1. Primary — a clean, precise result. State it plainly and do not over-qualify. Note the interval is described as excluding no effect, which is accurate.
2. Secondary — crosses 1 and is wide-ish. Name the reason (immaturity) and say what is needed.
3. Subgroup — the dramatic point estimate is the trap. Lead with the width and the n, not the 0.29.
4. Safety — crosses 0 but is narrow. This is the row most writers get wrong: a narrow interval around no difference is positive evidence of no difference, and saying so is more useful than “not significant”.
Check yourself: if your sentences for results 2 and 4 said roughly the same thing, re-read the four-interval table above. That distinction is the point of this module.
Primary: “The study regimen reduced the risk of progression or death by 29% (HR 0.71, 95% CI 0.55–0.92; p = 0.009), with the interval excluding no effect across its full range.”
Key secondary: “Overall survival data were immature at this analysis; the hazard ratio of 0.82 (95% CI 0.61–1.10) is compatible with a moderate benefit but does not exclude no effect, and longer follow-up is required.”
Subgroup: “Although the point estimate in this subgroup was numerically favourable (HR 0.29), the confidence interval was very wide (95% CI 0.08–1.04) and includes no effect. With 28 patients, this analysis is hypothesis-generating only.”
Safety: “The risk difference was 0.4 percentage points (95% CI −1.1 to 1.9), a precise estimate that effectively excludes any clinically meaningful difference in this endpoint.”
The four different jobs:
1. Primary — a clean, precise result. State it plainly and do not over-qualify. Note the interval is described as excluding no effect, which is accurate.
2. Secondary — crosses 1 and is wide-ish. Name the reason (immaturity) and say what is needed.
3. Subgroup — the dramatic point estimate is the trap. Lead with the width and the n, not the 0.29.
4. Safety — crosses 0 but is narrow. This is the row most writers get wrong: a narrow interval around no difference is positive evidence of no difference, and saying so is more useful than “not significant”.
Check yourself: if your sentences for results 2 and 4 said roughly the same thing, re-read the four-interval table above. That distinction is the point of this module.
Reflection
Find a published paper in your therapy area with a non-significant secondary endpoint. Is the interval wide or narrow? Did the authors distinguish inconclusive from no effect?
Most do not. Noticing it is what makes you useful on a publication team.
Next module
1.4 — The p value
Week 1 · Module 4
The p value
The most misused number in medicine — what it measures, the four things it does not, and how to write about significance without making a claim you cannot support
Your scenario
A slide in your MSL deck reads: “Highly significant benefit (p < 0.001).” Medical review rejects it. The comment: “p value describes evidence against the null, not the size or importance of the effect. State the effect size.” The absolute difference behind that p value was 1.2 percentage points, in a trial of 8,000 patients.
Definition
P value
The probability of observing data at least as extreme as yours, if the null hypothesis were true.
Everything hangs on that conditional. The p value is computed assuming there is no effect and asks how surprising your data would be under that assumption. A small p value means the data sit awkwardly with the null.
It does not tell you the probability that the null is true, the probability that the treatment works, or how large or important the effect is.
Everything hangs on that conditional. The p value is computed assuming there is no effect and asks how surprising your data would be under that assumption. A small p value means the data sit awkwardly with the null.
It does not tell you the probability that the null is true, the probability that the treatment works, or how large or important the effect is.
The four misreadings
The misreading
What is actually true
p = 0.04 means there is a 4% chance the null is truereverses the conditional
It means data this extreme would occur 4% of the time if the null were true
p = 0.04 means there is a 96% chance the treatment works
The p value says nothing about the probability that the treatment works
p < 0.001 means a big or important effectconfuses evidence with magnitude
A tiny effect gives a very small p value in a large enough trial. Always report the effect size.
p = 0.20 means there is no effect
Absence of evidence is not evidence of absence — look at the confidence interval width
Why 0.05
There is nothing scientific about 0.05. It is a convention, chosen for convenience in the 1920s, and p = 0.049 and p = 0.051 are near-identical pieces of evidence that get treated as opposite conclusions.
This is why modern reporting standards ask for exact p values with effect sizes and intervals, rather than a bare significant or not significant. Write p = 0.048, not p < 0.05, unless the value is below 0.001.
This is why modern reporting standards ask for exact p values with effect sizes and intervals, rather than a bare significant or not significant. Write p = 0.048, not p < 0.05, unless the value is below 0.001.
Style is journal-dependent
Details such as italicising P, capital or lower-case P, a space either side of the operator (P = 0.048 vs P=0.048), the number of decimal places, and a leading zero (0.048 vs .048) vary by journal and style guide. Check the target journal’s instructions for authors and follow them. The reporting principle does not change: give the exact value, with the effect size and its interval.
Spin language, and what to write instead
Phrases that get flagged
approached significance · trended towards benefit · borderline significant · a strong trend · marginally significant · failed to reach significance
All of these describe a non-significant result as though it were partly significant. Journals, regulators and MLR reviewers all treat them as spin, and BMJ and several others name them explicitly in their guidance.
Write instead: “The rate was numerically higher in the treatment arm (34% vs 28%), but the difference was not statistically significant (p = 0.09).”
Or, if the interval is informative: “The difference was 6 percentage points (95% CI −1 to 13), which does not exclude no effect.”
All of these describe a non-significant result as though it were partly significant. Journals, regulators and MLR reviewers all treat them as spin, and BMJ and several others name them explicitly in their guidance.
Write instead: “The rate was numerically higher in the treatment arm (34% vs 28%), but the difference was not statistically significant (p = 0.09).”
Or, if the interval is informative: “The difference was 6 percentage points (95% CI −1 to 13), which does not exclude no effect.”
Statistical significance is not clinical significance
| Scenario | Statistically significant? | What to write |
|---|---|---|
| 1.2 point difference on a 100-point scale, n = 8,000, p < 0.001 | Yes | Report the effect size prominently. State that the difference is below the accepted minimal clinically important difference, if one exists. |
| 9-month median OS gain, n = 60, p = 0.08 | No | Report the estimate and interval. Note the study was underpowered rather than implying no effect. |
| HR 0.71, n = 800, p = 0.009 | Yes | Report both. Effect is meaningful and precisely estimated — the straightforward case. |
| 0.4 point difference, n = 4,000, p = 0.31 | No | A precise null. Say the data effectively exclude a meaningful difference. |
The minimal clinically important difference
Many patient-reported outcome scales have a published MCID — the smallest change patients actually notice. If your statistically significant difference falls below the MCID, that fact belongs in the Discussion. Knowing to look it up is a mark of a senior writer.
Drills
Drill 1 of 3
A trial of 8,000 patients finds a 1.2-point difference on a 100-point quality-of-life scale, p < 0.001. The published MCID for the scale is 5 points. How should this be written?
A“Quality of life was highly significantly improved (p < 0.001).”
BOmit it, since the difference is not clinically meaningful
C“The between-group difference was 1.2 points (p < 0.001), below the published MCID of 5 points.”
D“A significant improvement in quality of life was demonstrated.”
Drill 2 of 3
Which statement about p = 0.03 is correct?
A“There is a 3% probability that the null hypothesis is true.”
B“There is a 97% chance the result would replicate.”
C“The effect is 3% larger than expected.”
D“If there were no true effect, data at least this extreme would occur 3% of the time.”
Drill 3 of 3
Your draft says “the difference approached statistical significance (p = 0.07).” What is the problem?
AApproached significance implies a state between significant and not, which does not exist
BNothing — the p value is reported exactly
CThe p value should be rounded to 0.1
DNon-significant results should not be reported
Interpretation drill
Rewrite each sentence so it survives medical review.
1. “The treatment was highly significant (p < 0.001), proving substantial clinical benefit.”
2. “The secondary endpoint showed a strong trend towards benefit and approached significance (p = 0.06).”
3. “With p = 0.02, there is a 98% probability that the drug is effective.”
4. “No difference was found between arms (p = 0.42).”
Context for 4: risk difference 3.1 percentage points, 95% CI −4.4 to 10.6, n = 180.
2. “The secondary endpoint showed a strong trend towards benefit and approached significance (p = 0.06).”
3. “With p = 0.02, there is a 98% probability that the drug is effective.”
4. “No difference was found between arms (p = 0.42).”
Context for 4: risk difference 3.1 percentage points, 95% CI −4.4 to 10.6, n = 180.
1. “The between-group difference was [effect size] (p < 0.001).”
Why: highly significant attaches magnitude to a p value; proving is unavailable to a single trial; substantial clinical benefit is an unsupported judgement. Supply the effect size and let it carry the argument.
2. “The rate was numerically higher in the treatment arm (34% vs 28%); the difference was not statistically significant (p = 0.06).”
Why: strong trend and approached significance are both flagged as spin. Numerically higher states what was seen without implying a conclusion.
3. “The difference was statistically significant (p = 0.02).”
Why: the 98% claim inverts the conditional. There is no way to convert a p value into a probability that the drug works — that would require a Bayesian analysis with a stated prior, which this is not.
4. “The risk difference was 3.1 percentage points (95% CI −4.4 to 10.6); the study was not powered to exclude a clinically relevant difference.”
Why: this is the most important of the four. No difference was found asserts equivalence from a non-significant result. The interval runs from a 4-point harm to a 10-point benefit — the study is uninformative, not negative. Writing it as no difference is a false negative claim, and in a safety context that is a serious error.
Score yourself: if you fixed 1, 2 and 3 but wrote 4 as “no significant difference was observed,” you improved the wording without fixing the error. Sentence 4 is the one that separates people who have learned the vocabulary from people who have learned the logic.
Why: highly significant attaches magnitude to a p value; proving is unavailable to a single trial; substantial clinical benefit is an unsupported judgement. Supply the effect size and let it carry the argument.
2. “The rate was numerically higher in the treatment arm (34% vs 28%); the difference was not statistically significant (p = 0.06).”
Why: strong trend and approached significance are both flagged as spin. Numerically higher states what was seen without implying a conclusion.
3. “The difference was statistically significant (p = 0.02).”
Why: the 98% claim inverts the conditional. There is no way to convert a p value into a probability that the drug works — that would require a Bayesian analysis with a stated prior, which this is not.
4. “The risk difference was 3.1 percentage points (95% CI −4.4 to 10.6); the study was not powered to exclude a clinically relevant difference.”
Why: this is the most important of the four. No difference was found asserts equivalence from a non-significant result. The interval runs from a 4-point harm to a 10-point benefit — the study is uninformative, not negative. Writing it as no difference is a false negative claim, and in a safety context that is a serious error.
Score yourself: if you fixed 1, 2 and 3 but wrote 4 as “no significant difference was observed,” you improved the wording without fixing the error. Sentence 4 is the one that separates people who have learned the vocabulary from people who have learned the logic.
Reflection
Search your last document for “significant”. For each hit, is the effect size reported alongside it?
A p value with no effect size beside it is almost always a sentence worth rewriting.
Next week
2.1 — Hypothesis testing, power and multiplicity
Week 2 · Module 1
Hypothesis testing, power and multiplicity
Why sample size is fixed before the first patient is enrolled, what 80% power actually promises, and why testing twelve endpoints without a plan produces a false positive almost half the time
Your scenario
A manuscript reviewer asks why your paper reports a significant result on the fourth secondary endpoint. You check the statistical analysis plan. It specifies a fixed-sequence testing hierarchy: secondary endpoints are tested in order, and testing stops at the first failure. Endpoint two failed. Endpoints three and four were therefore never formally tested — and you have reported one of them as significant.
The two errors
| Type I error (α) | Type II error (β) | |
|---|---|---|
| What happens | You conclude there is an effect when there is none | You miss a real effect |
| Everyday name | False positive | False negative |
| Conventional level | 0.05, usually two-sided | 0.20, giving 80% power |
| Who it harms | Patients given an ineffective drug; regulators guard this hard | Patients denied an effective drug; sponsors bear this risk |
| Controlled by | The significance threshold and multiplicity adjustment | Sample size and effect size assumptions |
Definition
Power
The probability of detecting a specified effect, if that effect truly exists. Power is
The word specified carries the whole meaning. A trial powered at 80% to detect a hazard ratio of 0.70 has considerably less than 80% power to detect a hazard ratio of 0.85. When someone says a trial was adequately powered, the correct follow-up question is: powered for what effect size?
Power depends on four things: the effect size you want to detect, the variability of the outcome, the significance level, and the sample size. Fix any three and the fourth follows.
1 − β.The word specified carries the whole meaning. A trial powered at 80% to detect a hazard ratio of 0.70 has considerably less than 80% power to detect a hazard ratio of 0.85. When someone says a trial was adequately powered, the correct follow-up question is: powered for what effect size?
Power depends on four things: the effect size you want to detect, the variability of the outcome, the significance level, and the sample size. Fix any three and the fourth follows.
Post hoc power is not a thing
After a study finishes, calculating power from the observed effect size — post hoc power or observed power — tells you nothing new. It is a deterministic function of the p value, so a non-significant result always yields low post hoc power, by arithmetic rather than by insight.
If a reviewer asks why a result was non-significant, the answer is the confidence interval, not a retrospective power calculation. Several journals explicitly reject post hoc power analyses.
If a reviewer asks why a result was non-significant, the answer is the confidence interval, not a retrospective power calculation. Several journals explicitly reject post hoc power analyses.
Multiplicity
Every additional test at α = 0.05 adds another chance of a false positive. The inflation is fast and it is not intuitive.
Probability of at least one false positive, independent tests at α = 0.05
1 test
5%
3 tests
14%
5 tests
23%
12 tests
46%
20 tests
64%
Read that again: with twenty endpoints and no adjustment, you are more likely than not to find at least one spurious significant result. This is why a statistical analysis plan must state the multiplicity strategy before unblinding, and why exploratory analyses must be labelled as such.
| Strategy | How it works | Where you will meet it |
|---|---|---|
| Bonferroni | Divide α by the number of tests. Simple and conservative. | Small numbers of comparisons; safety analyses. |
| Holm | Step-down version of Bonferroni. Uniformly more powerful, same protection. | Increasingly the default where Bonferroni was once used. |
| Fixed-sequence hierarchy | Endpoints tested in a prespecified order; testing stops at the first failure. | Most Phase III oncology and cardiovascular trials. |
| Alpha spending | The type I error budget is split across interim and final analyses. | Any trial with a formal interim analysis and a DSMB. |
| False discovery rate | Controls the proportion of significant findings that are false, rather than the chance of any. | Genomics, biomarker screening, high-dimensional data. |
What this means for you as a writer
Before you describe any endpoint as significant, check the SAP for the testing hierarchy. If the endpoint sits below a failure in the sequence, it was not formally tested and must be described as nominal or exploratory, however small its p value.
Writing “significant” for an endpoint below a hierarchy break is one of the few statistical errors that can have regulatory consequences, and it is entirely avoidable by reading one page of the SAP.
Writing “significant” for an endpoint below a hierarchy break is one of the few statistical errors that can have regulatory consequences, and it is entirely avoidable by reading one page of the SAP.
Modern Context — Frequentist vs. Bayesian Statistics in 60 Seconds
Most clinical trials use frequentist statistics (evaluating whether the observed data are improbable under the null hypothesis, producing p-values and confidence intervals). However, modern oncology, rare disease, and platform trials increasingly employ Bayesian methods:
- Frequentist (95% Confidence Interval): Assumes the true parameter is a fixed unknown constant. A 95% CI means that across repeated identical hypothetical trials, 95% of generated intervals would contain the true value.
- Bayesian (95% Credible Interval / CrI): Combines prior evidence with current trial data to produce a posterior probability distribution. A 95% Credible Interval means there is an actual 95% probability that the true parameter lies within that interval.
Drills
Drill 1 of 3
A SAP specifies a fixed-sequence hierarchy: PFS, then OS, then ORR, then quality of life. PFS is significant, OS is not. ORR has p = 0.01. How do you report ORR?
A“ORR was significantly higher (p = 0.01).”
B“ORR was higher in the treatment arm (nominal p = 0.01); formal testing stopped at OS, so this result is descriptive.”
COmit ORR from the paper
DReport ORR as significant and move OS lower in the hierarchy
Drill 2 of 3
A trial was powered at 90% to detect a hazard ratio of 0.70. The observed HR is 0.88 with p = 0.14. What is the correct interpretation?
AThe trial was well powered, so this proves there is no effect
BCalculate post hoc power to determine whether the trial was adequate
CThe trial was powered for a larger effect than observed; the CI shows what can and cannot be excluded
DThe endpoint was met, since HR is below 1
Drill 3 of 3
An exploratory analysis tests 40 biomarker subgroups at α = 0.05 with no adjustment. Two are significant. What should the Discussion say?
A“Two biomarkers predict response.”
BReport only the two significant biomarkers
CExploratory analyses should never be reported
D“Two of 40 exploratory tests were nominally significant, consistent with the number expected by chance; these require independent validation.”
Interpretation drill
Write the Results paragraph, respecting the hierarchy.
SAP hierarchy: 1. PFS → 2. OS → 3. ORR → 4. PROs. Test in order; stop at first non-significant.
Output:
PFS: HR 0.68 (95% CI 0.54–0.86); p = 0.001
OS: HR 0.86 (95% CI 0.68–1.09); p = 0.21
ORR: 47% vs 31%; p = 0.002
PRO score change: −4.1 vs −1.2; p = 0.03
Output:
PFS: HR 0.68 (95% CI 0.54–0.86); p = 0.001
OS: HR 0.86 (95% CI 0.68–1.09); p = 0.21
ORR: 47% vs 31%; p = 0.002
PRO score change: −4.1 vs −1.2; p = 0.03
Model paragraph:
“The primary endpoint was met: median progression-free survival favoured the study regimen (HR 0.68, 95% CI 0.54–0.86; p = 0.001). Overall survival did not differ significantly between arms (HR 0.86, 95% CI 0.68–1.09; p = 0.21), and in accordance with the prespecified testing hierarchy, formal testing stopped at this endpoint. Objective response rate was 47% versus 31% (nominal p = 0.002) and the mean change in patient-reported outcome score was −4.1 versus −1.2 (nominal p = 0.03); as these endpoints fall below the hierarchy break, both are reported descriptively and are not controlled for type I error.”
The three things that make this correct:
1. PFS is the only endpoint described as met or significant.
2. The hierarchy break is stated explicitly, in the sentence where it happens.
3. ORR and PRO are reported in full with nominal p values and an explicit statement that they are uncontrolled. Not omitted, not called significant.
The trap: ORR at p = 0.002 is a smaller p value than the primary endpoint. It is very tempting to call it significant, and it is the strongest-looking secondary result in the set. That temptation is exactly what the hierarchy exists to defeat.
“The primary endpoint was met: median progression-free survival favoured the study regimen (HR 0.68, 95% CI 0.54–0.86; p = 0.001). Overall survival did not differ significantly between arms (HR 0.86, 95% CI 0.68–1.09; p = 0.21), and in accordance with the prespecified testing hierarchy, formal testing stopped at this endpoint. Objective response rate was 47% versus 31% (nominal p = 0.002) and the mean change in patient-reported outcome score was −4.1 versus −1.2 (nominal p = 0.03); as these endpoints fall below the hierarchy break, both are reported descriptively and are not controlled for type I error.”
The three things that make this correct:
1. PFS is the only endpoint described as met or significant.
2. The hierarchy break is stated explicitly, in the sentence where it happens.
3. ORR and PRO are reported in full with nominal p values and an explicit statement that they are uncontrolled. Not omitted, not called significant.
The trap: ORR at p = 0.002 is a smaller p value than the primary endpoint. It is very tempting to call it significant, and it is the strongest-looking secondary result in the set. That temptation is exactly what the hierarchy exists to defeat.
Reflection
For the last trial you wrote up, what was the multiplicity strategy? If you cannot answer, find the SAP and locate it — it is usually a single paragraph in section 9 or 10.
Asking for the SAP before drafting Results marks you as experienced. Drafting without it is how hierarchy errors get in.
Next module
2.2 — Choosing and naming the right test
Week 2 · Module 2
Choosing and naming the right test
You will not select the test — but you will describe it in Methods, and naming the wrong one is a QC finding. A decision table that covers almost everything you will meet
Your scenario
Your Methods section says “differences between groups were compared using Student t-test.” The SAP specifies a Wilcoxon rank-sum test, because the outcome was skewed. QC catches it. It looks like carelessness — and to a reviewer who does not know you, it looks like you cannot read a SAP.
The decision table
Two questions determine almost every choice: what type is the outcome, and are the groups independent or paired.
| Outcome | Comparison | Standard test | Non-parametric alternative |
|---|---|---|---|
| Continuous | 2 independent groups | Independent t-test | Wilcoxon rank-sum (Mann–Whitney U) |
| 2 paired measurements | Paired t-test | Wilcoxon signed-rank | |
| 3+ groups | ANOVA | Kruskal–Wallis | |
| Categorical / binary | 2 independent groups | Chi-square | Fisher exact (small expected counts) |
| 2 paired measurements | McNemar test | — | |
| Ordinal | 2 groups | Wilcoxon rank-sum | Cochran–Armitage for trend |
| Time to event | 2+ groups | Log-rank test | Cox proportional hazards for adjusted estimates |
| Continuous, adjusted | 2+ groups + covariates | ANCOVA or linear regression | — |
| Binary, adjusted | 2+ groups + covariates | Logistic regression | — |
| Repeated over time | Within and between | Mixed model for repeated measures (MMRM) | — |
Parametric or non-parametric
Parametric tests (t-test, ANOVA) assume the outcome is roughly normally distributed. Non-parametric tests (Wilcoxon, Kruskal–Wallis) work on ranks and make no such assumption.
The practical signal, straight from Module 1.2: if the data were summarised with a median and IQR, the test is almost certainly non-parametric. If summarised with a mean and SD, it is probably parametric. This one heuristic will make your Methods sections consistent with your tables.
The practical signal, straight from Module 1.2: if the data were summarised with a median and IQR, the test is almost certainly non-parametric. If summarised with a mean and SD, it is probably parametric. This one heuristic will make your Methods sections consistent with your tables.
Naming errors that get caught in QC
Student t-test where the SAP says Wilcoxon — check the distribution and the SAP, not your habit.
Chi-square where expected cell counts are under 5 — should be Fisher exact.
Unpaired test for paired data — before and after in the same patient is paired.
t-test for time-to-event data — impossible with censoring.
ANOVA reported without saying which post hoc comparison was used.
Multivariate when you mean multivariable — multivariable means several predictors and one outcome, which is nearly always what you mean. Multivariate means several outcomes.
Chi-square where expected cell counts are under 5 — should be Fisher exact.
Unpaired test for paired data — before and after in the same patient is paired.
t-test for time-to-event data — impossible with censoring.
ANOVA reported without saying which post hoc comparison was used.
Multivariate when you mean multivariable — multivariable means several predictors and one outcome, which is nearly always what you mean. Multivariate means several outcomes.
Drills
Drill 1 of 3
Change in HbA1c from baseline to week 24 is compared between two arms, adjusted for baseline value and country. Which belongs in Methods?
AANCOVA with baseline value and country as covariates
BIndependent t-test
CLogistic regression
DLog-rank test
Drill 2 of 3
A rare adverse event occurs in 2 of 40 patients in one arm and 0 of 38 in the other. Which test?
AChi-square
BFisher exact test
CIndependent t-test
DMcNemar test
Drill 3 of 3
A baseline table reports a variable as median 18.5 (IQR 11.2–27.8). The Methods say the variable was compared with a t-test. What should you do?
ANothing — summary statistics and tests are chosen independently
BChange the table to mean and SD to match
CQuery it — a median with IQR suggests a non-parametric test was used
DDelete the test name from Methods
Interpretation drill
Write the statistical methods paragraph from this SAP extract.
SAP extracts:
• Primary endpoint PFS: Kaplan–Meier; stratified log-rank; HR from stratified Cox model. Stratification factors: region, prior lines.
• ORR: compared with chi-square; 95% CI by Clopper–Pearson.
• Change in EORTC QLQ-C30 score: MMRM with treatment, visit, treatment-by-visit interaction, and baseline score.
• Time since diagnosis (skewed): summarised as median (IQR); compared with Wilcoxon rank-sum.
• All tests two-sided at α = 0.05. Fixed-sequence hierarchy: PFS → OS → ORR.
• Primary endpoint PFS: Kaplan–Meier; stratified log-rank; HR from stratified Cox model. Stratification factors: region, prior lines.
• ORR: compared with chi-square; 95% CI by Clopper–Pearson.
• Change in EORTC QLQ-C30 score: MMRM with treatment, visit, treatment-by-visit interaction, and baseline score.
• Time since diagnosis (skewed): summarised as median (IQR); compared with Wilcoxon rank-sum.
• All tests two-sided at α = 0.05. Fixed-sequence hierarchy: PFS → OS → ORR.
Model paragraph:
“All tests were two-sided at a significance level of 0.05. Progression-free survival was estimated using the Kaplan–Meier method and compared between arms using a stratified log-rank test; hazard ratios with 95% confidence intervals were derived from a stratified Cox proportional hazards model, with region and number of prior lines of therapy as stratification factors. Objective response rate was compared using a chi-square test, with 95% confidence intervals calculated by the Clopper–Pearson method. Change from baseline in EORTC QLQ-C30 score was analysed using a mixed model for repeated measures including treatment, visit, treatment-by-visit interaction and baseline score. Continuous variables with skewed distributions were summarised as median (interquartile range) and compared using the Wilcoxon rank-sum test. Multiplicity was controlled by a fixed-sequence testing procedure in the order progression-free survival, overall survival, objective response rate.”
Five things to check in your version:
1. Stratified appears on both the log-rank test and the Cox model, and the stratification factors are named. Dropping the word stratified is a very common omission.
2. The CI method for ORR is named. Clopper–Pearson is not the only option and reviewers ask.
3. The MMRM terms are listed. An MMRM described without its model terms is not reproducible.
4. The Wilcoxon is tied to skewness, which keeps Methods consistent with the tables.
5. The multiplicity strategy is stated. This is the sentence most often missing from writer-drafted Methods, and its absence is a CONSORT gap.
“All tests were two-sided at a significance level of 0.05. Progression-free survival was estimated using the Kaplan–Meier method and compared between arms using a stratified log-rank test; hazard ratios with 95% confidence intervals were derived from a stratified Cox proportional hazards model, with region and number of prior lines of therapy as stratification factors. Objective response rate was compared using a chi-square test, with 95% confidence intervals calculated by the Clopper–Pearson method. Change from baseline in EORTC QLQ-C30 score was analysed using a mixed model for repeated measures including treatment, visit, treatment-by-visit interaction and baseline score. Continuous variables with skewed distributions were summarised as median (interquartile range) and compared using the Wilcoxon rank-sum test. Multiplicity was controlled by a fixed-sequence testing procedure in the order progression-free survival, overall survival, objective response rate.”
Five things to check in your version:
1. Stratified appears on both the log-rank test and the Cox model, and the stratification factors are named. Dropping the word stratified is a very common omission.
2. The CI method for ORR is named. Clopper–Pearson is not the only option and reviewers ask.
3. The MMRM terms are listed. An MMRM described without its model terms is not reproducible.
4. The Wilcoxon is tied to skewness, which keeps Methods consistent with the tables.
5. The multiplicity strategy is stated. This is the sentence most often missing from writer-drafted Methods, and its absence is a CONSORT gap.
Reflection
Open the last Methods section you wrote. Does every reported comparison have a named test, and does every test match the summary statistic used in the corresponding table?
The table-to-Methods consistency check takes ten minutes and catches errors QC would otherwise send back to you.
Next module
2.3 — Effect measures: RR, OR, HR, NNT
Week 2 · Module 3
Effect measures: RR, OR, HR and NNT
Four ways to express the same benefit, three of which are routinely described in language that belongs to one of the others — and the relative-versus-absolute distinction that drives most promotional review findings
Your scenario
Your slide says: “50% reduction in risk.” The underlying data: event rate 2% in the control arm, 1% on treatment. The relative risk reduction is indeed 50%. The absolute risk reduction is one percentage point, and the number needed to treat is 100. Both numbers are true. Only one of them is honest on its own.
The four measures
| Measure | What it is | Null value | How to describe it |
|---|---|---|---|
| Risk ratio (RR) | Risk in treated ÷ risk in control | 1 | “The risk was 30% lower” for RR 0.70. Relative language. |
| Odds ratio (OR) | Odds in treated ÷ odds in control | 1 | “The odds were 30% lower.” Odds, not risk — unless the outcome is rare. |
| Hazard ratio (HR) | Ratio of instantaneous event rates over time | 1 | “The risk of progression or death was reduced by 32%.” Not a ratio of medians. |
| Risk difference (ARR) | Risk in control − risk in treated | 0 | “4 fewer events per 100 patients treated.” Absolute language. |
| NNT | 1 ÷ absolute risk reduction | ∞ | “25 patients would need to be treated to prevent one event.” |
The same trial, five ways — control event rate 20%, treatment 10%
Risk ratio
0.10 ÷ 0.20 = 0.50 → “risk halved”, or “50% relative risk reduction”
Odds ratio
(0.10/0.90) ÷ (0.20/0.80) = 0.44 → note this is further from 1 than the RR
Risk difference
0.20 − 0.10 = 10 percentage points
NNT
1 ÷ 0.10 = 10 patients treated to prevent one event
Now change the baseline risk to 2% vs 1%. The risk ratio is still 0.50 — still a 50% reduction. But the risk difference is now 1 percentage point and the NNT is 100. The relative measure is identical; the clinical value is a tenth. This is why promotional review insists on absolute numbers alongside relative ones.
The odds ratio trap
An odds ratio is not a risk ratio, and describing OR 2.0 as doubles the risk is wrong whenever the outcome is common.
With a 5% baseline risk, OR 2.0 corresponds to roughly RR 1.9 — close enough that the shorthand is forgivable. With a 40% baseline risk, OR 2.0 corresponds to roughly RR 1.4. Describing that as doubling the risk overstates the effect by more than half.
Safe wording: “the odds of response were approximately twice as high.” Say odds when you have odds.
With a 5% baseline risk, OR 2.0 corresponds to roughly RR 1.9 — close enough that the shorthand is forgivable. With a 40% baseline risk, OR 2.0 corresponds to roughly RR 1.4. Describing that as doubling the risk overstates the effect by more than half.
Safe wording: “the odds of response were approximately twice as high.” Say odds when you have odds.
What a hazard ratio is not
A hazard ratio of 0.68 does not mean median survival is 32% longer, and it does not mean 32% of patients benefited. It is the ratio of instantaneous event rates, averaged over the follow-up period, and it assumes those rates stay in constant proportion.
Correct: “a 32% reduction in the risk of progression or death.”
Wrong: “survival was 32% longer”, “32% of patients benefited”, “patients lived 32% longer”.
The relationship between a hazard ratio and a difference in median survival depends on the shape of the curves and is not fixed.
Correct: “a 32% reduction in the risk of progression or death.”
Wrong: “survival was 32% longer”, “32% of patients benefited”, “patients lived 32% longer”.
The relationship between a hazard ratio and a difference in median survival depends on the shape of the curves and is not fixed.
The Absolute vs. Relative Risk Matrix — Never Report One Without the Other
Consider a trial of 2,000 patients with cardiovascular disease:
The Golden Compliance Rule: Promotional claims often shout "50% reduction in risk!" without mentioning that the absolute benefit is 2 patients out of 100. In scientific and regulatory medical writing, always state the baseline risk, the absolute risk reduction, and the NNT alongside any relative risk ratio.
- Control Arm (1,000 patients): 40 strokes occur → Baseline Event Rate = 4.0% (0.040)
- Treatment Arm (1,000 patients): 20 strokes occur → Treatment Event Rate = 2.0% (0.020)
| Metric | Calculation | Value & Exact Medical Writing Interpretation |
|---|---|---|
| Relative Risk (RR) | 2.0% / 4.0% | 0.50 — Patients on drug had half the risk of stroke. |
| Relative Risk Reduction (RRR) | (4.0% - 2.0%) / 4.0% | 50% — Relative reduction in stroke rate. |
| Absolute Risk Reduction (ARR) | 4.0% - 2.0% | 2.0% — Absolute reduction in stroke incidence. |
| Number Needed to Treat (NNT) | 1 / ARR (1 / 0.02) | 50 — You must treat 50 patients to prevent 1 stroke. |
Therapeutic vs. Diagnostic Metrics — A Critical Distinction
The four effect measures above (RR, OR, HR, ARR) evaluate therapeutic efficacy (how well a treatment alters patient outcomes over time). However, in modern precision medicine and companion diagnostics, you will also evaluate diagnostic test accuracy (how well an assay classifies patients as having or not having a target biomarker). In Module 2.5, we master the metrics of diagnostic accuracy: Sensitivity, Specificity, Positive/Negative Predictive Values, and ROC-AUC curves.
Drills
Figure 2.3 — Why a relative reduction alone is not enough. Both scenarios give a risk ratio of 0.50 and both can honestly be described as a 50% reduction in risk. In scenario A you prevent one event for every 10 patients treated; in scenario B, one for every 100. This is why promotional review requires the absolute figures alongside the relative claim — the relative measure is identical and the clinical value differs tenfold.
Drill 1 of 3
Control event rate 2%, treatment 1%. Which is acceptable for a promotional piece?
A“50% reduction in risk.”
B“Risk was halved.”
C“1 percentage point reduction.”
D“The event rate was 1% versus 2% — an absolute reduction of 1 percentage point (50% relative reduction).”
Drill 2 of 3
A study of a common outcome (baseline 40%) reports OR 2.1. How should it be described?
A“The odds were approximately twice as high.”
B“The risk was approximately doubled.”
C“210% of patients responded.”
D“There was a 2.1% increase in risk.”
Drill 3 of 3
HR 0.68 for PFS. Median PFS 8.2 vs 5.1 months. Which sentence is correct?
A“Survival was 32% longer with the study regimen.”
B“The regimen reduced the risk of progression or death by 32% (HR 0.68); median PFS was 8.2 versus 5.1 months.”
C“32% of patients benefited from the regimen.”
D“Patients had a 0.68 probability of surviving.”
Interpretation drill
Write one accurate sentence for each result, for a lay summary.
1. RR 0.70 (95% CI 0.58–0.84); control event rate 24%
2. OR 1.85 (95% CI 1.21–2.83); baseline response rate 38%
3. HR 0.68 (95% CI 0.54–0.86); median PFS 8.2 vs 5.1 months
4. ARR 7.2 percentage points; NNT 14
2. OR 1.85 (95% CI 1.21–2.83); baseline response rate 38%
3. HR 0.68 (95% CI 0.54–0.86); median PFS 8.2 vs 5.1 months
4. ARR 7.2 percentage points; NNT 14
1. “Among people taking the comparator, about 24 in 100 had an event. Among those taking the study drug, about 17 in 100 did — a reduction of roughly 7 in every 100 people treated.”
Why: a lay summary converts the relative measure into natural frequencies. 24% × 0.70 ≈ 17%.
2. “People taking the study drug were more likely to respond; the odds of responding were about 1.9 times higher.”
Why: the outcome is common at 38%, so this OR must not be described as almost doubling the chance of response. If the audience needs a risk statement, request the adjusted risk difference.
3. “The study drug lowered the risk of the cancer getting worse, or of death, by about a third. On average, people taking it went about three months longer before their cancer worsened.”
Why: the HR becomes a risk reduction; the median difference is reported as an observed time difference, not derived from the HR.
4. “About 7 fewer events occurred for every 100 people treated. Put another way, about 14 people would need to take the drug for one additional person to avoid an event.”
Why: ARR and NNT are two views of the same number, and pairing them is more informative than either alone.
The common failure: using percentages for everything. Natural frequencies — 17 in 100 rather than 17% — are consistently better understood by patients and by clinicians, and health literacy research supports it. This matters for lay summaries, patient information and increasingly for plain-language abstracts.
Why: a lay summary converts the relative measure into natural frequencies. 24% × 0.70 ≈ 17%.
2. “People taking the study drug were more likely to respond; the odds of responding were about 1.9 times higher.”
Why: the outcome is common at 38%, so this OR must not be described as almost doubling the chance of response. If the audience needs a risk statement, request the adjusted risk difference.
3. “The study drug lowered the risk of the cancer getting worse, or of death, by about a third. On average, people taking it went about three months longer before their cancer worsened.”
Why: the HR becomes a risk reduction; the median difference is reported as an observed time difference, not derived from the HR.
4. “About 7 fewer events occurred for every 100 people treated. Put another way, about 14 people would need to take the drug for one additional person to avoid an event.”
Why: ARR and NNT are two views of the same number, and pairing them is more informative than either alone.
The common failure: using percentages for everything. Natural frequencies — 17 in 100 rather than 17% — are consistently better understood by patients and by clinicians, and health literacy research supports it. This matters for lay summaries, patient information and increasingly for plain-language abstracts.
Reflection
Find a promotional piece or press release in your therapy area quoting a relative risk reduction. Is the absolute figure given anywhere? What is the baseline risk?
The omission of the absolute figure is the single most common finding in promotional review of efficacy claims.
Next module
2.4 — Reporting precision and intervals
Week 2 · Module 4
Reporting precision and intervals
The conventions that make statistical text pass QC first time — decimals, denominators, intervals, and the CONSORT requirement that every effect estimate carries its uncertainty
Your scenario
A CSR section comes back from QC with fourteen findings. None concerns the science. Every one is a reporting convention: a percentage without its denominator, decimals that do not match between arms, a hazard ratio with no confidence interval, and p = 0.000, which is not a possible value.
The rules
| Rule | Right |
|---|---|
| Every effect estimate carries an interval. CONSORT item 17a. | HR 0.68 (95% CI 0.54–0.86) |
| Every percentage carries its denominator. | 18/240 (7.5%), not 7.5% |
| Match decimal places across arms and across a table. | 8.2 vs 5.1 months, not 8.2 vs 5 |
| Leading zero on values below 1. | 0.68, not .68 |
| Exact p values to three decimals; below that, use the inequality. | p = 0.009; p < 0.001; never p = 0.000 |
| State the confidence level. | 95% CI, not CI |
| En dash for intervals; to when a value is negative. | 0.54–0.86; −4.4 to 10.6 |
| Do not over-report precision beyond the measurement. | 62.1 years, not 62.1374 years |
| Percentages to one decimal at most, and to none if n < 100. | With n = 40, 3/40 (8%), not (7.5%) |
| Name the interval method where it is not obvious. | 95% CI by Clopper–Pearson |
The negative-value trap
An en dash between two numbers means to. When one limit is negative, the dash becomes unreadable: −4.4–10.6 looks like a typing error.
Write −4.4 to 10.6. This applies to any interval containing a negative value — mean differences, risk differences, and change-from-baseline scores.
Write −4.4 to 10.6. This applies to any interval containing a negative value — mean differences, risk differences, and change-from-baseline scores.
Precision that oversells
Reporting 7.5% when the denominator is 40 implies a precision the data do not have — with 3 events out of 40, one more event moves the figure to 10%.
Convention: with n under 100, give percentages as whole numbers. With n under 20, many journals ask for counts only. Over-precise percentages are a common signal to reviewers that the writer is transcribing output rather than reading it.
Convention: with n under 100, give percentages as whole numbers. With n under 20, many journals ask for counts only. Over-precise percentages are a common signal to reviewers that the writer is transcribing output rather than reading it.
Drills
Drill 1 of 3
Which is correctly reported?
A“HR .68 (CI 0.54-0.86), p=0.000”
B“HR 0.68, significant at p < 0.05”
C“HR 0.68 (95% CI 0.54–0.86); p < 0.001”
D“HR 0.6832 (95% CI 0.5417–0.8614)”
Drill 2 of 3
A risk difference is 3.1 percentage points, with limits of −4.4 and 10.6. How should it be written?
A“3.1 (95% CI −4.4–10.6)”
B“3.1 (95% CI 4.4–10.6)”
C“3.1 percentage points (not significant)”
D“3.1 percentage points (95% CI −4.4 to 10.6)”
Drill 3 of 3
Three of 40 patients had a grade 4 event. How should this appear in a safety table?
A“3/40 (8%)”
B“7.5%”
C“3 patients”
D“7.50% (n = 3)”
Interpretation drill
Correct every reporting error. There are twelve.
“Median PFS was 8.2 versus 5 months (HR .68, CI 0.54-0.86, p=0.000). Objective response was achieved by 46.75% of patients. Grade 3 or higher events occurred in 7.5%. The mean difference in QoL score was 3.1374 (95% CI -4.4-10.6). Discontinuation occurred in 18 patients. The result was significant at p<0.05. Median follow-up was 24 months.”
Denominators: 240 randomised, 40 in the safety subgroup for grade 3+.
Denominators: 240 randomised, 40 in the safety subgroup for grade 3+.
Corrected: “Median PFS was 8.2 versus 5.1 months (HR 0.68, 95% CI 0.54–0.86; p < 0.001). Objective response was achieved by 112/240 patients (46.7%). Grade 3 or higher events occurred in 3/40 patients (8%). The mean difference in quality-of-life score was 3.1 (95% CI −4.4 to 10.6). Discontinuation occurred in 18/240 patients (7.5%). Median follow-up was 24 months (95% CI or IQR as available).”
The twelve:
1. 5 months → 5.1 months — decimal precision must match across arms.
2. .68 → 0.68 — leading zero.
3. CI → 95% CI — state the level.
4. Hyphen → en dash in 0.54–0.86.
5. p=0.000 → p < 0.001, italic, with spaces.
6. 46.75% → one decimal at most, and add the count and denominator.
7. 7.5% on n = 40 → 3/40 (8%) — denominator and appropriate rounding.
8. 3.1374 → 3.1 — over-precision the interval contradicts.
9. -4.4-10.6 → −4.4 to 10.6 — negative limit takes to, and a true minus sign.
10. 18 patients → 18/240 (7.5%).
11. significant at p<0.05 → redundant once the exact p value is given; delete.
12. Median follow-up 24 months → give the IQR or reverse Kaplan–Meier interval; a bare median follow-up is uninterpretable.
Score yourself: 10 or more and your reporting will pass QC first time. Items 7, 9 and 12 are the ones experienced writers still miss.
The twelve:
1. 5 months → 5.1 months — decimal precision must match across arms.
2. .68 → 0.68 — leading zero.
3. CI → 95% CI — state the level.
4. Hyphen → en dash in 0.54–0.86.
5. p=0.000 → p < 0.001, italic, with spaces.
6. 46.75% → one decimal at most, and add the count and denominator.
7. 7.5% on n = 40 → 3/40 (8%) — denominator and appropriate rounding.
8. 3.1374 → 3.1 — over-precision the interval contradicts.
9. -4.4-10.6 → −4.4 to 10.6 — negative limit takes to, and a true minus sign.
10. 18 patients → 18/240 (7.5%).
11. significant at p<0.05 → redundant once the exact p value is given; delete.
12. Median follow-up 24 months → give the IQR or reverse Kaplan–Meier interval; a bare median follow-up is uninterpretable.
Score yourself: 10 or more and your reporting will pass QC first time. Items 7, 9 and 12 are the ones experienced writers still miss.
Reflection
Take the last results table you produced. Does every percentage have a denominator, and does every estimate have an interval?
Missing denominators are the single most common QC finding on writer-drafted tables.
Next module
2.5 — Biomarkers, ROC curves and diagnostic accuracy
Week 2 · Module 5
Biomarkers, ROC curves and diagnostic accuracy
Sensitivity, specificity, predictive values, and AUC — the essential statistics of companion diagnostics, patient stratification, and precision oncology
Your scenario
You are drafting the Clinical Pharmacology & Biomarker Validation section for a novel oncology companion diagnostic (CDx). The assay identifies patients with a target gene mutation for a new targeted therapy. A junior colleague writes in the draft: “The novel PCR assay demonstrated 92% sensitivity and therefore has a 92% positive predictive value in clinical practice.” Your regulatory director marks the sentence in red: “Prevalence fallacy. Rewrite immediately before the FDA briefing document is submitted.” Sensitivity and Positive Predictive Value are not the same thing. This module teaches you how to calculate, interpret, and report diagnostic and biomarker data without error.
1. The 2×2 Diagnostic Contingency Table
Every diagnostic test, companion diagnostic (CDx), and biomarker assay is evaluated against a Reference Standard (the clinical truth or gold standard). Results fall into four mutually exclusive cells:
Test Result
Reference Standard: Disease / Mutation Present (D+)
Reference Standard: Disease / Mutation Absent (D−)
Test Positive (T+)
True Positive (TP)
Patient has disease & tests positive.
Patient has disease & tests positive.
False Positive (FP)
Patient is healthy & tests positive (Type I error).
Patient is healthy & tests positive (Type I error).
Test Negative (T−)
False Negative (FN)
Patient has disease & tests negative (Type II error).
Patient has disease & tests negative (Type II error).
True Negative (TN)
Patient is healthy & tests negative.
Patient is healthy & tests negative.
2. Sensitivity and Specificity — Intrinsic Test Properties
Sensitivity and Specificity describe how well the assay itself performs. They are intrinsic to the test and do not change when the test is used in different populations.
Sensitivity (True Positive Rate)
The probability that a patient with the disease/mutation tests positive.
Sensitivity = TP ÷ (TP + FN)A test with 95% sensitivity misses only 5% of patients with the disease (low false-negative rate). Crucial for screening tests.
Specificity (True Negative Rate)
The probability that a patient without the disease/mutation tests negative.
Specificity = TN ÷ (TN + FP)A test with 98% specificity has only a 2% false-positive rate. Crucial for confirmatory tests before giving toxic chemotherapy.
3. PPV and NPV — The Clinical Reality (Prevalence Dependency)
Clinicians and patients don't know the truth in advance—they only have the test result in hand. They ask: "Given that my test is positive, what is the probability that I actually have the mutation?"
The Prevalence Fallacy — Why PPV Changes in Different Populations
Unlike Sensitivity and Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) depend heavily on the prevalence of the disease in the tested population:
Medical Writer's Rule: Never report a standalone PPV percentage without stating the exact disease prevalence of the study cohort.
- Positive Predictive Value (PPV):
TP ÷ (TP + FP)— Probability that a test-positive patient truly has the disease. - Negative Predictive Value (NPV):
TN ÷ (TN + FN)— Probability that a test-negative patient is truly disease-free.
| Population | Prevalence | Test Sensitivity / Specificity | Resulting PPV |
|---|---|---|---|
| Oncology Clinic (High-risk) | 20% (200/1,000) | 90% Sensitivity / 95% Specificity | 81.8% PPV |
| General Screening (Low-risk) | 1% (10/1,000) | 90% Sensitivity / 95% Specificity | 15.3% PPV (85% of positives are false!) |
4. Receiver Operating Characteristic (ROC) Curves & AUC
When a biomarker is continuous (e.g., PSA level, ctDNA copy number, PD-L1 tumour proportion score), you must pick a cut-off threshold. Lowering the threshold increases sensitivity but reduces specificity. The ROC Curve plots this trade-off across every possible cut-off value.
X-Axis & Y-Axis of ROC
Y-Axis: Sensitivity (True Positive Rate, from 0 to 1.0).
X-Axis: 1 − Specificity (False Positive Rate, from 0 to 1.0).
X-Axis: 1 − Specificity (False Positive Rate, from 0 to 1.0).
A perfect test shoots straight up to (0, 1.0) — 100% sensitivity with 0% false positives.
The 45° Diagonal (AUC = 0.50)
A diagonal line from (0,0) to (1,1) represents a test with zero discrimination — no better than flipping a coin.
Any curve close to the diagonal line represents an uninformative biomarker.
Area Under the Curve (AUC / c-statistic)
The AUC measures the overall discriminatory ability of the biomarker across all cut-offs:
• 0.50: No discrimination
• 0.70–0.80: Acceptable discrimination
• 0.80–0.90: Excellent discrimination
• >0.90: Outstanding discrimination
• 0.50: No discrimination
• 0.70–0.80: Acceptable discrimination
• 0.80–0.90: Excellent discrimination
• >0.90: Outstanding discrimination
Always report AUC with its 95% Confidence Interval and p-value.
Youden's Index (J)
The optimal cut-off threshold on a ROC curve that maximizes both sensitivity and specificity:
J = Sensitivity + Specificity − 1The point on the curve furthest vertically from the 45° diagonal line.
5. Inter-Rater Agreement — Cohen's Kappa (κ)
In clinical trials involving pathology scoring (e.g., HER2 immunohistochemistry 0, 1+, 2+, 3+), two pathologists will review the same slide. Simple percentage agreement is misleading because two observers will agree on many cases purely by chance. Cohen's Kappa ($\kappa$) corrects for chance agreement:
| Kappa (κ) Value | Level of Agreement | Regulatory & Clinical Context |
|---|---|---|
| < 0.40 | Poor to Fair | Unacceptable for a clinical trial diagnostic endpoint. |
| 0.41 – 0.60 | Moderate | May require central laboratory re-adjudication. |
| 0.61 – 0.80 | Substantial | Standard regulatory benchmark for companion diagnostics. |
| 0.81 – 1.00 | Almost Perfect / Excellent | High concordance across independent central reviewers. |
6. Standard Sentence Templates for Medical Writers
How to Write Diagnostic & Biomarker Results
Diagnostic Accuracy: "At the pre-specified cut-off of ≥1%, the companion diagnostic assay demonstrated a sensitivity of 91.4% (95% CI: 85.2–95.6) and a specificity of 94.8% (95% CI: 90.1–97.7). In this study population (disease prevalence 15.0%), the positive predictive value was 75.3% (95% CI: 68.1–81.4) and the negative predictive value was 98.4% (95% CI: 95.8–99.5)."
ROC-AUC Discriminatory Power: "The novel plasma biomarker demonstrated excellent discriminatory accuracy for distinguishing responders from non-responders, with an area under the ROC curve of 0.88 (95% CI: 0.82–0.94; p < 0.001; Figure 3)."
Inter-Rater Concordance: "Inter-observer concordance between the two independent pathologists for HER2 expression scoring was substantial, with an overall agreement of 89.2% and a Cohen's kappa coefficient of 0.78 (95% CI: 0.70–0.86)."
ROC-AUC Discriminatory Power: "The novel plasma biomarker demonstrated excellent discriminatory accuracy for distinguishing responders from non-responders, with an area under the ROC curve of 0.88 (95% CI: 0.82–0.94; p < 0.001; Figure 3)."
Inter-Rater Concordance: "Inter-observer concordance between the two independent pathologists for HER2 expression scoring was substantial, with an overall agreement of 89.2% and a Cohen's kappa coefficient of 0.78 (95% CI: 0.70–0.86)."
Drills
Drill 1 of 3
A screening test has 99% sensitivity and 95% specificity for a rare disease with a prevalence of 1 in 1,000 (0.1%). If a randomly screened person tests positive, what is the approximate probability that they actually have the disease (PPV)?
A99%
BApproximately 2%
C95%
D50%
Drill 2 of 3
An oncology assay ROC curve has an Area Under the Curve (AUC) of 0.52 (95% CI: 0.44–0.60; p = 0.65). How should a medical writer report this finding?
A"The biomarker demonstrated moderate predictive efficacy."
B"The biomarker had high specificity but low sensitivity."
C"The biomarker showed no discriminatory ability beyond chance (AUC 0.52; 95% CI: 0.44–0.60; p = 0.65)."
D"The assay had 52% overall accuracy."
Drill 3 of 3
Two independent pathologists evaluate PD-L1 expression across 200 biopsy samples. The inter-rater agreement analysis reports Cohen's kappa = 0.74 (95% CI: 0.65–0.83). What does this indicate?
APoor inter-observer concordance requiring complete slide re-evaluation
BExactly 74% of slides had identical scores
CThe assay has 74% sensitivity
DSubstantial inter-observer agreement beyond chance
Practice Exercise
A clinical validation study of an EGFR mutation assay reports: 120 True Positives, 10 False Positives, 15 False Negatives, 355 True Negatives. Calculate: (1) Sensitivity, (2) Specificity, (3) PPV in this cohort. Write a two-sentence summary paragraph suitable for a CSR Section 9.5.
Sens = 120/135 (88.9%); Spec = 355/365 (97.3%); PPV = 120/130 (92.3%).
Next week
3.1 — Randomisation, blinding and allocation
Week 3 · Module 1
Randomisation, blinding and allocation
Four separate safeguards that writers routinely collapse into one word — and the interview question that separates people who have read a protocol from people who have not
Your scenario
Interview question: “This is an open-label trial. Does that mean allocation was not concealed?” Most candidates say yes. The answer is no — and knowing why is one of the fastest ways to demonstrate that you have actually read protocols rather than summarised them.
The four safeguards, in time order
| Safeguard | When it acts | What it prevents | Fails when |
|---|---|---|---|
| Randomisation | At assignment | Systematic differences between arms, including on factors nobody measured | Assignment follows a predictable pattern (alternate patients, date of birth) |
| Allocation concealment | Immediately before and at assignment | The enrolling investigator steering a patient into a preferred arm | The next assignment is guessable — unsealed lists, translucent envelopes |
| Blinding (masking) | After assignment, throughout | Differential treatment, reporting and outcome assessment | Distinctive side effects, differing formulations, unblinding events |
| Stratification | At assignment | Chance imbalance in known strong prognostic factors | Too many strata for the sample size, leaving thin cells |
The distinction that gets asked about
Allocation concealment ≠ blinding
Allocation concealment is about the moment of assignment: can the person enrolling the patient know or predict which arm the next patient will get? If they can, they can consciously or unconsciously steer sicker patients away from the experimental arm.
Blinding is about everything afterwards: do the patient, investigator, and outcome assessor know which treatment was given?
An open-label trial can have perfect allocation concealment — the assignment is made by a central interactive response system and revealed only after the patient is irrevocably enrolled. Everyone then knows the treatment, but nobody could have gamed the assignment. Empirically, poor allocation concealment inflates effect estimates more than lack of blinding does.
Blinding is about everything afterwards: do the patient, investigator, and outcome assessor know which treatment was given?
An open-label trial can have perfect allocation concealment — the assignment is made by a central interactive response system and revealed only after the patient is irrevocably enrolled. Everyone then knows the treatment, but nobody could have gamed the assignment. Empirically, poor allocation concealment inflates effect estimates more than lack of blinding does.
Levels of blinding
| Term | What it usually means — but always check the protocol |
|---|---|
| Open-label | Everyone knows. Common where formulations differ, or in oncology where toxicity profiles are unmistakable. |
| Single-blind | Usually the patient is blinded. Ambiguous — CONSORT asks you to state who was blinded rather than use this label. |
| Double-blind | Usually patient and investigator. Also ambiguous; state the roles. |
| Blinded independent central review (BICR) | Radiological outcomes assessed centrally by assessors unaware of allocation. Standard in open-label oncology trials to protect the endpoint even when treatment is visible. |
| Double-dummy | Each arm receives active drug plus placebo matching the other arm, so the regimens look identical. |
CONSORT wants roles, not labels
CONSORT item 11a asks you to state who was blinded after assignment — participants, care providers, outcome assessors — and how. Writing “double-blind” alone does not satisfy it.
Correct: “Participants, investigators and outcome assessors were blinded to treatment assignment; the study drug and placebo were identical in appearance.”
Correct: “Participants, investigators and outcome assessors were blinded to treatment assignment; the study drug and placebo were identical in appearance.”
Baseline p values
Some baseline tables include a p value column comparing arms. CONSORT explicitly discourages this. In a randomised trial any imbalance is by definition due to chance, so testing for it answers a question with a known answer.
Judge imbalance by clinical importance, not by p value. If a strong prognostic factor is visibly imbalanced, the answer is a prespecified adjusted analysis, not a significance test.
Judge imbalance by clinical importance, not by p value. If a strong prognostic factor is visibly imbalanced, the answer is a prespecified adjusted analysis, not a significance test.
Drills
Drill 1 of 3
An open-label trial uses a central interactive web response system for assignment. What can you say about allocation concealment?
AAllocation was adequately concealed; the investigator could not predict the next assignment
BIt was not concealed, because the trial was open-label
CConcealment cannot be assessed in open-label trials
DConcealment is irrelevant once randomisation is used
Drill 2 of 3
A reviewer notes that mean age is 61.2 in one arm and 64.8 in the other, p = 0.04. What should the response say?
A“Randomisation failed and the trial is invalid.”
B“In a randomised trial, baseline imbalance arises by chance; we report the prespecified age-adjusted analysis, which was consistent with the primary result.”
C“We have added a post hoc adjustment for age.”
D“Baseline differences do not require comment.”
Drill 3 of 3
An open-label oncology trial uses blinded independent central review for the primary endpoint. Why?
ATo increase the sample size
BTo convert the trial into a double-blind design
CTo protect the endpoint from assessment bias when treatment allocation is visible
DTo control for multiplicity
Interpretation drill
Write the design paragraph for Methods, satisfying CONSORT items 8–11.
Protocol extracts:
• Randomised 1:1 via central IWRS, permuted blocks of 4, stratified by region (Asia / non-Asia) and prior lines (1 vs ≥2).
• Open-label. Patients and investigators aware of assignment.
• Primary endpoint PFS assessed by blinded independent central review per RECIST v1.1.
• Sequence generated by an independent statistician not otherwise involved in the trial.
• Randomised 1:1 via central IWRS, permuted blocks of 4, stratified by region (Asia / non-Asia) and prior lines (1 vs ≥2).
• Open-label. Patients and investigators aware of assignment.
• Primary endpoint PFS assessed by blinded independent central review per RECIST v1.1.
• Sequence generated by an independent statistician not otherwise involved in the trial.
Model paragraph:
“Patients were randomly assigned in a 1:1 ratio to the study regimen or control. The randomisation sequence was generated by an independent statistician not otherwise involved in the conduct of the trial, using permuted blocks of four, and was stratified by region (Asia versus non-Asia) and number of prior lines of therapy (1 versus ≥2). Allocation was implemented through a central interactive web response system, so that investigators were unaware of the next assignment before a patient was enrolled. The trial was open-label: patients and investigators were aware of treatment assignment after randomisation. To limit assessment bias, the primary endpoint of progression-free survival was assessed by blinded independent central review according to RECIST version 1.1.”
The four CONSORT items covered:
8a — method of sequence generation (permuted blocks, independent statistician).
8b — type of randomisation and stratification factors, with their levels spelled out.
9 — allocation concealment mechanism (central IWRS), stated explicitly as concealment rather than left implied.
11a — who was blinded and who was not, by role, plus the mitigation used for the endpoint.
What most drafts get wrong: writing “patients were randomised 1:1 in an open-label trial” and stopping. That sentence satisfies none of the four items, and it leaves a reviewer unable to judge the trial's internal validity. Note also that the block size is disclosed here — some sponsors withhold it during conduct to protect concealment, so check before publishing it.
“Patients were randomly assigned in a 1:1 ratio to the study regimen or control. The randomisation sequence was generated by an independent statistician not otherwise involved in the conduct of the trial, using permuted blocks of four, and was stratified by region (Asia versus non-Asia) and number of prior lines of therapy (1 versus ≥2). Allocation was implemented through a central interactive web response system, so that investigators were unaware of the next assignment before a patient was enrolled. The trial was open-label: patients and investigators were aware of treatment assignment after randomisation. To limit assessment bias, the primary endpoint of progression-free survival was assessed by blinded independent central review according to RECIST version 1.1.”
The four CONSORT items covered:
8a — method of sequence generation (permuted blocks, independent statistician).
8b — type of randomisation and stratification factors, with their levels spelled out.
9 — allocation concealment mechanism (central IWRS), stated explicitly as concealment rather than left implied.
11a — who was blinded and who was not, by role, plus the mitigation used for the endpoint.
What most drafts get wrong: writing “patients were randomised 1:1 in an open-label trial” and stopping. That sentence satisfies none of the four items, and it leaves a reviewer unable to judge the trial's internal validity. Note also that the block size is disclosed here — some sponsors withhold it during conduct to protect concealment, so check before publishing it.
Reflection
For the last trial you wrote up, could an enrolling investigator have predicted the next assignment? How do you know?
The answer is in the protocol randomisation section, usually two paragraphs. Being able to cite it is what the interview question is really testing.
Next module
3.2 — Analysis populations
Week 3 · Module 2
Analysis populations
ITT, mITT, per-protocol, evaluable, safety — five populations, five different denominators, and the mistake of using the wrong one for adverse event tables
Your scenario
A safety table in your draft uses n = 240 as the denominator for every adverse event row. The randomised population was 240; the number who received at least one dose was 233. Seven patients who never took the drug are counted in the denominator for its side effects. Every percentage in the table is slightly wrong, in the sponsor's favour.
The five populations
| Population | Who is included | Analysed by | Used for |
|---|---|---|---|
| Intention-to-treat (ITT) | Everyone randomised, regardless of what they received or whether they were followed | Arm randomised to | Primary efficacy. Preserves randomisation. |
| Modified ITT (mITT) | Randomised patients meeting some minimum criterion — usually at least one dose and one post-baseline assessment | Arm randomised to | Efficacy where a definition is needed. Definition varies by trial — always check the SAP. |
| Per-protocol (PP) | Patients who completed treatment without major protocol deviations | Arm randomised to | Supportive analysis; primary in some non-inferiority settings. |
| Evaluable | Patients with an assessable outcome (e.g. measurable disease at baseline and one scan) | Arm randomised to | Response-rate endpoints. |
| Safety | All who received at least one dose | Treatment actually received | All adverse event reporting. |
The one rule to carry away
Efficacy is analysed by the arm the patient was randomised to. Safety is analysed by the treatment they actually received.
A patient randomised to control who mistakenly received the study drug is counted in the control arm for efficacy and in the study drug arm for safety. This is not inconsistency — the two analyses answer different questions. Efficacy asks about a treatment strategy; safety asks about an exposure.
A patient randomised to control who mistakenly received the study drug is counted in the control arm for efficacy and in the study drug arm for safety. This is not inconsistency — the two analyses answer different questions. Efficacy asks about a treatment strategy; safety asks about an exposure.
Why per-protocol is not the honest analysis
It is tempting to think the per-protocol analysis is more truthful because it shows what happens when patients actually take the drug. It is not.
Excluding patients after randomisation breaks randomisation. Patients who deviate from protocol are systematically different — often sicker, or experiencing toxicity — and they do not deviate at equal rates in both arms. A per-protocol analysis therefore compares two non-randomly-selected groups.
PP is a supportive analysis. If ITT and PP disagree substantially, that discrepancy is itself a finding and belongs in the Discussion.
Excluding patients after randomisation breaks randomisation. Patients who deviate from protocol are systematically different — often sicker, or experiencing toxicity — and they do not deviate at equal rates in both arms. A per-protocol analysis therefore compares two non-randomly-selected groups.
PP is a supportive analysis. If ITT and PP disagree substantially, that discrepancy is itself a finding and belongs in the Discussion.
The mITT warning
There is no standard definition of modified ITT. Some trials require one dose; others require one dose plus a post-baseline assessment; others exclude patients who turned out to be ineligible.
Because the definition is discretionary, mITT is the analysis population most open to criticism, and reviewers scrutinise it. Never write "mITT" without reproducing the trial's own definition in the same sentence, and check that your definition matches the SAP exactly.
Because the definition is discretionary, mITT is the analysis population most open to criticism, and reviewers scrutinise it. Never write "mITT" without reproducing the trial's own definition in the same sentence, and check that your definition matches the SAP exactly.
Drills
Drill 1 of 3
A patient randomised to the treatment arm withdraws consent before receiving any dose. How are they handled?
AExcluded from all analyses
BMoved to the control arm since they received nothing
CIncluded in the safety population
DIncluded in ITT in the treatment arm; excluded from the safety population
Drill 2 of 3
240 randomised, 233 received at least one dose, 198 completed per protocol. What denominator belongs in the adverse event table?
A233
B240
C198
DWhichever gives the cleanest percentages
Drill 3 of 3
The ITT analysis gives HR 0.88 (p = 0.14); the per-protocol analysis gives HR 0.71 (p = 0.01). What should the paper report?
AThe per-protocol result, since it shows what happens with actual treatment
BITT as primary, PP as supportive, and discuss why they differ
CAn average of the two estimates
DOnly ITT, omitting PP entirely
Interpretation drill
Write the analysis populations paragraph and identify the correct denominator for each table.
Disposition: 240 randomised (120 per arm). 233 received ≥1 dose (118 study drug, 115 control). 226 had ≥1 post-baseline assessment. 198 completed without major deviation. 4 patients randomised to control received study drug in error.
Tables to produce: (1) primary PFS analysis, (2) objective response rate, (3) treatment-emergent adverse events, (4) supportive sensitivity analysis of PFS.
Tables to produce: (1) primary PFS analysis, (2) objective response rate, (3) treatment-emergent adverse events, (4) supportive sensitivity analysis of PFS.
Model paragraph:
“Efficacy analyses were conducted in the intention-to-treat population, comprising all 240 randomised patients analysed according to the treatment arm to which they were assigned. The modified intention-to-treat population, used for response-rate analyses, comprised the 226 patients who received at least one dose and had at least one post-baseline tumour assessment. The safety population comprised the 233 patients who received at least one dose, analysed according to the treatment actually received; four patients randomised to control who received the study drug in error were included in the study-drug group for safety analyses. The per-protocol population, comprising the 198 patients who completed treatment without a major protocol deviation, was used for a supportive sensitivity analysis.”
The four denominators:
1. Primary PFS — ITT, n = 240 (120 per arm). Randomisation preserved.
2. Objective response rate — mITT, n = 226. Response cannot be assessed without a post-baseline scan, and the definition is stated explicitly.
3. Treatment-emergent adverse events — safety, n = 233, split 122 study drug and 111 control, not 118/115. The four cross-over patients move to the study-drug group because that is what they were exposed to.
4. Supportive sensitivity analysis — per-protocol, n = 198, explicitly labelled supportive.
The item that catches people: table 3. The instinct is to keep the arms at 118 and 115 because that is how the doses were assigned by randomisation. Safety follows exposure, so the four misdosed patients move. If your answer kept them in control, re-read the rule box above — this exact scenario appears in interviews and in real CSRs.
“Efficacy analyses were conducted in the intention-to-treat population, comprising all 240 randomised patients analysed according to the treatment arm to which they were assigned. The modified intention-to-treat population, used for response-rate analyses, comprised the 226 patients who received at least one dose and had at least one post-baseline tumour assessment. The safety population comprised the 233 patients who received at least one dose, analysed according to the treatment actually received; four patients randomised to control who received the study drug in error were included in the study-drug group for safety analyses. The per-protocol population, comprising the 198 patients who completed treatment without a major protocol deviation, was used for a supportive sensitivity analysis.”
The four denominators:
1. Primary PFS — ITT, n = 240 (120 per arm). Randomisation preserved.
2. Objective response rate — mITT, n = 226. Response cannot be assessed without a post-baseline scan, and the definition is stated explicitly.
3. Treatment-emergent adverse events — safety, n = 233, split 122 study drug and 111 control, not 118/115. The four cross-over patients move to the study-drug group because that is what they were exposed to.
4. Supportive sensitivity analysis — per-protocol, n = 198, explicitly labelled supportive.
The item that catches people: table 3. The instinct is to keep the arms at 118 and 115 because that is how the doses were assigned by randomisation. Safety follows exposure, so the four misdosed patients move. If your answer kept them in control, re-read the rule box above — this exact scenario appears in interviews and in real CSRs.
Reflection
In your last safety table, was the denominator the randomised or the treated population? Check.
If it was the randomised population, every percentage in that table understated the event rate.
Next module
3.3 — Estimands, intercurrent events and missing data
Week 3 · Module 3
Estimands, intercurrent events and missing data
ICH E9(R1) — the framework that forces you to state precisely what question a trial is answering, before anyone looks at the data. Now expected in every regulatory submission
Your scenario
A patient starts rescue medication at week 8 and their symptom score improves. Is that a treatment success? It depends entirely on what question you are asking — and before ICH E9(R1), trials frequently did not say. Two statisticians could analyse the same dataset, both correctly, and reach different conclusions because they had silently answered different questions.
Definition
Estimand
A precise description of the treatment effect the trial is designed to estimate, stated before the trial begins.
ICH E9(R1), the 2019 addendum to the statistical principles guideline, requires five attributes:
Attribute 3 is the one that was previously left implicit, and it is where almost all the disagreement lived.
ICH E9(R1), the 2019 addendum to the statistical principles guideline, requires five attributes:
1. Population — which patients2. Variable — which endpoint3. Intercurrent events — what post-randomisation events might occur, and how each is handled4. Population-level summary — difference in means, hazard ratio, risk difference5. Treatment condition — the regimens being comparedAttribute 3 is the one that was previously left implicit, and it is where almost all the disagreement lived.
Intercurrent events
An intercurrent event is anything happening after randomisation that affects the interpretation or the very existence of the outcome. These are not protocol deviations and they are not missing data — though they can cause missing data.
| Event | Why it complicates the outcome |
|---|---|
| Discontinuation for toxicity | The patient is no longer on treatment; is their later score a treatment result? |
| Discontinuation for lack of efficacy | Same, but informatively — the reason is related to the outcome. |
| Rescue or additional medication | Any subsequent improvement may belong to the rescue, not the study drug. |
| Treatment switching or crossover | Common in oncology; dilutes an overall survival comparison. |
| Death from an unrelated cause | A symptom score cannot exist after death — the variable does not merely go missing, it ceases to be defined. |
The five strategies
| Strategy | How the Event is Handled | Concrete Clinical Analogy (Blood Pressure Trial) | The Question It Answers |
|---|---|---|---|
| Treatment policy | Ignore the event — collect and use the outcome regardless of what happened. | Patient develops mild cough at Week 4, stops study drug, and takes generic pill. Blood pressure is still measured at Week 12 and analysed. | What is the effect of prescribing the treatment strategy in real clinical practice? (Standard ITT). |
| Composite | Incorporate the intercurrent event directly into a composite binary endpoint. | Patient requires emergency antihypertensive rescue at Week 6. Patient is automatically classified as a "Treatment Failure / Non-responder". | Did the patient achieve target blood pressure without needing rescue medication? |
| Hypothetical | Estimate what the outcome would have been had the event not occurred. | Patient moves to another country or suffers an unrelated fracture. Statistical models estimate their Week 12 blood pressure had they stayed. | What would the drug effect be in the hypothetical scenario where no dropouts occurred? |
| While on treatment | Evaluate the outcome only up to the exact moment the intercurrent event occurred. | Patient takes drug for 5 weeks before discontinuing due to headache. Blood pressure reduction is evaluated up to Week 5 only. | What is the rate of benefit or adverse events while the patient is actively taking the drug? (Safety tables). |
| Principal stratum | Restrict the analysis to the subset of patients who would never experience the event under either treatment arm. | Identify the biological subpopulation that would not develop cough regardless of whether given drug or placebo. | What is the treatment effect in the specific biological subpopulation capable of tolerating the drug? |
Why this matters to you as a writer
The estimand determines what your Results section is allowed to claim. A treatment-policy estimand supports statements about a treatment strategy in practice. A while-on-treatment estimand supports statements about the effect during exposure — and describing that as the effect of the drug generally would overreach.
Regulators now expect the estimand to be stated explicitly in the protocol, the SAP and the CSR. If you cannot find it, ask. Writing Results without knowing the estimand is writing an answer without knowing the question.
Regulators now expect the estimand to be stated explicitly in the protocol, the SAP and the CSR. If you cannot find it, ask. Writing Results without knowing the estimand is writing an answer without knowing the question.
Missing data
| Mechanism | Meaning | Consequence |
|---|---|---|
| MCAR missing completely at random | Missingness unrelated to anything — a sample tube broke | Complete-case analysis is unbiased, just less precise. Rare in practice. |
| MAR missing at random | Missingness explained by observed variables — sicker patients drop out, and you measured sickness | Multiple imputation or MMRM are valid. The usual working assumption. |
| MNAR missing not at random | Missingness depends on the unobserved value itself — patients drop out because they deteriorated, unmeasured | No method fixes this. Requires sensitivity analysis under explicit assumptions. |
Last observation carried forward
LOCF assumes a patient's outcome froze at the moment they dropped out. In a progressive disease this is implausible and systematically biased — usually in favour of the treatment, because patients who drop out are those doing worst and their deterioration is never recorded.
LOCF is now discouraged in regulatory settings. If you find it in a study you are writing up, that is a limitation to state, not a detail to pass over. Preferred approaches are multiple imputation, MMRM, or a prespecified estimand strategy that handles the event directly rather than patching the gap.
LOCF is now discouraged in regulatory settings. If you find it in a study you are writing up, that is a limitation to state, not a detail to pass over. Preferred approaches are multiple imputation, MMRM, or a prespecified estimand strategy that handles the event directly rather than patching the gap.
Drills
Drill 1 of 3
In an oncology trial, 40% of control-arm patients cross over to the experimental drug after progression. Which estimand strategy answers "what is the OS benefit if nobody had crossed over"?
ATreatment policy
BComposite
CHypothetical
DWhile on treatment
Drill 2 of 3
18% of outcome data are missing, handled by LOCF, in a trial of a progressive neurodegenerative disease. What is the concern?
AThe sample size is reduced
BIt inflates the standard error
CLOCF cannot be used with continuous outcomes
DLOCF assumes patients stopped deteriorating at dropout, which biases results
Drill 3 of 3
Which is an intercurrent event under ICH E9(R1)?
AA patient starts rescue medication at week 8
BA patient was enrolled who did not meet an inclusion criterion
CA patient missed a scheduled visit
DThe protocol was amended during the trial
Interpretation drill
Write the estimand paragraph, covering all five attributes.
Trial: adults with moderate-to-severe atopic dermatitis, randomised to drug X or placebo for 16 weeks.
Endpoint: proportion achieving EASI-75 at week 16.
Intercurrent events: (a) use of rescue topical corticosteroids — handled as treatment failure; (b) permanent discontinuation of study drug — handled as treatment failure.
Summary: risk difference between arms.
Endpoint: proportion achieving EASI-75 at week 16.
Intercurrent events: (a) use of rescue topical corticosteroids — handled as treatment failure; (b) permanent discontinuation of study drug — handled as treatment failure.
Summary: risk difference between arms.
Model paragraph:
“The primary estimand was defined as follows. The population was adults with moderate-to-severe atopic dermatitis meeting the trial eligibility criteria. The variable was achievement of at least a 75% improvement in Eczema Area and Severity Index score (EASI-75) at week 16. Two intercurrent events were addressed using a composite strategy: patients who used rescue topical corticosteroids, and patients who permanently discontinued study drug, were classified as not achieving EASI-75 regardless of their observed score. The population-level summary was the difference in the proportion achieving EASI-75 between arms. The treatment condition was drug X compared with placebo over 16 weeks.”
The five attributes, checked:
1. Population — stated, and tied to eligibility criteria.
2. Variable — named precisely, with the threshold and the timepoint.
3. Intercurrent events — both named, and the strategy named as composite. This is where most drafts fall short: they list the events without naming the strategy, or name a strategy without listing the events.
4. Summary — risk difference, stated explicitly. Not left to be inferred from the tables.
5. Treatment condition — both arms and the duration.
Why the composite strategy here: in dermatology, using rescue steroids is itself a sign the treatment is not working. Counting a patient as a responder because their skin improved on steroids would credit the study drug with the steroid effect. Composite folds the event into the definition of failure, which matches the clinical question. Note this also means the analysis needs no imputation for these patients — the composite strategy has answered the missing-data question by definition.
“The primary estimand was defined as follows. The population was adults with moderate-to-severe atopic dermatitis meeting the trial eligibility criteria. The variable was achievement of at least a 75% improvement in Eczema Area and Severity Index score (EASI-75) at week 16. Two intercurrent events were addressed using a composite strategy: patients who used rescue topical corticosteroids, and patients who permanently discontinued study drug, were classified as not achieving EASI-75 regardless of their observed score. The population-level summary was the difference in the proportion achieving EASI-75 between arms. The treatment condition was drug X compared with placebo over 16 weeks.”
The five attributes, checked:
1. Population — stated, and tied to eligibility criteria.
2. Variable — named precisely, with the threshold and the timepoint.
3. Intercurrent events — both named, and the strategy named as composite. This is where most drafts fall short: they list the events without naming the strategy, or name a strategy without listing the events.
4. Summary — risk difference, stated explicitly. Not left to be inferred from the tables.
5. Treatment condition — both arms and the duration.
Why the composite strategy here: in dermatology, using rescue steroids is itself a sign the treatment is not working. Counting a patient as a responder because their skin improved on steroids would credit the study drug with the steroid effect. Composite folds the event into the definition of failure, which matches the clinical question. Note this also means the analysis needs no imputation for these patients — the composite strategy has answered the missing-data question by definition.
Reflection
For a trial you have written up, what were the intercurrent events and how were they handled? If the protocol predates 2019 it may not use the word estimand — what did it do instead?
Pre-2019 protocols usually handled this implicitly through the analysis population definition and an imputation rule. Finding those two things tells you the estimand in all but name.
Next module
3.4 — Time-to-event and survival analysis
Week 3 · Module 4
Time-to-event and survival analysis
Kaplan–Meier curves, censoring, hazard ratios and the proportional hazards assumption — the analysis you will describe more often than any other, and the one most often described wrongly
Your scenario
Your draft reads: “Median overall survival was not reached, indicating no survival benefit.” The opposite is true. Median not reached means fewer than half the patients in that arm had died at the time of analysis — in an immunotherapy trial that is usually the most encouraging line in the results.
Censoring
Definition
Censoring
A patient is censored when they have not had the event by the time you last observed them. You know they were event-free up to that point, and nothing after.
Reasons: the study ended, the patient withdrew, they were lost to follow-up, or they reached a competing endpoint.
Censoring is not missing data and not failure. It is partial information, and survival methods exist precisely to use it. A t-test cannot — which is why time-to-event data need Kaplan–Meier and Cox, not a comparison of means.
Reasons: the study ended, the patient withdrew, they were lost to follow-up, or they reached a competing endpoint.
Censoring is not missing data and not failure. It is partial information, and survival methods exist precisely to use it. A t-test cannot — which is why time-to-event data need Kaplan–Meier and Cox, not a comparison of means.
Informative censoring
Survival methods assume censoring is non-informative: patients censored at a given time have the same future risk as those still under observation.
That assumption breaks when patients are censored because they were deteriorating — dropping out from toxicity, or withdrawing as their disease worsened. When dropout rates differ between arms, informative censoring should be named as a limitation. Look at the disposition table before you write the survival section; if one arm lost noticeably more patients, that is your limitation paragraph.
That assumption breaks when patients are censored because they were deteriorating — dropping out from toxicity, or withdrawing as their disease worsened. When dropout rates differ between arms, informative censoring should be named as a limitation. Look at the disposition table before you write the survival section; if one arm lost noticeably more patients, that is your limitation paragraph.
Reading a Kaplan–Meier curve
| Feature | What it tells you |
|---|---|
| Each step down | An event occurred. Step height grows as fewer patients remain at risk. |
| Tick marks | Censored patients. Heavy early ticks mean a lot of early dropout — investigate why. |
| Where the curve crosses 0.5 | The median. If it never crosses, the median is not reached. |
| Numbers at risk | The most important row on the figure. When it falls into single figures, the tail of the curve is unreliable. |
| Curves that cross | Proportional hazards is violated. A single hazard ratio may be misleading. |
| Early separation then parallel | A one-off early benefit rather than a continuing one. |
| Late separation | Delayed effect, typical of immunotherapy. The log-rank test loses power here. |
Figure 3.4a — Reading a Kaplan–Meier curve. Each step down is an event; steps get taller as fewer patients remain at risk. Tick marks are censored patients — still event-free when last seen. The medians are read where each curve crosses 50%: 5.6 and 9.4 months. The numbers-at-risk row is the most important part of the figure: by 24 months it has fallen to 18 and 7, so the right-hand end of both curves rests on very few patients and should not be described as a plateau.
The tail of the curve
The right-hand end of a Kaplan–Meier curve is estimated from very few patients and moves dramatically when one of them has an event. A curve that appears to plateau at 30% survival may rest on four patients.
Always check the numbers-at-risk row before describing the tail. If it reads 6 and 4, do not write about a plateau or a durable response rate. This is one of the most common overinterpretations in oncology communications.
Always check the numbers-at-risk row before describing the tail. If it reads 6 and 4, do not write about a plateau or a durable response rate. This is one of the most common overinterpretations in oncology communications.
The hazard ratio and its assumption
Definition
Hazard ratio
The ratio of the instantaneous event rates in the two arms, averaged across the follow-up period.
It is not a ratio of medians, not a proportion of patients who benefited, and not a percentage increase in survival time.
HR 0.68 means the rate of events in the treatment arm is 68% of the rate in the control arm at any given moment — a 32% reduction in the risk of the event.It is not a ratio of medians, not a proportion of patients who benefited, and not a percentage increase in survival time.
Proportional hazards
The Cox model assumes the hazard ratio is constant over time. When it is not, a single hazard ratio summarises something that changes — and can be actively misleading.
Signs of violation: curves that cross; curves that separate late (delayed effect); curves that converge after early separation.
What is done about it: report restricted mean survival time (RMST), which does not need the assumption; present milestone survival rates at fixed timepoints; or use a weighted log-rank test. Diagnostics include Schoenfeld residuals — you will see this named in SAPs.
If you are writing up a trial where the curves cross, a single hazard ratio in the abstract is not sufficient, and saying so is a genuinely senior contribution.
Signs of violation: curves that cross; curves that separate late (delayed effect); curves that converge after early separation.
What is done about it: report restricted mean survival time (RMST), which does not need the assumption; present milestone survival rates at fixed timepoints; or use a weighted log-rank test. Diagnostics include Schoenfeld residuals — you will see this named in SAPs.
If you are writing up a trial where the curves cross, a single hazard ratio in the abstract is not sufficient, and saying so is a genuinely senior contribution.
Drills
Figure 3.4b — Two patterns that break the proportional hazards assumption (schematic). The Cox model assumes the hazard ratio is constant over time. In both panels it is not, so one hazard ratio summarises something that changes. When you see either pattern, the abstract needs more than a single HR: restricted mean survival time, milestone rates at fixed timepoints, or a weighted log-rank test.
Drill 1 of 3
Median OS is not reached in the treatment arm and is 18.4 months in control. What does this mean?
AThe treatment arm performed worse
BFewer than half of treated patients had died; the median cannot yet be estimated
CThe data are missing for that arm
DAn analysis error occurred
Drill 2 of 3
Two survival curves are superimposed for 8 months, then separate clearly. What should you consider?
AThere is no difference between arms
BThe hazard ratio fully captures the effect
CA delayed treatment effect; proportional hazards may be violated
DThe early data are erroneous and should be excluded
Drill 3 of 3
A KM curve appears to plateau at 28% survival beyond 36 months. The numbers-at-risk row reads 5 and 3 at those timepoints. What should you write?
A“A plateau was observed, suggesting a cured fraction of 28%.”
BOmit the long-term data entirely
C“28% of patients achieved long-term response.”
D“The 36-month survival estimate was 28% (95% CI …), though few patients remained at risk beyond this point.”
Interpretation drill
Write the survival results paragraph.
Output:
Median PFS: 9.4 months (95% CI 7.8–11.2) vs 5.6 months (95% CI 4.9–6.8)
HR 0.62 (95% CI 0.49–0.79); stratified log-rank p < 0.001
Median follow-up 22.4 months (IQR 18.1–26.9)
12-month PFS rate: 41% vs 19%
Median OS: not reached vs 24.1 months; HR 0.79 (95% CI 0.58–1.08); p = 0.14
Numbers at risk at 24 months: 18 vs 7
Schoenfeld residual test for PH: p = 0.61
Censoring: 34% treatment arm, 29% control arm
Median PFS: 9.4 months (95% CI 7.8–11.2) vs 5.6 months (95% CI 4.9–6.8)
HR 0.62 (95% CI 0.49–0.79); stratified log-rank p < 0.001
Median follow-up 22.4 months (IQR 18.1–26.9)
12-month PFS rate: 41% vs 19%
Median OS: not reached vs 24.1 months; HR 0.79 (95% CI 0.58–1.08); p = 0.14
Numbers at risk at 24 months: 18 vs 7
Schoenfeld residual test for PH: p = 0.61
Censoring: 34% treatment arm, 29% control arm
Model paragraph:
“At a median follow-up of 22.4 months (IQR 18.1–26.9), median progression-free survival was 9.4 months (95% CI 7.8–11.2) with the study regimen and 5.6 months (95% CI 4.9–6.8) with control, corresponding to a 38% reduction in the risk of progression or death (HR 0.62, 95% CI 0.49–0.79; stratified log-rank p < 0.001). The proportion free from progression at 12 months was 41% versus 19%. The proportional hazards assumption was supported by testing of Schoenfeld residuals (p = 0.61). Median overall survival was not reached in the study regimen arm and was 24.1 months in the control arm (HR 0.79, 95% CI 0.58–1.08; p = 0.14); overall survival data were immature at this analysis, with 18 and 7 patients respectively remaining at risk at 24 months.”
Seven things this paragraph does right:
1. Opens with median follow-up. Survival results are uninterpretable without it, and it belongs first.
2. HR described as a risk reduction, not as a ratio of medians or a percentage of patients.
3. Stratified log-rank named, matching the SAP.
4. Milestone rate given, which is more useful than the median alone when curves may not be proportional.
5. PH assumption addressed explicitly. Most writer drafts omit this even when the diagnostic is in the output.
6. Not reached used correctly, and immaturity stated as the reason rather than implied.
7. Numbers at risk quoted so the reader can judge the tail.
What to notice about the censoring figures: 34% versus 29% is a modest difference and probably does not need a limitation sentence. If it had been 45% versus 25%, informative censoring would belong in the Discussion. Knowing where that line sits comes with practice — but knowing to look at the numbers is the part you can adopt today.
“At a median follow-up of 22.4 months (IQR 18.1–26.9), median progression-free survival was 9.4 months (95% CI 7.8–11.2) with the study regimen and 5.6 months (95% CI 4.9–6.8) with control, corresponding to a 38% reduction in the risk of progression or death (HR 0.62, 95% CI 0.49–0.79; stratified log-rank p < 0.001). The proportion free from progression at 12 months was 41% versus 19%. The proportional hazards assumption was supported by testing of Schoenfeld residuals (p = 0.61). Median overall survival was not reached in the study regimen arm and was 24.1 months in the control arm (HR 0.79, 95% CI 0.58–1.08; p = 0.14); overall survival data were immature at this analysis, with 18 and 7 patients respectively remaining at risk at 24 months.”
Seven things this paragraph does right:
1. Opens with median follow-up. Survival results are uninterpretable without it, and it belongs first.
2. HR described as a risk reduction, not as a ratio of medians or a percentage of patients.
3. Stratified log-rank named, matching the SAP.
4. Milestone rate given, which is more useful than the median alone when curves may not be proportional.
5. PH assumption addressed explicitly. Most writer drafts omit this even when the diagnostic is in the output.
6. Not reached used correctly, and immaturity stated as the reason rather than implied.
7. Numbers at risk quoted so the reader can judge the tail.
What to notice about the censoring figures: 34% versus 29% is a modest difference and probably does not need a limitation sentence. If it had been 45% versus 25%, informative censoring would belong in the Discussion. Knowing where that line sits comes with practice — but knowing to look at the numbers is the part you can adopt today.
Reflection
Find a Kaplan–Meier figure in a recent paper in your therapy area. What are the numbers at risk at the last timepoint shown? Does the text describe the tail more confidently than those numbers support?
In perhaps a third of oncology papers, the answer is yes.
Next week
4.1 — Non-inferiority, confounding and IPW
Week 4 · Module 1
Non-inferiority, confounding and IPW
Two designs where the usual logic inverts — non-inferiority trials, where a non-significant result is the goal, and observational studies, where the entire analysis exists to compensate for the absence of randomisation
Your scenario
Interview question: “This real-world study used inverse probability of treatment weighting. What does that buy you, and what does it not?” The first half is answerable from a textbook. The second half — that IPW addresses only measured confounders and can do nothing about unmeasured ones — is what the question is actually testing.
Part one — non-inferiority
Definition
Non-inferiority margin
The largest loss of efficacy that would still be clinically acceptable, in exchange for some other advantage — better safety, oral rather than intravenous administration, lower cost.
The margin must be prespecified and clinically justified, usually by preserving a defined fraction of the active comparator's established effect over placebo. A margin chosen after seeing the data is not a margin.
Non-inferiority is concluded when the confidence interval for the difference excludes the margin — not when p exceeds 0.05.
The margin must be prespecified and clinically justified, usually by preserving a defined fraction of the active comparator's established effect over placebo. A margin chosen after seeing the data is not a margin.
Non-inferiority is concluded when the confidence interval for the difference excludes the margin — not when p exceeds 0.05.
The inversion that catches people out
In a superiority trial, a non-significant result means you failed to show a difference.
In a non-inferiority trial, failing to show a difference is not success either. Absence of evidence is not evidence of non-inferiority. A small, underpowered, sloppily conducted trial will fail to detect a difference and that is not a result — it is a lack of one.
Non-inferiority is established only when the interval is narrow enough to exclude the margin. Sloppiness therefore works in favour of a non-inferiority conclusion if you judge it by p values, which is exactly why regulators insist on interval-based decisions and on per-protocol analyses as co-primary.
In a non-inferiority trial, failing to show a difference is not success either. Absence of evidence is not evidence of non-inferiority. A small, underpowered, sloppily conducted trial will fail to detect a difference and that is not a result — it is a lack of one.
Non-inferiority is established only when the interval is narrow enough to exclude the margin. Sloppiness therefore works in favour of a non-inferiority conclusion if you judge it by p values, which is exactly why regulators insist on interval-based decisions and on per-protocol analyses as co-primary.
Reading a non-inferiority result — margin set at HR 1.25
HR 1.02 (0.88–1.18)
Upper limit 1.18 is below 1.25. Non-inferiority demonstrated.
HR 1.04 (0.79–1.37)
Upper limit 1.37 exceeds 1.25. Non-inferiority not demonstrated — the trial was too imprecise, even though the point estimate looks fine.
HR 0.85 (0.71–0.98)
Excludes the margin and excludes 1. Non-inferiority demonstrated, and superiority may be claimed if prespecified in a testing sequence.
Row two is the one to internalise. The point estimate suggests the treatments are equivalent. The interval says the trial could not tell. Reporting row two as “non-inferior” is a serious error, and it is the error a wide interval invites.
Why per-protocol matters here
In a superiority trial, ITT is conservative — non-compliance dilutes any true difference toward the null.
In a non-inferiority trial that dilution pushes toward the desired conclusion. ITT is therefore anti-conservative, and regulators normally expect both ITT and per-protocol analyses to support non-inferiority. If only one does, that is a finding, not a footnote.
In a non-inferiority trial that dilution pushes toward the desired conclusion. ITT is therefore anti-conservative, and regulators normally expect both ITT and per-protocol analyses to support non-inferiority. If only one does, that is a finding, not a footnote.
Figure 4.1 — Reading a non-inferiority result. The decision rests entirely on where the upper limit falls, not on the point estimate and not on a p value. Row two has a point estimate closer to 1.0 than row three, and still fails: its interval reaches into the shaded zone, so an unacceptable loss of efficacy has not been excluded. That is a trial too imprecise to answer its own question — which is why judging non-inferiority by p values rewards sloppiness.
Part two — confounding in observational studies
Definition
Confounding by indication
In observational data, patients receive a treatment for a reason. Sicker patients may get the more aggressive drug; fitter patients may get the one with more toxicity.
Any raw comparison therefore mixes the effect of the drug with the effect of the reasons it was chosen. In a randomised trial this cannot happen — assignment is by chance and unrelated to prognosis. In real-world data it is the default state, and every method below exists to compensate for it.
Any raw comparison therefore mixes the effect of the drug with the effect of the reasons it was chosen. In a randomised trial this cannot happen — assignment is by chance and unrelated to prognosis. In real-world data it is the default state, and every method below exists to compensate for it.
| Method | How it works | Limitation |
|---|---|---|
| Multivariable regression | Include confounders as covariates in the outcome model | Assumes the model form is right; strains when there are many covariates and few events. |
| Propensity score matching | Estimate each patient's probability of receiving treatment given their covariates; match treated to untreated with similar scores | Discards unmatched patients, so the analysed population may no longer resemble the target population. |
| Propensity score stratification | Group patients into strata of similar score and compare within strata | Residual imbalance within strata. |
| IPW inverse probability of treatment weighting | Weight each patient by the inverse of their probability of receiving the treatment they actually got, creating a pseudo-population balanced on measured covariates | Patients with extreme scores get very large weights, destabilising the estimate. Usually handled by trimming or stabilised weights. |
| Instrumental variables | Exploit a factor that affects treatment but not outcome directly | Valid instruments are rare and the assumptions are untestable. |
The sentence that must appear in every write-up
Every method above balances measured confounders only. None of them can address unmeasured or unmeasurable confounding. Performance status, frailty, patient preference and physician judgement are frequently unrecorded and frequently decisive.
Required limitation wording: “Although IPW balanced measured baseline characteristics, residual confounding by unmeasured factors cannot be excluded.”
This is also the second half of the interview question. Candidates who explain the method but omit this are marked down, because the omission suggests they learned the technique without understanding what it is for.
Required limitation wording: “Although IPW balanced measured baseline characteristics, residual confounding by unmeasured factors cannot be excluded.”
This is also the second half of the interview question. Candidates who explain the method but omit this are marked down, because the omission suggests they learned the technique without understanding what it is for.
How balance is demonstrated
After weighting or matching, authors report the standardised mean difference (SMD) for each covariate, before and after. An SMD below 0.1 is the conventional threshold for adequate balance.
Unlike a p value, the SMD does not depend on sample size, which is why it is preferred here. If a paper reports p values to demonstrate balance after matching, that is a weakness worth noting.
Unlike a p value, the SMD does not depend on sample size, which is why it is preferred here. If a paper reports p values to demonstrate balance after matching, that is a weakness worth noting.
Drills
Drill 1 of 3
Non-inferiority margin is HR 1.25. Result: HR 1.04 (95% CI 0.79–1.37). What can be concluded?
ANon-inferiority is not demonstrated; the interval crosses the margin
BNon-inferiority is demonstrated, since HR is close to 1
CThe new treatment is inferior
DNon-inferiority is demonstrated, since p > 0.05
Drill 2 of 3
A real-world study uses IPW and reports all standardised mean differences below 0.05 after weighting. What limitation must still be stated?
AThe sample size was too small
BResidual confounding by unmeasured factors cannot be excluded
CThe covariates were not adequately balanced
DIPW is not an accepted method
Drill 3 of 3
Why does a non-inferiority trial normally require per-protocol as well as ITT analysis?
ATo increase statistical power
BBecause regulators require it, with no statistical rationale
CITT dilutes differences, which biases toward concluding non-inferiority
DTo handle safety reporting
Interpretation drill
Write the Methods and limitations text for this real-world study.
Study: retrospective cohort, 1,842 patients from an electronic health record database, comparing regimen A with regimen B for overall survival.
Analysis: propensity score estimated by logistic regression including age, sex, ECOG, stage, histology, prior lines, comorbidity index and year of treatment. Stabilised IPW applied; weights trimmed at the 1st and 99th percentiles. Balance assessed by standardised mean differences (all <0.10 after weighting). Weighted Cox model for the hazard ratio.
Result: HR 0.81 (95% CI 0.68–0.96).
Not captured in the database: performance status trajectory, smoking history, patient preference, biomarker status.
Analysis: propensity score estimated by logistic regression including age, sex, ECOG, stage, histology, prior lines, comorbidity index and year of treatment. Stabilised IPW applied; weights trimmed at the 1st and 99th percentiles. Balance assessed by standardised mean differences (all <0.10 after weighting). Weighted Cox model for the hazard ratio.
Result: HR 0.81 (95% CI 0.68–0.96).
Not captured in the database: performance status trajectory, smoking history, patient preference, biomarker status.
Methods:
“Propensity scores for receipt of regimen A were estimated using logistic regression including age, sex, ECOG performance status, disease stage, histology, number of prior lines of therapy, comorbidity index and year of treatment initiation. Stabilised inverse probability of treatment weights were derived from these scores and trimmed at the 1st and 99th percentiles to limit the influence of extreme weights. Covariate balance was assessed using standardised mean differences, with a value below 0.10 considered acceptable; all covariates met this criterion after weighting. Overall survival was compared using a weighted Cox proportional hazards model.”
Limitations:
“This analysis has several limitations inherent to its retrospective, database design. Although inverse probability weighting achieved balance across all measured baseline covariates, residual confounding by unmeasured factors cannot be excluded. In particular, performance status trajectory, smoking history, patient preference and biomarker status were not captured in the database, and each may plausibly have influenced both treatment selection and survival. The findings should therefore be regarded as hypothesis-generating and interpreted as an association rather than a causal effect.”
Six elements a reviewer checks for:
1. The covariates in the propensity model are listed, not summarised as “baseline characteristics”.
2. Stabilised weights and trimming are both named, with the trimming thresholds.
3. Balance is demonstrated by SMD with a stated threshold — not by p values.
4. The outcome model is named as weighted.
5. The unmeasured confounders are named specifically. A generic sentence about residual confounding is worth far less than naming the four variables that are actually absent.
6. The conclusion is framed as association, and the study is labelled hypothesis-generating.
The interview answer, in one sentence: IPW creates a pseudo-population in which measured confounders are balanced, so the comparison approximates what randomisation would have achieved on those variables — but it is powerless against anything the database did not record, which is why the limitation paragraph is not boilerplate.
“Propensity scores for receipt of regimen A were estimated using logistic regression including age, sex, ECOG performance status, disease stage, histology, number of prior lines of therapy, comorbidity index and year of treatment initiation. Stabilised inverse probability of treatment weights were derived from these scores and trimmed at the 1st and 99th percentiles to limit the influence of extreme weights. Covariate balance was assessed using standardised mean differences, with a value below 0.10 considered acceptable; all covariates met this criterion after weighting. Overall survival was compared using a weighted Cox proportional hazards model.”
Limitations:
“This analysis has several limitations inherent to its retrospective, database design. Although inverse probability weighting achieved balance across all measured baseline covariates, residual confounding by unmeasured factors cannot be excluded. In particular, performance status trajectory, smoking history, patient preference and biomarker status were not captured in the database, and each may plausibly have influenced both treatment selection and survival. The findings should therefore be regarded as hypothesis-generating and interpreted as an association rather than a causal effect.”
Six elements a reviewer checks for:
1. The covariates in the propensity model are listed, not summarised as “baseline characteristics”.
2. Stabilised weights and trimming are both named, with the trimming thresholds.
3. Balance is demonstrated by SMD with a stated threshold — not by p values.
4. The outcome model is named as weighted.
5. The unmeasured confounders are named specifically. A generic sentence about residual confounding is worth far less than naming the four variables that are actually absent.
6. The conclusion is framed as association, and the study is labelled hypothesis-generating.
The interview answer, in one sentence: IPW creates a pseudo-population in which measured confounders are balanced, so the comparison approximates what randomisation would have achieved on those variables — but it is powerless against anything the database did not record, which is why the limitation paragraph is not boilerplate.
Reflection
For a real-world study in your therapy area, list three clinically important variables the data source could not capture. That list is your limitations paragraph.
Being able to produce that list quickly is the difference between writing a limitations section and copying one.
Next module
4.2 — Subgroups, forest plots and interaction
Week 4 · Module 2
Subgroups, forest plots and interaction
The most over-interpreted figure in medicine — how to read one properly, why the interaction p value matters more than any individual row, and what you are allowed to write
Your scenario
A medical affairs colleague asks you to build a slide around one row of a forest plot: the subgroup aged under 65, HR 0.41. It looks compelling. The subgroup has 34 patients, the interval runs from 0.16 to 1.07, the interaction p value is 0.38, and there are fourteen other subgroups on the same plot. Your answer to that request is a test of whether you understand subgroup analysis.
Reading a forest plot
| Element | What it tells you, and what it does not |
|---|---|
| Square | The point estimate for that subgroup. Its size usually reflects the number of patients or events — a small square is a warning. |
| Horizontal line | The confidence interval. Width is the first thing to look at, before position. |
| Vertical line at 1 | No effect. Whether a subgroup interval crosses it is not the main question. |
| Diamond | The overall estimate. This is the reliable number on the figure. |
| Interaction p value | Whether the effect genuinely differs between subgroups. This is the only test that matters, and it is usually printed small at the right. |
| Number of subgroups | The multiplicity denominator. Fifteen subgroups at α = 0.05 means roughly a 54% chance of at least one spurious result. |
Figure 4.2 — A forest plot, read properly. The diamond is the reliable number on this figure. Square size reflects subgroup size, so the small square on the Asia row is a warning before you read its position. Asia shows the most striking estimate on the plot (HR 0.51) and its interaction p value is 0.19 — no evidence the effect differs by region, and not reportable as a differential effect. Only PD-L1 has a significant interaction (p = 0.02). Read the right-hand column before the middle of the figure.
Definition
Interaction test
A formal test of whether the treatment effect differs across levels of a subgroup variable.
Comparing whether one subgroup is significant and another is not is not a valid way to claim a difference between them. A subgroup with 34 patients will be non-significant almost regardless of the true effect, purely because it is small.
The interaction test asks the right question directly. Subgroup analyses are typically very underpowered for interaction — a trial powered for an overall effect has little power to detect a difference in that effect — so a non-significant interaction p value is the expected result and does not establish uniformity either.
Comparing whether one subgroup is significant and another is not is not a valid way to claim a difference between them. A subgroup with 34 patients will be non-significant almost regardless of the true effect, purely because it is small.
The interaction test asks the right question directly. Subgroup analyses are typically very underpowered for interaction — a trial powered for an overall effect has little power to detect a difference in that effect — so a non-significant interaction p value is the expected result and does not establish uniformity either.
The three failures
1. Reading rows instead of the interaction. Subgroup A significant, subgroup B not, therefore the drug works only in A. Invalid. Look at the interaction p value.
2. Ignoring multiplicity. Fifteen subgroups means several will look extreme by chance. The most extreme row on a forest plot is, by construction, the one most likely to be a chance finding.
3. Ignoring width. HR 0.41 in 34 patients with an interval from 0.16 to 1.07 is not a strong effect. It is an unstable estimate that happens to have landed low.
2. Ignoring multiplicity. Fifteen subgroups means several will look extreme by chance. The most extreme row on a forest plot is, by construction, the one most likely to be a chance finding.
3. Ignoring width. HR 0.41 in 34 patients with an interval from 0.16 to 1.07 is not a strong effect. It is an unstable estimate that happens to have landed low.
The credibility checklist
Before writing anything about a subgroup finding, ask:
1. Was it prespecified in the SAP, or found afterwards?
2. Was it one of few subgroups, or one of many?
3. Is the interaction test significant?
4. Is the direction consistent with the overall effect, or a reversal? Reversals are far less credible than differences in magnitude.
5. Is there a biological rationale that existed before the data?
6. Has it replicated in another trial?
A finding that satisfies one or two of these is hypothesis-generating. A finding that satisfies all six may be real — and that combination is rare.
1. Was it prespecified in the SAP, or found afterwards?
2. Was it one of few subgroups, or one of many?
3. Is the interaction test significant?
4. Is the direction consistent with the overall effect, or a reversal? Reversals are far less credible than differences in magnitude.
5. Is there a biological rationale that existed before the data?
6. Has it replicated in another trial?
A finding that satisfies one or two of these is hypothesis-generating. A finding that satisfies all six may be real — and that combination is rare.
Drills
Drill 1 of 3
Forest plot, 15 subgroups. Overall HR 0.72. One subgroup (n = 34) shows HR 0.41 (95% CI 0.16–1.07); interaction p = 0.38. What can you write?
A“The treatment is particularly effective in this subgroup.”
B“59% of younger patients benefited.”
COmit the subgroup analysis entirely
D“Treatment effect was consistent across subgroups; no significant interaction was observed (p = 0.38).”
Drill 2 of 3
Subgroup A: HR 0.62 (0.44–0.87), p = 0.006. Subgroup B: HR 0.79 (0.51–1.22), p = 0.29. Interaction p = 0.41. What is the correct interpretation?
AThere is no evidence the effect differs between A and B
BThe treatment works in A but not in B
CThe effect is significantly larger in A
DSubgroup B shows harm
Drill 3 of 3
Which subgroup finding is the most credible?
AA post hoc finding, one of 22 subgroups, reversing the overall direction of effect
BA prespecified finding, one of 4 subgroups, with significant interaction, prior biological rationale, and replication in a second trial
CThe subgroup with the largest effect size on the plot
DThe smallest subgroup, since its p value was lowest
Interpretation drill
Write the subgroup paragraph for the Results, and the corresponding Discussion sentence.
Overall: HR 0.72 (95% CI 0.59–0.88)
Prespecified subgroups (n = 6):
Age <65 (n = 210): HR 0.68 (0.51–0.91)
Age ≥65 (n = 190): HR 0.77 (0.57–1.04) — interaction p = 0.58
PD-L1 <1% (n = 148): HR 0.94 (0.66–1.34)
PD-L1 ≥1% (n = 252): HR 0.58 (0.44–0.77) — interaction p = 0.02
Region Asia (n = 96): HR 0.51 (0.29–0.90)
Region non-Asia (n = 304): HR 0.79 (0.63–0.99) — interaction p = 0.14
Prespecified subgroups (n = 6):
Age <65 (n = 210): HR 0.68 (0.51–0.91)
Age ≥65 (n = 190): HR 0.77 (0.57–1.04) — interaction p = 0.58
PD-L1 <1% (n = 148): HR 0.94 (0.66–1.34)
PD-L1 ≥1% (n = 252): HR 0.58 (0.44–0.77) — interaction p = 0.02
Region Asia (n = 96): HR 0.51 (0.29–0.90)
Region non-Asia (n = 304): HR 0.79 (0.63–0.99) — interaction p = 0.14
Results:
“Treatment effect was examined in six prespecified subgroups. The benefit was generally consistent by age (interaction p = 0.58) and by region (interaction p = 0.14). A significant interaction was observed by PD-L1 expression (interaction p = 0.02): the hazard ratio was 0.58 (95% CI 0.44–0.77) in patients with PD-L1 expression of at least 1% and 0.94 (95% CI 0.66–1.34) in those below 1%.”
Discussion:
“The observed interaction by PD-L1 expression is consistent with the established role of PD-L1 as a predictive biomarker for checkpoint inhibition and was prespecified; nevertheless, the trial was not powered for subgroup comparisons, and this finding requires prospective confirmation before it can inform patient selection.”
The four judgements behind this:
1. Age and region are reported as consistent, with their interaction p values, and no individual rows are highlighted. The Asia subgroup at HR 0.51 looks striking and has an interaction p of 0.14 — it is not reportable as a differential effect.
2. PD-L1 is treated differently, because the interaction test is significant at 0.02. This is the one row that earns its own sentence.
3. Even so, the Discussion hedges it: prespecified and biologically plausible, but underpowered and unconfirmed. Two of six credibility criteria are not met.
4. Six subgroups, not fifteen. That matters and is worth stating, because it bears directly on how much multiplicity concern applies.
The trap in this drill: the Asia subgroup. HR 0.51 is the most impressive number on the list, it is nominally significant, and a medical affairs colleague will want a slide about it. The interaction p of 0.14 is the answer, and being able to say that clearly — and kindly — is a large part of the job.
“Treatment effect was examined in six prespecified subgroups. The benefit was generally consistent by age (interaction p = 0.58) and by region (interaction p = 0.14). A significant interaction was observed by PD-L1 expression (interaction p = 0.02): the hazard ratio was 0.58 (95% CI 0.44–0.77) in patients with PD-L1 expression of at least 1% and 0.94 (95% CI 0.66–1.34) in those below 1%.”
Discussion:
“The observed interaction by PD-L1 expression is consistent with the established role of PD-L1 as a predictive biomarker for checkpoint inhibition and was prespecified; nevertheless, the trial was not powered for subgroup comparisons, and this finding requires prospective confirmation before it can inform patient selection.”
The four judgements behind this:
1. Age and region are reported as consistent, with their interaction p values, and no individual rows are highlighted. The Asia subgroup at HR 0.51 looks striking and has an interaction p of 0.14 — it is not reportable as a differential effect.
2. PD-L1 is treated differently, because the interaction test is significant at 0.02. This is the one row that earns its own sentence.
3. Even so, the Discussion hedges it: prespecified and biologically plausible, but underpowered and unconfirmed. Two of six credibility criteria are not met.
4. Six subgroups, not fifteen. That matters and is worth stating, because it bears directly on how much multiplicity concern applies.
The trap in this drill: the Asia subgroup. HR 0.51 is the most impressive number on the list, it is nominally significant, and a medical affairs colleague will want a slide about it. The interaction p of 0.14 is the answer, and being able to say that clearly — and kindly — is a large part of the job.
Reflection
Find a forest plot in a recent paper. How many subgroups? What are the interaction p values? Does the abstract mention a subgroup, and if so was its interaction test significant?
Abstracts frequently highlight a subgroup whose interaction test was not significant. Once you start checking, you will see it often.
Next module
4.3 — Reading tables, figures and listings
Week 4 · Module 3
Reading tables, figures and listings
The TFL pack is where your Results section comes from. Knowing how to navigate it, what to cross-check, and which discrepancies to query is the practical core of the job
Your scenario
You are handed a 900-page TFL pack and asked for a draft Results section in four days. Nobody explains the numbering, nobody tells you which tables matter, and the statistician is on another project. What you do in the first two hours determines whether the draft is right.
How a TFL pack is organised
Numbering follows ICH E3 section order, so the pack maps onto the CSR itself. Once you know the mapping you can find anything.
| Series | Content | What you use it for |
|---|---|---|
| 14.1.x | Disposition, demographics, baseline characteristics, protocol deviations | Analysis population denominators; baseline paragraph; the disposition figure |
| 14.2.x | Efficacy — primary, secondary, subgroups, sensitivity analyses | The efficacy results, in SAP order |
| 14.3.x | Safety — exposure, TEAEs by SOC and PT, deaths, SAEs, discontinuations, labs, vitals | The safety results |
| 16.2.x | Patient-level listings — individual data | Narratives, checking individual cases, resolving a discrepancy |
Your first two hours
1. Find the disposition table (usually 14.1.1) and write down every population n. Everything downstream uses these.
2. Open the SAP and find the estimand, the testing hierarchy and the analysis population definitions. One page, and it governs every claim you can make.
3. Locate the primary endpoint table and the primary figure. Read them together.
4. Build a one-page number sheet: every figure you will quote, with its table number beside it.
That number sheet is what makes your draft checkable — by you, by QC, and by the statistician — and it is what stops you re-hunting the same figure four times.
2. Open the SAP and find the estimand, the testing hierarchy and the analysis population definitions. One page, and it governs every claim you can make.
3. Locate the primary endpoint table and the primary figure. Read them together.
4. Build a one-page number sheet: every figure you will quote, with its table number beside it.
That number sheet is what makes your draft checkable — by you, by QC, and by the statistician — and it is what stops you re-hunting the same figure four times.
The cross-checks
Run these on every pack. They catch most of what QC would otherwise send back, and occasionally they catch a real error in the outputs.
| Check | What a mismatch means |
|---|---|
| Do the arm totals in each table match the disposition table? | Wrong analysis population applied, or a table from an earlier data cut. |
| Does the safety denominator equal the treated population, not the randomised one? | The Module 3.2 error. Very common in first drafts. |
| Do the medians in the table match the point where the KM curve crosses 0.5? | Figure and table may come from different runs or different cut-off dates. |
| Do subgroup n values sum to the overall n? | Missing biomarker or covariate data — which needs stating. |
| Does the number of events match between the primary table and the KM figure? | Different censoring rules applied. |
| Does the abstract number match the table you cited? | The most common late-stage error, usually caused by a data refresh after the abstract was drafted. |
| Do percentages recompute from their counts? | Transcription error, or a different denominator than the column header states. |
When numbers disagree
Never resolve a discrepancy yourself. Do not pick the number that looks right, do not recalculate and substitute, and do not quietly use the one that supports the narrative.
Raise it, in writing, with the table numbers and the two conflicting values. Nine times out of ten there is an explanation — a different data cut, a different population, a footnote you missed. The tenth time you have found a real error, and you will have found it before a regulator did.
Writers who query well become the writers statisticians want to work with. This is the single fastest route to being trusted with more senior work.
Raise it, in writing, with the table numbers and the two conflicting values. Nine times out of ten there is an explanation — a different data cut, a different population, a footnote you missed. The tenth time you have found a real error, and you will have found it before a regulator did.
Writers who query well become the writers statisticians want to work with. This is the single fastest route to being trusted with more senior work.
AI-Assisted TFL Extraction — The Medical Writer's Verification Guardrail
Many modern medical writing teams use specialized AI tools or OCR platforms to extract tables from 900-page SAS outputs into Word drafts. While this speeds up drafting, large language models and extraction tools suffer from three predictable failure modes:
Privacy first: TFLs and SAS output are confidential trial data. Use only tools your sponsor or employer has approved for that data, and never paste unpublished or patient-level results into a public AI tool. If you are unsure whether a tool is approved, ask before you upload.
- Column Header Inversion: Models frequently swap the Experimental and Placebo columns when table spacing is tight, reporting the active arm's results under control.
- Denominator Dropping: A model may extract "Grade 3 AEs: 24 (12.0%)" from Table 14.3.1 without verifying whether the denominator was the Randomized Set ($N = 220$) or the Safety Set ($N = 200$).
- Decimal Shift & P-Value Rounding: Extracting $p = 0.048$ as $p < 0.001$ or dropping leading zeros (0.05 → .05).
Privacy first: TFLs and SAS output are confidential trial data. Use only tools your sponsor or employer has approved for that data, and never paste unpublished or patient-level results into a public AI tool. If you are unsure whether a tool is approved, ask before you upload.
Drills
Drill 1 of 3
The primary efficacy table shows n = 240; the safety table shows n = 233. What should you do?
AQuery it as an inconsistency
BUse 240 throughout for consistency
CNothing — these are different populations, and both are correct
DUse 233 throughout for consistency
Drill 2 of 3
Table 14.2.1 gives median PFS as 9.4 months. The Kaplan–Meier figure appears to cross 0.5 at about 8.6 months. What do you do?
AUse the table figure — tables are more precise than reading a graph
BAverage the two values
CUse the figure, since readers will see the curve
DCheck the data cut-off and population in the footers of both; query if they match
Drill 3 of 3
Subgroup n values by PD-L1 status sum to 400, but the ITT population is 428. What does this mean and what do you write?
A28 patients lacked evaluable PD-L1 data; report the biomarker-evaluable population explicitly
BThe table is wrong; query it as an error
CAssign the 28 patients to the larger subgroup
DUse 428 as the denominator for both subgroups
Interpretation drill
You have been handed this extract. List every query you would raise, with table references, before drafting.
Table 14.1.1 Disposition: 428 randomised (214/214). 419 treated (211/208). 396 with ≥1 post-baseline assessment.
Table 14.2.1 Primary PFS (ITT): n = 428. Median 9.4 vs 6.1 months. HR 0.68 (95% CI 0.55–0.84). Events 142/198. Footer: data cut-off 15 Jan 2026.
Figure 14.2.2 KM curve for PFS: curve crosses 0.5 at approximately 8.6 months in the treatment arm. Footer: data cut-off 30 Sep 2025.
Table 14.3.1 TEAEs: n = 428 (214/214). Any grade 391 (91.4%). Grade ≥3 168 (39.3%).
Table 14.2.6 Subgroups by PD-L1: <1% n = 148; ≥1% n = 252.
Table 14.2.1 Primary PFS (ITT): n = 428. Median 9.4 vs 6.1 months. HR 0.68 (95% CI 0.55–0.84). Events 142/198. Footer: data cut-off 15 Jan 2026.
Figure 14.2.2 KM curve for PFS: curve crosses 0.5 at approximately 8.6 months in the treatment arm. Footer: data cut-off 30 Sep 2025.
Table 14.3.1 TEAEs: n = 428 (214/214). Any grade 391 (91.4%). Grade ≥3 168 (39.3%).
Table 14.2.6 Subgroups by PD-L1: <1% n = 148; ≥1% n = 252.
Four queries, in order of seriousness:
1. Table 14.3.1 — wrong safety denominator. The TEAE table uses n = 428 (214/214), which is the randomised population. Safety must use the treated population: 419 (211/208). Every percentage in that table is therefore understated. This is the most serious of the four and it changes published numbers.
2. Figure 14.2.2 versus Table 14.2.1 — different data cut-offs. The table is cut at 15 January 2026 and the figure at 30 September 2025. That fully explains the 9.4 versus 8.6 month discrepancy. Request a refreshed figure at the current cut. Do not publish a figure and a table from different cuts.
3. Table 14.2.1 — events denominator unclear. Events are given as 142/198, but neither number corresponds to an arm size of 214. Ask whether these are events per arm (142 and 198) rather than a fraction, and confirm the intended presentation.
4. Table 14.2.6 — subgroup n values sum to 400, not 428. Twenty-eight patients presumably had no evaluable PD-L1 result. Confirm this and obtain the wording for the biomarker-evaluable population, which must be stated in Methods and in the subgroup results.
What you have also established, without querying: the 396 patients with a post-baseline assessment is the likely mITT denominator for response endpoints. Check the SAP definition before using it.
Score yourself: query 1 is the one that matters most — it is a real error that would reach publication. If you found queries 2 and 4 but not 1, re-read Module 3.2. If you found all four, you are working at the level agencies hire for.
1. Table 14.3.1 — wrong safety denominator. The TEAE table uses n = 428 (214/214), which is the randomised population. Safety must use the treated population: 419 (211/208). Every percentage in that table is therefore understated. This is the most serious of the four and it changes published numbers.
2. Figure 14.2.2 versus Table 14.2.1 — different data cut-offs. The table is cut at 15 January 2026 and the figure at 30 September 2025. That fully explains the 9.4 versus 8.6 month discrepancy. Request a refreshed figure at the current cut. Do not publish a figure and a table from different cuts.
3. Table 14.2.1 — events denominator unclear. Events are given as 142/198, but neither number corresponds to an arm size of 214. Ask whether these are events per arm (142 and 198) rather than a fraction, and confirm the intended presentation.
4. Table 14.2.6 — subgroup n values sum to 400, not 428. Twenty-eight patients presumably had no evaluable PD-L1 result. Confirm this and obtain the wording for the biomarker-evaluable population, which must be stated in Methods and in the subgroup results.
What you have also established, without querying: the 396 patients with a post-baseline assessment is the likely mITT denominator for response endpoints. Check the SAP definition before using it.
Score yourself: query 1 is the one that matters most — it is a real error that would reach publication. If you found queries 2 and 4 but not 1, re-read Module 3.2. If you found all four, you are working at the level agencies hire for.
Reflection
Next time you receive a TFL pack, build the one-page number sheet before writing anything. How long did it take, and how many numbers did you have to re-find later?
Most writers report the sheet takes ninety minutes and saves considerably more than that across a draft cycle.
Next module
4.4 — Capstone and exit assessment
Week 4 · Module 4
Capstone and exit assessment
A full output pack, a Results section to write, and twenty seeded statistical errors in a draft to catch — the task an agency assessment actually sets
Your scenario
This is the real test: not whether you can define a hazard ratio, but whether you catch an overclaim in someone else's draft at four in the afternoon on a deadline. The pass mark is 16 of 20.
How to do this properly
Set a timer for 50 minutes. Read the output pack, then read the draft, then write your list of findings — each with the correction and the reason. Only then reveal the rubric.
Reading the rubric first turns this into a comprehension exercise and measures nothing.
Reading the rubric first turns this into a comprehension exercise and measures nothing.
The output pack
Read this first. Every number in the draft below should be checkable against it.
Design: randomised, double-blind, placebo-controlled Phase III. Stratified by region and prior lines. SAP hierarchy: PFS → OS → ORR → PRO. Estimand: treatment policy for PFS.
Disposition: 402 randomised (201/201). 389 treated (196/193). 371 with ≥1 post-baseline scan.
PFS (ITT, n = 402): median 8.9 (95% CI 7.4–10.3) vs 5.2 (95% CI 4.4–6.1) months. HR 0.66 (95% CI 0.53–0.82); stratified log-rank p < 0.001. Schoenfeld PH test p = 0.44.
OS (ITT): median not reached vs 21.7 months. HR 0.84 (95% CI 0.62–1.14); p = 0.26. At risk at 24 months: 41 vs 22.
ORR (mITT, n = 371): 44% vs 29%; p = 0.003.
PRO change: −5.2 vs −1.8; p = 0.02. Published MCID for the scale = 8 points.
Safety (n = 389): grade ≥3 TEAE 147 (37.8%). Discontinuation for AE 31 (8.0%).
Subgroups (6 prespecified): region Asia n = 88, HR 0.47 (0.26–0.85); non-Asia n = 314, HR 0.72 (0.56–0.93); interaction p = 0.19.
Median follow-up: 19.8 months (IQR 16.2–24.1).
Disposition: 402 randomised (201/201). 389 treated (196/193). 371 with ≥1 post-baseline scan.
PFS (ITT, n = 402): median 8.9 (95% CI 7.4–10.3) vs 5.2 (95% CI 4.4–6.1) months. HR 0.66 (95% CI 0.53–0.82); stratified log-rank p < 0.001. Schoenfeld PH test p = 0.44.
OS (ITT): median not reached vs 21.7 months. HR 0.84 (95% CI 0.62–1.14); p = 0.26. At risk at 24 months: 41 vs 22.
ORR (mITT, n = 371): 44% vs 29%; p = 0.003.
PRO change: −5.2 vs −1.8; p = 0.02. Published MCID for the scale = 8 points.
Safety (n = 389): grade ≥3 TEAE 147 (37.8%). Discontinuation for AE 31 (8.0%).
Subgroups (6 prespecified): region Asia n = 88, HR 0.47 (0.26–0.85); non-Asia n = 314, HR 0.72 (0.56–0.93); interaction p = 0.19.
Median follow-up: 19.8 months (IQR 16.2–24.1).
The draft to review — 20 seeded errors
List every finding with its correction and reason. Write your answers before revealing.
“This randomised trial proves that the study regimen definitely improves outcomes in advanced disease. Median PFS was 8.9 versus 5 months (HR .66, CI 0.53-0.82, p=0.000), meaning survival was 34% longer with the study regimen. The primary endpoint was highly significant.
Overall survival showed no benefit (HR 0.84, p = 0.26); median OS was not reached in the treatment arm, indicating the data are missing for that arm. Objective response rate was significantly higher (44% vs 29%, p = 0.003), as was quality of life, which improved significantly (p = 0.02), demonstrating a clinically meaningful benefit to patients.
Grade 3 or higher adverse events occurred in 36.6% of the 402 randomised patients, and 31 patients discontinued.
In subgroup analysis the treatment was particularly effective in Asian patients (HR 0.47, p = 0.01), suggesting that ethnicity predicts response and that this population should be prioritised. A total of 402 patients were randomised and the mITT population of 371 was used for all efficacy analyses.”
Overall survival showed no benefit (HR 0.84, p = 0.26); median OS was not reached in the treatment arm, indicating the data are missing for that arm. Objective response rate was significantly higher (44% vs 29%, p = 0.003), as was quality of life, which improved significantly (p = 0.02), demonstrating a clinically meaningful benefit to patients.
Grade 3 or higher adverse events occurred in 36.6% of the 402 randomised patients, and 31 patients discontinued.
In subgroup analysis the treatment was particularly effective in Asian patients (HR 0.47, p = 0.01), suggesting that ethnicity predicts response and that this population should be prioritised. A total of 402 patients were randomised and the mITT population of 371 was used for all efficacy analyses.”
The twenty findings
Claim calibration (5)
1. proves — no single trial proves anything. Replace with demonstrated, which the randomised design does license.
2. definitely improves outcomes — definitely has no place in a scientific claim, and outcomes is vague. Name the endpoint.
3. highly significant — attaches magnitude to a p value. Give the effect size instead.
4. OS showed no benefit — HR 0.84 with an interval of 0.62–1.14 is an immature, imprecise estimate, not a demonstration of no benefit. Write overall survival data were immature; the confidence interval does not exclude a benefit.
5. demonstrating a clinically meaningful benefit for the PRO — the change is −5.2 versus −1.8, a difference of 3.4 points against a published MCID of 8. It is statistically significant and below the threshold of clinical importance. This must be stated.
Statistical interpretation (6)
6. survival was 34% longer — a hazard ratio is not a percentage increase in survival time. Write a 34% reduction in the risk of progression or death.
7. median OS not reached, indicating the data are missing — the reverse. Fewer than half the patients had died; report as not reached and note immaturity.
8. ORR described as significantly higher — the hierarchy broke at OS (p = 0.26), so ORR was not formally tested. Report as nominal p = 0.003, descriptive.
9. PRO described as significant — same hierarchy problem, two steps further down. Nominal only.
10. subgroup: particularly effective in Asian patients — the interaction p is 0.19. There is no evidence the effect differs by region.
11. ethnicity predicts response / should be prioritised — a treatment recommendation from a non-significant interaction in an 88-patient subgroup. This is the most serious finding in the draft.
Populations and denominators (3)
12. 36.6% of the 402 randomised patients — safety uses the treated population, n = 389. And 147/389 is 37.8%, not 36.6%; the draft has computed against the wrong denominator.
13. 31 patients discontinued — no denominator and no percentage. Write 31/389 (8.0%).
14. mITT used for all efficacy analyses — false. PFS and OS used ITT (n = 402); only ORR used mITT (n = 371).
Reporting conventions (6)
15. 5 months → 5.2 months, matching the decimal precision of 8.9.
16. .66 → 0.66, leading zero.
17. CI → 95% CI.
18. Hyphen in 0.53-0.82 → en dash.
19. p=0.000 → p < 0.001, italic, with spaces.
20. Median follow-up omitted entirely — survival results are uninterpretable without it. Add at a median follow-up of 19.8 months (IQR 16.2–24.1).
Model corrected version
“At a median follow-up of 19.8 months (IQR 16.2–24.1), the study regimen demonstrated a benefit in progression-free survival: median PFS was 8.9 months (95% CI 7.4–10.3) versus 5.2 months (95% CI 4.4–6.1), corresponding to a 34% reduction in the risk of progression or death (HR 0.66, 95% CI 0.53–0.82; stratified log-rank p < 0.001). The proportional hazards assumption was supported (Schoenfeld p = 0.44).
Overall survival data were immature: median OS was not reached with the study regimen and was 21.7 months with placebo (HR 0.84, 95% CI 0.62–1.14; p = 0.26), with 41 and 22 patients respectively at risk at 24 months. As overall survival did not meet the prespecified threshold, formal testing stopped at this endpoint; subsequent endpoints are reported descriptively. Objective response rate was 44% versus 29% (nominal p = 0.003) in the modified intention-to-treat population (n = 371). The mean change in patient-reported outcome score was −5.2 versus −1.8 (nominal p = 0.02); the between-group difference of 3.4 points is below the published minimal clinically important difference of 8 points.
In the safety population (n = 389), grade 3 or higher treatment-emergent adverse events occurred in 147 patients (37.8%) and 31 patients (8.0%) discontinued because of an adverse event.
Treatment effect was consistent across the six prespecified subgroups, with no significant interaction by region (interaction p = 0.19).”
Scoring
16–20 — pass. You can be trusted to review statistical text unsupervised, and you would have caught findings 5, 11 and 12 before they reached a regulator.
11–15 — close. Note which of the four categories your misses fell in and re-do that week's drills.
Under 11 — repeat the sentence-level drills in Weeks 2 and 3 before retrying. Do not retake this immediately.
The three that matter most: finding 5 (statistical significance below the MCID), finding 11 (a treatment recommendation from a non-significant interaction), and finding 12 (the wrong safety denominator). Each of those reaches patients or regulators if nobody catches it. The reporting conventions matter for QC; those three matter for people.
Claim calibration (5)
1. proves — no single trial proves anything. Replace with demonstrated, which the randomised design does license.
2. definitely improves outcomes — definitely has no place in a scientific claim, and outcomes is vague. Name the endpoint.
3. highly significant — attaches magnitude to a p value. Give the effect size instead.
4. OS showed no benefit — HR 0.84 with an interval of 0.62–1.14 is an immature, imprecise estimate, not a demonstration of no benefit. Write overall survival data were immature; the confidence interval does not exclude a benefit.
5. demonstrating a clinically meaningful benefit for the PRO — the change is −5.2 versus −1.8, a difference of 3.4 points against a published MCID of 8. It is statistically significant and below the threshold of clinical importance. This must be stated.
Statistical interpretation (6)
6. survival was 34% longer — a hazard ratio is not a percentage increase in survival time. Write a 34% reduction in the risk of progression or death.
7. median OS not reached, indicating the data are missing — the reverse. Fewer than half the patients had died; report as not reached and note immaturity.
8. ORR described as significantly higher — the hierarchy broke at OS (p = 0.26), so ORR was not formally tested. Report as nominal p = 0.003, descriptive.
9. PRO described as significant — same hierarchy problem, two steps further down. Nominal only.
10. subgroup: particularly effective in Asian patients — the interaction p is 0.19. There is no evidence the effect differs by region.
11. ethnicity predicts response / should be prioritised — a treatment recommendation from a non-significant interaction in an 88-patient subgroup. This is the most serious finding in the draft.
Populations and denominators (3)
12. 36.6% of the 402 randomised patients — safety uses the treated population, n = 389. And 147/389 is 37.8%, not 36.6%; the draft has computed against the wrong denominator.
13. 31 patients discontinued — no denominator and no percentage. Write 31/389 (8.0%).
14. mITT used for all efficacy analyses — false. PFS and OS used ITT (n = 402); only ORR used mITT (n = 371).
Reporting conventions (6)
15. 5 months → 5.2 months, matching the decimal precision of 8.9.
16. .66 → 0.66, leading zero.
17. CI → 95% CI.
18. Hyphen in 0.53-0.82 → en dash.
19. p=0.000 → p < 0.001, italic, with spaces.
20. Median follow-up omitted entirely — survival results are uninterpretable without it. Add at a median follow-up of 19.8 months (IQR 16.2–24.1).
Model corrected version
“At a median follow-up of 19.8 months (IQR 16.2–24.1), the study regimen demonstrated a benefit in progression-free survival: median PFS was 8.9 months (95% CI 7.4–10.3) versus 5.2 months (95% CI 4.4–6.1), corresponding to a 34% reduction in the risk of progression or death (HR 0.66, 95% CI 0.53–0.82; stratified log-rank p < 0.001). The proportional hazards assumption was supported (Schoenfeld p = 0.44).
Overall survival data were immature: median OS was not reached with the study regimen and was 21.7 months with placebo (HR 0.84, 95% CI 0.62–1.14; p = 0.26), with 41 and 22 patients respectively at risk at 24 months. As overall survival did not meet the prespecified threshold, formal testing stopped at this endpoint; subsequent endpoints are reported descriptively. Objective response rate was 44% versus 29% (nominal p = 0.003) in the modified intention-to-treat population (n = 371). The mean change in patient-reported outcome score was −5.2 versus −1.8 (nominal p = 0.02); the between-group difference of 3.4 points is below the published minimal clinically important difference of 8 points.
In the safety population (n = 389), grade 3 or higher treatment-emergent adverse events occurred in 147 patients (37.8%) and 31 patients (8.0%) discontinued because of an adverse event.
Treatment effect was consistent across the six prespecified subgroups, with no significant interaction by region (interaction p = 0.19).”
Scoring
16–20 — pass. You can be trusted to review statistical text unsupervised, and you would have caught findings 5, 11 and 12 before they reached a regulator.
11–15 — close. Note which of the four categories your misses fell in and re-do that week's drills.
Under 11 — repeat the sentence-level drills in Weeks 2 and 3 before retrying. Do not retake this immediately.
The three that matter most: finding 5 (statistical significance below the MCID), finding 11 (a treatment recommendation from a non-significant interaction), and finding 12 (the wrong safety denominator). Each of those reaches patients or regulators if nobody catches it. The reporting conventions matter for QC; those three matter for people.
Retake the diagnostic
Return to Module 1.1 and take the 24-item diagnostic again. Your first score was stored as the baseline; the difference is the only honest measure of what these four weeks changed.
Expect movement of one band. Band C to Band B is a normal and substantial four-week result.
What to do next
Build the habit, not the knowledge. On your next project, do three things: ask for the SAP before you draft; build the one-page number sheet from Module 4.3; and check every safety denominator against the treated population. Those three habits catch most of what this course covered.
Then pair this with the writing. The Scientific English course fixes how you write sentences; this course fixes what you are allowed to claim in them. Module 3.2 there and Modules 1.4 and 4.2 here are the same skill approached from two directions.
Then pair this with the writing. The Scientific English course fixes how you write sentences; this course fixes what you are allowed to claim in them. Module 3.2 there and Modules 1.4 and 4.2 here are the same skill approached from two directions.
Final reflection
Your diagnostic score at the start, and now. Which module changed how you read output most? What still feels uncertain?
Write it down. In six months this is the only record you will have of what you actually fixed.
Next module
4.5 — HEOR preview: from clinical to economic evidence
Week 4 · Bridge Module
HEOR preview: from clinical to economic evidence
The statistical skills you have built read clinical data — but payers, HTA bodies, and formulary committees read economic data too. This bridge module shows you where the two meet.
Why this matters for you
You have spent four weeks learning to catch a wrong denominator, a broken testing hierarchy, or an overclaim. In health economics, the same rigour applies — but the question changes from "Does it work?" to "Is it worth it?" The analysts who answer that question use the same trial data you have just learned to read.
From efficacy to value
A positive Phase III trial proves a treatment works. But a payer's question is different: how much additional health does it buy, and at what cost? This is the domain of Health Economics and Outcomes Research (HEOR).
HEOR writers take the clinical evidence you now understand — hazard ratios, response rates, safety profiles, quality-of-life scores — and translate it into economic models that inform pricing, reimbursement, and formulary decisions. If you have completed this statistics course, you already understand the inputs. This module shows you what happens to them next.
The core HEOR metrics
QALY (Quality-Adjusted Life Year): one year of perfect health = 1.0 QALY. A year lived at a utility of 0.7 (moderate disability) = 0.7 QALYs. QALYs combine survival time with quality of life into one number.
ICER (Incremental Cost-Effectiveness Ratio): the additional cost of the new treatment divided by the additional QALYs it produces versus the comparator. An ICER of £30,000/QALY means each extra year of perfect health costs £30,000.
NNT in economic context: you learned NNT as "how many patients must be treated for one to benefit." In HEOR, NNT feeds directly into budget impact: if NNT = 8 and the treatment costs £40,000 per patient per year, the cost per responder is £320,000.
ICER (Incremental Cost-Effectiveness Ratio): the additional cost of the new treatment divided by the additional QALYs it produces versus the comparator. An ICER of £30,000/QALY means each extra year of perfect health costs £30,000.
NNT in economic context: you learned NNT as "how many patients must be treated for one to benefit." In HEOR, NNT feeds directly into budget impact: if NNT = 8 and the treatment costs £40,000 per patient per year, the cost per responder is £320,000.
How clinical statistics feed economic models
Every number you have learned to check in this course has a second life in a cost-effectiveness model:
Clinical input → Economic output
Kaplan–Meier curves → extrapolated beyond trial follow-up to estimate lifetime survival, using parametric distributions (Weibull, log-normal, generalised gamma). The shape of the tail drives the model more than the median.
Hazard ratios → applied to baseline survival curves to estimate the treatment arm's long-term trajectory. If the proportional hazards assumption fails (you tested this in Module 3.4), the model must use a different approach.
Utility values from PRO data → mapped to QALY weights. The EQ-5D is the instrument most HTA bodies accept. The PRO change of −5.2 versus −1.8 from the capstone? An economist would ask whether that maps to a utility difference, and whether that difference exceeds the minimum important difference — exactly the judgement you made in finding 5 of the capstone.
Safety rates → costed as adverse-event management. Grade ≥3 events carry hospital costs; discontinuation rates affect duration of treatment and therefore total drug cost.
Hazard ratios → applied to baseline survival curves to estimate the treatment arm's long-term trajectory. If the proportional hazards assumption fails (you tested this in Module 3.4), the model must use a different approach.
Utility values from PRO data → mapped to QALY weights. The EQ-5D is the instrument most HTA bodies accept. The PRO change of −5.2 versus −1.8 from the capstone? An economist would ask whether that maps to a utility difference, and whether that difference exceeds the minimum important difference — exactly the judgement you made in finding 5 of the capstone.
Safety rates → costed as adverse-event management. Grade ≥3 events carry hospital costs; discontinuation rates affect duration of treatment and therefore total drug cost.
Quiz
A treatment extends median overall survival by 4.2 months (from 14.0 to 18.2 months) with mean utility of 0.72 during the extra time. Approximately how many incremental QALYs does this represent?
4.2 QALYs
0.25 QALYs
0.35 QALYs
0.72 QALYs
Willingness-to-pay thresholds
An ICER is meaningless without a threshold. Different countries set different ceilings for what they will pay per QALY gained:
Key thresholds
NICE (England): generally £20,000–£30,000 per QALY. End-of-life and highly specialised technologies can go higher.
WHO benchmark: historically 1–3× GDP per capita, though this rule of thumb is increasingly debated.
US: no single threshold; ICER (the Institute) uses $50,000–$200,000 per QALY as a range. Payers negotiate individually.
Practical rule: if the ICER exceeds the threshold, the treatment is "not cost-effective" at its current price — which usually means the manufacturer must lower the price, offer a discount, or propose a managed-access scheme.
WHO benchmark: historically 1–3× GDP per capita, though this rule of thumb is increasingly debated.
US: no single threshold; ICER (the Institute) uses $50,000–$200,000 per QALY as a range. Payers negotiate individually.
Practical rule: if the ICER exceeds the threshold, the treatment is "not cost-effective" at its current price — which usually means the manufacturer must lower the price, offer a discount, or propose a managed-access scheme.
Quiz
A new oncology drug costs £82,000 more than standard of care over a patient's lifetime and produces 1.8 additional QALYs. What is the ICER, and would NICE likely consider it cost-effective at standard thresholds?
£82,000/QALY — above threshold
£45,556/QALY — above standard threshold
£14,909/QALY — within threshold
Budget impact versus cost-effectiveness
Cost-effectiveness asks "is it worth it per patient?" Budget impact asks "can we afford it across the whole eligible population?" Both matter, and they can point in opposite directions. A treatment can be cost-effective (good ICER) but have enormous budget impact because the eligible population is large.
As a medical writer, you may never build the model — but you will write the value dossier, the AMCP dossier, or the NICE submission that presents its results. Understanding what the numbers mean, where they come from, and how they connect to the clinical evidence is what separates a writer who transcribes from one who catches errors.
Drill — connecting clinical to economic
Below is a simplified scenario. Calculate the ICER and identify which clinical input is most uncertain.
New drug vs. standard of care:
Incremental cost: $124,000 per patient over 5 years
PFS HR: 0.58 (95% CI 0.42–0.80)
OS HR: 0.79 (95% CI 0.58–1.08)
Mean utility on treatment: 0.68
Incremental life-years gained (modelled): 1.4 years
Incremental QALYs (modelled): 0.95
Calculate: the ICER. Then state which clinical input carries the most uncertainty and why that matters for the economic conclusion.
Incremental cost: $124,000 per patient over 5 years
PFS HR: 0.58 (95% CI 0.42–0.80)
OS HR: 0.79 (95% CI 0.58–1.08)
Mean utility on treatment: 0.68
Incremental life-years gained (modelled): 1.4 years
Incremental QALYs (modelled): 0.95
Calculate: the ICER. Then state which clinical input carries the most uncertainty and why that matters for the economic conclusion.
ICER = $124,000 ÷ 0.95 = $130,526/QALY.
The most uncertain input is the OS hazard ratio — its confidence interval crosses 1.0 (0.58–1.08), meaning the survival benefit is not statistically established. The modelled 1.4 incremental life-years depend heavily on extrapolation beyond the trial, and the choice of survival curve (Weibull vs. log-normal vs. generalised gamma) will swing the ICER by tens of thousands of dollars.
This is exactly the kind of uncertainty an HTA committee will probe, and exactly the kind a medical writer must flag rather than gloss over.
The most uncertain input is the OS hazard ratio — its confidence interval crosses 1.0 (0.58–1.08), meaning the survival benefit is not statistically established. The modelled 1.4 incremental life-years depend heavily on extrapolation beyond the trial, and the choice of survival curve (Weibull vs. log-normal vs. generalised gamma) will swing the ICER by tens of thousands of dollars.
This is exactly the kind of uncertainty an HTA committee will probe, and exactly the kind a medical writer must flag rather than gloss over.
Drill — NNT to budget impact
A treatment has NNT = 12 for complete response. Annual drug cost is €48,000 per patient. The eligible population in your country is 6,000 patients per year. What is the drug cost per responder, and the annual budget impact if all eligible patients are treated?
Cost per responder: NNT × annual cost = 12 × €48,000 = €576,000 per responder.
Annual budget impact: 6,000 × €48,000 = €288 million.
This is a treatment that works (NNT of 12 is reasonable in oncology) but has a very large budget impact. The payer will negotiate a discount, a cap, or a performance-based scheme. As a writer, you will see these numbers in AMCP dossiers and NICE evidence submissions.
Annual budget impact: 6,000 × €48,000 = €288 million.
This is a treatment that works (NNT of 12 is reasonable in oncology) but has a very large budget impact. The payer will negotiate a discount, a cap, or a performance-based scheme. As a writer, you will see these numbers in AMCP dossiers and NICE evidence submissions.
Drill — spot the HEOR error
A value dossier states: "The treatment is highly cost-effective, with an ICER of $87,000 per QALY gained versus standard of care. Sensitivity analysis showed the ICER remained below $100,000/QALY in 92% of simulations." What is wrong with this claim?
The claim that $87,000/QALY is "highly cost-effective" is unsupported without stating against which threshold. At NICE's £20,000–£30,000 standard, $87,000 (≈ £69,000) is well above threshold. Even at US thresholds ($50,000–$200,000), calling it "highly" cost-effective is an overclaim — it sits in the middle of the range.
The sensitivity analysis is also misleading: saying the ICER "remained below $100,000" in 92% of simulations is a cherry-picked threshold. The relevant question is what percentage of simulations fell below the applicable willingness-to-pay threshold — and that threshold must be stated.
This is the economic equivalent of finding 3 from the capstone: highly significant attaching magnitude to a p value. Same error, different currency.
The sensitivity analysis is also misleading: saying the ICER "remained below $100,000" in 92% of simulations is a cherry-picked threshold. The relevant question is what percentage of simulations fell below the applicable willingness-to-pay threshold — and that threshold must be stated.
This is the economic equivalent of finding 3 from the capstone: highly significant attaching magnitude to a p value. Same error, different currency.
Quiz
Which of the following is the best description of a QALY?
One year of additional survival regardless of health state
A disease-specific measure of treatment response
One year of life weighted by health-related quality, on a 0–1 utility scale
The cost required to extend life by one year
Your pathway forward
This preview scratched the surface. The full HEOR & Market Access Writing pathway in IMRAD Academy's professional tier covers:
What the HEOR pathway teaches
Cost-effectiveness models: decision trees, Markov models, partitioned survival analysis — how they work, what inputs they need, and how to write about their results without overclaiming.
Value dossiers and HTA submissions: AMCP Format, NICE evidence submissions, G-BA benefit assessments — the documents that get drugs onto formularies.
Systematic literature reviews for HEOR: how SLRs in the economic context differ from clinical SLRs, and why network meta-analyses (NMAs) feed economic models.
Budget impact modelling: the payer's spreadsheet — how it differs from cost-effectiveness analysis and when each is needed.
Real-world evidence in HEOR: claims data, registry data, and how they supplement trial-based models.
Value dossiers and HTA submissions: AMCP Format, NICE evidence submissions, G-BA benefit assessments — the documents that get drugs onto formularies.
Systematic literature reviews for HEOR: how SLRs in the economic context differ from clinical SLRs, and why network meta-analyses (NMAs) feed economic models.
Budget impact modelling: the payer's spreadsheet — how it differs from cost-effectiveness analysis and when each is needed.
Real-world evidence in HEOR: claims data, registry data, and how they supplement trial-based models.
The connection to what you just learned
Every module in the HEOR pathway builds on what you learned here. The survival extrapolation module uses the Kaplan–Meier and hazard ratio concepts from Weeks 3–4. The utility mapping module uses the PRO interpretation from Week 2. The NMA module uses the effect measures and forest plots from Modules 2.3 and 4.2. This statistics course is the prerequisite — you are ready.
Final reflection
Having seen how clinical statistics feed into economic evidence, which part of this course do you now see differently? Would you check anything additional in a clinical study report knowing it will feed a cost-effectiveness model?
Most writers say they would pay more attention to utility data, discontinuation rates (which drive treatment duration and cost), and the shape of the survival curve tail — all things the model depends on.
You have completed the course
Statistics for Medical Writers
IMRAD Academy — Read like a scientist. Write like a professional.
Statistics for Medical Writers · v1.0