This chapter sets out how the model is checked against observed data. Three calibrations set inputs, each in its own run mode (chapter 8): quit success, to the national results of the services in 2024 (chapter 4); coverage before 2026, to NHS England’s referral data (below); and admissions without the service, to the Ottawa control group (Tab_14b, below). Everything else in this chapter is a check, with nothing calibrated to it: the shares of referrals by sex and IMD quintile, quit attempts and quits in 2022/23, deaths after discharge, and the model’s quitters against the quitters in two evaluations. Inside the code, the inputs are checked against their contracts when they are read (R/check_inputs.R), and the run stops with a message if a check fails rather than carrying on with a default; each run writes run_check.csv with its headline figures.
The years from the index year (2019) to 2025 are matched to referrals to the service in NHS England’s tobacco patient-level data (extract 67_017, June 2026). The extract counts admissions of acute inpatients recorded as smoking with an in-house referral to the tobacco dependence service, by month from April 2022, broken down by age group, sex, ethnic group and IMD quintile. Counts are rounded to the nearest 5 and small counts suppressed. scripts/01_data_prep_real/08_process_pld_referrals.R turns it into national totals by month and calendar year: each month’s total is the mean of the four breakdowns’ totals, a suppressed count is taken as 2, and the script stops if the four totals don’t agree within what rounding and suppression allow.
What is matched. The service is opt-out, so the model’s referrals are the admissions of identified smokers within the service’s reach, with the service (identified_smokers_adms), on the same admissions basis as the data. Referred patients who go on to have a specialist assessment are “seen”, a later stage. Within-trust coverage for 2023, 2024 and 2025, the three full calendar years in the data, is solved so that the model’s referrals equal the data’s (R/calibrate_history.R). Within a year referrals are in proportion to reach, so each year’s coverage is the target divided by trust coverage times the referrals at full reach; the years are linked through the quitters carried forward, so this is repeated until every year is within 0.01% of its target. Trust coverage is NHS England’s share of trusts with a live service by financial year (5% in 2020/21, 16%, 51%, 86%, 93% and 91% in 2025/26), with a calendar year taken as a quarter of the financial year ending in it and three quarters of the one starting in it. Reach before 2023 keeps the values assumed in earlier versions of the model.
Checks. Tab_X compares the model with the data for each calibrated year: the national total (equal by construction) and the shares of referrals by sex and IMD quintile, which are not calibrated. A share passes if it is within 5 percentage points of the data. The shares show whether the model’s mix of identified smokers, which comes from the hospital data and the smoking calibration (chapter 3), matches the mix of patients the services refer. Tab_X2 gives referrals, specialist assessments and people assessed by year with the observed annual totals, and scripts/04_reports/figures/fig_rollout_calibration.R draws them against the monthly data for the report (from the disclosure-controlled results).
Limitations. Not every trust submitted data every month (63 of 111 did in 2024 (Agrawal et al. 2026)), so the data undercount referrals and the calibrated coverage is, if anything, too low. The age groups in the extract don’t line up with the model’s, so age is not checked. Tab_X also checks quit attempts and four-week quits in 2022/23 against an earlier ScHARR analysis of the patient-level data (user_inputs/historical_calibration_panel.csv). Quit attempts pass within 25%. The four-week quit rate, overall and by IMD quintile, is checked against a range, 12% to 54%, rather than a point, because the pathway calibration sets it from the 2024 data.
Most business case modelling of hospital tobacco dependence treatment in England has used one study, the evaluation of the Ottawa Model for Smoking Cessation by Mullen et al. (Mullen et al. 2017), and it has been used in different ways (Gillespie 2024). Hospital-TTD-Mod estimates benefits differently, from the relative risks of each smoking-related condition, so its results need to be explained against Ottawa’s. The model does this in two places, both for one annual cohort of patients (the 2026 cohort), because Ottawa’s results are per patient over two years:
Both are built by R/build_ottawa_reconciliation.R, with Ottawa’s numbers in section 10 of 00_config.R. Tab_14c lists the numbers behind them. Two checks sit beside them, with nothing calibrated to either: Tab_14d compares the model’s deaths after discharge with Ottawa’s, and Tab_14e compares the model’s quitters with the quitters in Ottawa and in a London service based on it, for admissions and for deaths.
The evaluation compared 641 patients who smoked and were admitted to 14 Ontario hospitals before the programme started (usual care) with 726 admitted after it started, between 2005 and 2009, using linked administrative data. It was not randomised. Patients had a mean age of 52, and around 70% had high overall morbidity.
Outcomes were followed for two years after discharge. The headline result, an 11.6 percentage point fall in the proportion of patients readmitted within two years (45.2% to 33.6%), counts patients with at least one readmission. Figure 1 of the paper also gives the number of readmissions in both groups together and the adjusted hazard ratio, which together give the readmissions per smoker in each group. Treating the hazard ratio as the ratio of the two groups’ readmission rates, the control group’s rate is the number of readmissions divided by (641 + 726 \(\times\) hazard ratio):
| Ottawa control group, per smoker | Readmissions (both groups) | Hazard ratio | Control group rate |
|---|---|---|---|
| Smoking-related, year 1 | 490 | 0.69 | 0.43 |
| Smoking-related, years 1 and 2 | 718 | 0.79 | 0.59 |
| Smoking-related, year 2 | 0.16 | ||
| All causes, years 1 and 2 | 1,042 | 0.79 | 0.86 |
Readmissions are counted after the index admission, and patients who died stay in the denominator. Ottawa’s smoking-related conditions follow the 2014 Surgeon General’s report, which is close to, but not the same as, the model’s list of 42 conditions.
Six months after discharge, 35.2% of intervention patients and 20.4% of control patients who could be reached had not smoked in the past six months, a difference of 14.8 percentage points. Only around a third of patients were reached. Counting patients who could not be reached as smoking, the rates were 12.4% and 7.0%, a difference of 5.4 points. Across both groups, patients who had not smoked at six months were less likely to be readmitted over two years than those who carried on smoking (hazard ratio 0.78, 95% CI 0.58 to 1.03).
Without the service, patients keep being admitted in each follow-up year at a multiple of the admissions they had after their first in the index year (chapter 5). The multiple is \(k_1\) in follow-up year 1 and \(k_2\) in years 2 to 5 (follow_up_admissions_factor_y1 and follow_up_admissions_factor_y2plus in scenarios_control_panel.csv).
\(k_1\) and \(k_2\) are set so that, in the no-service arm of the national run, the smoking-related admissions of patients who smoked at the index admission, divided by the number of those patients, match the readmission targets in user_inputs/readmission_targets.csv: today the Ottawa control group, 0.43 in year 1 and 0.16 in year 2. English figures can replace them there, with their source, without a code change. As in Ottawa, the patients who smoked include those who quit without the service, and deaths are allowed for. Every outcome condition in the model is smoking-related, so all of the model’s follow-up admissions count.
Admissions in the model are proportional to \(k\), so one run gives the factors that match. Tab_14b shows, for each follow-up year, the model’s admissions per smoker, the target, the factor used in the run and the factor that matches the target. The factors are set by R/calibrate_follow_up.R (MAIN_RUN_MODE “calibrate_follow_up”, chapter 8). They depend on the effects of quitting, because patients who quit without the service benefit too, and the share of admissions that smokers account for depends on how much lower former smokers’ risk is. The scenarios that change the effects of quitting (FOLLOW_UP_OWN_FACTOR_SCENARIOS) get their own factors and every other scenario the base case’s; the scenarios in FOLLOW_UP_NO_DECLINE_SCENARIOS get \(k_2\) equal to \(k_1\). In the full run Tab_14b then shows the model and Ottawa rates equal in years 1 and 2.
Area runs keep the national factors, so differences in case mix between areas carry through to their results. Ottawa followed patients for two years, so years 3 to 5 carry on at the year 2 rate. Readmissions in Ottawa fall sharply after the first year, mainly because of readmissions soon after discharge; Sens_FollowUp_NoDecline shows the effect of assuming they don’t.
Ottawa’s patients were sicker than English inpatients on average, so its readmission rates may be higher than England’s. In a London evaluation of a service based on the Ottawa Model, 22.4% of patients seen were readmitted to the same Trust between 31 days and one year after discharge (Robins et al. 2024), which counts only one Trust. The corresponding figures in Ottawa were around 25% in the control group and 20% in the intervention group (38.4% and 26.7% readmitted within a year, less 13.3% and 7.1% within 30 days). England’s readmission rates therefore look similar to Ottawa’s, and calibrating to Ottawa is not expected to understate them.
Ottawa’s control group had 11.4% of patients dead within one year of the index admission and 15.1% within two; its intervention group 5.4% and 7.9%. Tab_14d gives the proportion of smokers at the index admission who have died by the end of each follow-up year in each arm of the national run, next to these. Nothing is calibrated to it. Ottawa’s patients had more illness than English inpatients on average, but were younger (mean age 52), so the no-service arm and Ottawa’s control group should be of a similar size; figures well above Ottawa’s would suggest that the death rates by condition (chapter 5, “Deaths after discharge”) are too high. Ottawa’s fall in deaths is far larger than its 5 to 15 point difference in quit rates could give through quitting, so the model’s difference between arms is expected to be much smaller.
Two evaluations compared patients who quit with patients who carried on smoking, for readmissions and for deaths. They are the only such estimates for hospital inpatients, and the model uses them as checks, not targets (chapter 5, “Why Ottawa’s quitters are a check, not a target”):
| Outcome | Study | Window | Quit status | Estimate (95% CI) |
|---|---|---|---|---|
| Readmission, any cause | Ottawa (Mullen et al. 2017) | 2 years | Not smoked in the past 6 months, patients reached | HR 0.78 (0.58 to 1.03) |
| Readmission to the same trust | London (Robins et al. 2024) | 31 days to 1 year | Not smoking at 30 days; patients not reached counted as smoking | OR 0.83 (0.51 to 1.35) |
| Death, any cause | Ottawa | 2 years | As above | HR 0.53 (0.32 to 0.83) |
| Death, any cause | London | 31 days to 1 year | As above | OR 1.41 (0.65 to 3.05) |
Tab_14e gives the model’s equivalent beside each, for the 2026 cohort of the national run. For admissions it is \(1 - P_W / (Q imes a_W)\), where \(P_W\) is the admissions prevented by the effect on illness in window \(W\), \(Q\) the extra 6-month quitters and \(a_W\) the no-service admissions per smoker in the window; over two years this is the equivalent hazard ratio from the staircase. For deaths it is a quitter’s cumulative death rate from all causes over a continuing smoker’s, in the same patient groups, weighted by the service’s 6-month quitters. London’s window is read as follow-up year 1 and its odds ratios as ratios of risks, which is close while fewer than a third of patients have the outcome. Each row says whether the model’s value lies inside the study’s interval.
How to read them:
A model value below a study’s interval (a larger effect than the study found) would be a reason to look again at the effects of quitting.
| Step | Readmissions avoided over two years | What changes from the step before |
|---|---|---|
| 1 | Ottawa’s 11.6% reduction in the proportion of patients readmitted \(\times\) patients seen by the service | The method used in past business cases (Royal College of Physicians, ASH) |
| 1 (all identified smokers) | As Step 1 \(\times\) identified smokers | As in the Manchester business case |
| 1 (NHS England) | 0.154 \(\times\) patients seen | NHS England’s method (below) |
| 2 | Ottawa control readmissions per smoker, all causes (0.86) \(\times\) (1 \(-\) 0.79) \(\times\) identified smokers | Counts readmissions rather than patients readmitted, and applies to all identified smokers, because Ottawa compared every smoker admitted, not only those who took up support |
| 3 | As Step 2 with smoking-related readmissions (0.59, hazard ratio 0.79) | Keeps the outcomes the model covers |
| 4 | Step 3 \(\times\) extra 6-month quitters per identified smoker / 0.148 | Scales Ottawa’s effect to the service’s extra quitters; a second row uses the intention-to-treat difference of 0.054 |
| 5 | Extra 6-month quitters \(\times\) 0.59 \(\times\) (1 \(-\) 0.78) | Keeps only the part of Ottawa’s effect that quitting explains |
| 6 | Extra 6-month quitters \(\times\) the model’s no-service rate over two years \(\times\) (1 \(-\) 0.78) | Uses England’s readmission rates from the model; equal to Step 5 nationally once calibrated |
| 7 | Hospital-TTD-Mod, admissions prevented in follow-up years 1 and 2 by the effect on illness | The model; the admissions of patients alive because they quit are left out, because a hazard ratio compares patients while they are alive |
The table also gives the ratio of Step 7 to Step 6 and, for context, the model’s admissions prevented over all five follow-up years for the same cohort.
# Extract from R/build_ottawa_reconciliation.R
ottawa_control_rate <- function(events, hr) {
events / (OTTAWA_N_CONTROL + OTTAWA_N_INTERVENTION * hr)
}
ott_sr_2y <- ottawa_control_rate(OTTAWA_SR_READMISSIONS_2YR, OTTAWA_SR_HR_2YR)
step3 <- n_identified * ott_sr_2y * (1 - OTTAWA_SR_HR_2YR)
step4 <- step3 * model_gap / ott_gap_responders
step5 <- extra_quits * ott_sr_2y * (1 - OTTAWA_HR_QUITTERS_2YR)
step6 <- extra_quits * model_sr_2y * (1 - OTTAWA_HR_QUITTERS_2YR)
step7 <- model_2yStep 1: three past methods. The report’s Ottawa appendix describes how past models used Ottawa, from our 2024 review (Gillespie 2024). The Royal College of Physicians and ASH models applied Ottawa’s fall in the proportion readmitted to patients taking up support, and the Manchester business case to every smoker admitted. NHS England’s model splits Ottawa’s 1,042 readmissions between the groups with an adjuster \(a = p(1 - \Delta / 2r)\), where \(r\) = 1,042 / 1,367 is the readmissions per patient in both groups, \(p\) = 726 / 1,367 the intervention group’s share of patients and \(\Delta\) = 0.12 the fall in the proportion readmitted, rounded (NHSE_OTTAWA_ARR_USED in 00_config.R). That gives 0.830 readmissions per patient in the control group and 0.702 in the intervention group, a relative fall of \(d\) = 0.154, which the model multiplies by patients treated. nhse_ottawa_relative_change() reproduces this and stops if it does not give the workbook’s 0.154. Counting readmissions consistently would give \(r_c - r_i\) = 0.128 per patient.
From Step 1 to Step 4. Past business cases differ from Ottawa in three ways: most count patients rather than readmissions, most apply the effect to patients seen rather than to every smoker, and all assume England achieves Ottawa’s difference in quit rates. Steps 2 to 4 put each of these right.
From Step 4 to Step 5: the part of Ottawa’s effect that runs through quitting. Ottawa evaluated the whole programme: bedside counselling, medication, and eight follow-up calls over six months with call-backs from a nurse. Hospital-TTD-Mod estimates only the benefits that come from stopping smoking. Step 5 asks how large Ottawa’s effect would have been if quitting had been its only route, using Ottawa’s own numbers.
If the only route is through being quit at six months, a group in which a proportion \(p\) have quit has a readmission hazard, relative to continuing smokers, of \((1 - p) + p \times h\), where \(h\) is the quitters’ hazard ratio. The hazard ratio between the intervention and control groups is then
\[HR_{quit} = \frac{1 - p_i (1 - h)}{1 - p_c (1 - h)}.\]
With \(p_i = 0.352\), \(p_c = 0.204\) and \(h = 0.78\), \(HR_{quit} = 0.966\), against the 0.79 observed. Quitting therefore accounts for \((1 - 0.966) / (1 - 0.79)\), or about 16%, of the observed effect. Step 5 in Tab_14 uses the simpler approximation that the relative reduction is the difference in quit rates times \((1 - h)\), which gives 15%. Mullen et al. read the similar size of the quitters’ hazard ratio (0.78) and the programme’s (0.79) as support for quitting as the mechanism. But the first compares people who quit with people who did not, and the second compares two groups in which only 15 percentage points more people quit, so if quitting were the only route the second would be much closer to 1 than the first.
Only around a third of patients were reached at six months, and both the quit rates and the quitters’ hazard ratio come from them. The share explained under different assumptions:
| Quitters’ hazard ratio \(h\) | Quit rates among those reached (35.2% and 20.4%) | Intention to treat (12.4% and 7.0%) |
|---|---|---|
| 0.78 (Mullen et al.) | 16% | 6% |
| 0.58 (lower 95% confidence limit) | 32% | 11% |
| 0.40 | 48% | 16% |
| 0 (quitters never readmitted) | 88% | 28% |
Loss to follow-up does not change the conclusion through the quit rates. With \(h = 0.78\), matching the observed 0.79 would need more than 100% of the intervention group to have quit, whatever the control group’s rate. Counting patients who were not reached as smoking makes the share smaller, and for the share to be larger, the patients who were not reached would need a bigger difference in quit rates than those who were. The conclusion does depend on the quitters’ hazard ratio, which is observational and imprecise. Quitting could explain nearly all of the effect only if quitters had almost no readmissions. The usual bias in comparisons of quitters with continuing smokers, that people who manage to quit are healthier, makes \(h\) look stronger than it is, not weaker. Reported quitting was not checked biochemically, which weakens \(h\) but also overstates the difference in quit rates; if a fifth of reported quitters were still smoking, the share would still be about 16%.
Three other results in the paper do not depend on follow-up and point the same way. Readmissions not related to smoking fell by as much as smoking-related ones (hazard ratio 0.79 for both). About half of the fall in readmissions within a year had already happened within 30 days of discharge (6.1 of 11.7 percentage points), when the biological benefits of quitting are smallest, although some benefits do come within weeks (Shahab 2013). And the same calculation with the quitters’ hazard ratio for death (0.53) explains about 19% of the fall in deaths within two years (15.1% to 7.9%).
The rest of Ottawa’s effect may come from other parts of the programme, such as cutting down, contact after discharge or medication use, or from differences between the groups compared, which differed at baseline in history of heart attack and cancer. A London service based on the Ottawa Model also found fewer readmissions among patients who received it, whether or not they quit (Robins et al. 2025). This does not mean Ottawa’s results are wrong. It means the programme’s effect on readmissions was larger than its effect on smoking can explain, and the model, which counts only the benefits of quitting, does not include the remainder. In that respect the model is conservative.
From Step 5 to Step 6. Nationally the two are equal, because the model’s readmission rates without the service are calibrated to Ottawa’s. For an area they differ by the area’s case mix.
From Step 6 to Step 7. Step 6 gives each extra quitter Ottawa’s quitters’ hazard ratio of 0.78 from discharge. The ratio of Step 7 to Step 6, \(r\), compares the model’s effect for a quitter over the same two years with Mullen et al.’s, on the same readmission rates. It is equivalent to a quitters’ hazard ratio of \(1 - r \times (1 - 0.78)\), which Tab_14e sets beside Ottawa’s interval. Nothing is calibrated to it. An \(r\) below 1 means the model’s quitters do less well over the first two years than Ottawa’s: the conditions with evidence from people who already have the disease (heart disease, stroke, COPD and the conditions that borrow from them) carry their studies’ effects, and the rest recover along the published curves, which are slow in the first year. The Ottawa estimate is itself observational, and people who manage to quit may differ in other ways from those who do not.
So the central estimate reproduces Ottawa’s control-group readmission rates, and its quitters’ benefit can be read against Ottawa’s and London’s. The difference between Step 4 and Step 7 is then the part of Ottawa’s programme effect that its quit rates can’t explain (Step 4 to Step 5), the service’s quit rates in England (Steps 2 to 4), and any gap between the model’s quitters and Ottawa’s (Step 6 to Step 7).
Five years. Ottawa followed patients for two years. The model counts benefits for five, so the five-year total is the better guide to the model’s results.