Great fun!

]]>The two candidates in the runoff were Emmanuel Macron and Marine Le Pen and Macron comfortably won in the second round. My predictions (made before the Fillon scandal concerning payments to his family members from the public purse for no work) gave this a first round result probability of 3% and a second round probability of 7%. The surprisal for the first round is 5.1 bits (expected surprisal 1.1 bits), so not a very good prediction. Second round surprisal of 3.8 bits versus expected 1.4 bits is not quite so bad, but still not great.

Go France for confounding my expectations!

So Theresa May called an early general election (a possibility I had not considered when making my predictions), and then partially lost it (a possibility I doubt I would have considered). Some of the probabilities I put together are obviously affected by this, but less than they would have been by an increased Conservative majority. Jeremy Hunt survives another reshuffle as Health Secretary, Boris Johnson remains Foreign Secretary, Jeremy Corbyn remains Labour party leader, now in a much stronger position than previously.

]]>This carries a surprisal of 0.515, versus expected surprisal (Shannon entropy) of 0.881. Good going!

]]>The news has been published in the BMJ^{4} and has already attracted comment (see response in the article).

The NICE guidance includes information to support the NHS in adopting the recommendations.

My previous blog post (about the draft guidance) gives more detailed background on this area, and the final guidance appears to be very similar to the draft guidance.

- Snowsill T, Coelho H, Huxley N, Jones-Hughes T, Briscoe S, Frayling I, Hyde C. Molecular testing for Lynch syndrome in people with colorectal cancer: systematic reviews and economic evaluation. Health Technol Assess (In Press).
- Snowsill T, Huxley N, Hoyle M, Jones-Hughes T, Coelho H, Cooper C, Frayling I, Hyde C. A model-based assessment of the cost-utility of strategies to identify Lynch syndrome in early-onset colorectal cancer patients. BMC Cancer 2015; 15(313)
- Snowsill T, Huxley N, Hoyle M, Jones-Hughes T, Coelho H, Cooper C, Frayling I, Hyde C. A systematic review and economic evaluation of diagnostic strategies for Lynch syndrome. Health Technol Assess 2014; 18(58)
- Gulland A. All patients with colorectal cancer should be tested for genetic condition, NICE advises. BMJ 2017; 356:j998

I made a particularly bad prediction a few weeks ago in only giving “Other” candidates a 2% chance (in total) of winning the French Socialist Party presidential primary. It turned out in the end that not only did Benoit Hamon make it through to the second round ahead of Arnaud Montebourg, he then won the second round.

That measly 2% corresponds to a surprisal of 5.64 bits. My predicted Shannon entropy for this contest was 1.12 bits. What this tells us is that I underestimated the uncertainty in this contest quite considerably.

We shall wait to see how well my predictions hold up when the full Presidential contest is held.

]]>

So the title for Episode VIII was revealed as “The Last Jedi”.

To assess my predictive ability, I include the surprisal () and the Shannon entropy (). I am aiming to both have low surprisal (i.e., low ) but also to be well calibrated, so that .

Total | 7.79 | 5.10 | ||
---|---|---|---|---|

Item | P(Contains) | P(Does not contain) | E[I] | I |

Force | 0.2 | 0.8 | 0.722 | 0.322 |

Dark/Darkness/Dark side | 0.4 | 0.6 | 0.971 | 0.737 |

First Order | 0.2 | 0.8 | 0.722 | 0.322 |

Resistance | 0.2 | 0.8 | 0.722 | 0.322 |

Knights of Ren | 0.1 | 0.9 | 0.469 | 0.152 |

Republic | 0.05 | 0.95 | 0.286 | 0.074 |

Empire | 0.05 | 0.95 | 0.286 | 0.074 |

Rebellion/Rebel Alliance | 0.05 | 0.95 | 0.286 | 0.074 |

Jedi | 0.3 | 0.7 | 0.881 | 1.737 |

Sith | 0.1 | 0.9 | 0.469 | 0.152 |

Rescue | 0.4 | 0.6 | 0.971 | 0.737 |

Battle/War | 0.2 | 0.8 | 0.722 | 0.322 |

Name of a character | 0.05 | 0.95 | 0.286 | 0.074 |

As it turns out, I ended up with , which suggests (if anything) that I was not sufficiently bold in my predictions (i.e., I should have had somewhat lower predictions across the board).

These calculations are all assuming that these are independent predictions, which they are not really (I wouldn’t expect a title to have *both* Jedi *and* Sith, for example), but they are at least somewhat informative.

In terms of my specific prediction, I gave a 30% chance to the title containing Jedi, which was the 3^{rd} highest chance I gave, behind “Rescue” and “Dark/Darkness/Dark side” (both on 40%), so I don’t think that’s too bad!

]]>

My predictions for the title of Episode VIII are:

- Contains the Force: 20%
- Contains Dark, Darkness or the Dark Side: 40%
- Contains the First Order: 20%
- Contains the Resistance: 20%
- Contains the Knights of Ren: 10%
- Contains the Republic: 5%
- Contains the Empire: 5%
- Contains the Rebellion or the Rebel Alliance: 5%
- Contains Jedi: 30%
- Contains Sith: 10%
- Contains Rescue: 40%
- Contains Battle or War: 20%
- Contains the name of a character: 5%

- Rey’s father is revealed: 70%
- Luke Skywalker: 80%
- Obi-Wan Kenobi or descendant: 3%
- Han Solo: 2%
- Lor San Tekka: 1%
- No-one (immaculate/Force conception): 4%
- Someone else: 10%

- Rey’s mother is revealed: 10%
- Leia: 2%
- Mon Mothma: 1%
- Someone else: 97%

- Leia: 40%
- Luke: 40%
- General Hux: 30%
- Snap Wexley: 25%
- Captain Phasma: 25%
- Poe: 20%
- Finn: 15%
- Chewbacca: 10%
- Snoke: 5%
- Mas Kenata: 5%
- BB-8: 5%
- Rey: 2%
- Kylo: 1%
- C-3PO: 1%
- R2-D2: 1%

- Force ghost of Anakin: 15%
- Lando: 10%
- Force ghost of Obi-Wan: 5%
- Force ghost of Yoda: 3%
- Wedge: 1%

Twenty-seventeen is unlikely to see a return to boring times, with Brexit negotiations commencing after Article 50 is triggered, elections in France and Germany, and Trump’s first year in office, in addition to continuing conflicts in the Middle East.

At the end of the year I will review my predictions. Each outcome will be scored with its self-information (measured in bits), , with a low result being better. I will also attempt to assess calibration across predictions.

Angela Merkel’s CDU have been losing ground in the opinion polls for the last 18 months (which neatly mirrors the increased poll share for Alternative für Deutschland), but are still on track to be the largest party in the Bundestag. The SPD have lost some ground in the last 12 months, mirrored by the rise in the polling average for the Green Party. Current polling suggests all these parties, as well as the FDP and the Left Party will exceed the 5% threshold required to be allocated seats in the Mixed Member Proportional system.

The three previous coalitions have been CDU—FDP (2009–2013) and CDU—SPD (2005–2009 and 2013–present). Angela Merkel would be most keen to reform a coalition with the FDP, while Sigmar Gabriel (SPD) would potentially seek to form a coalition with the Green and the Left parties. Both major parties would be keen to avoid a coalition with AfD. Current polling puts both CDU—FDP and SPD—Left—Green at 40–45% and AfD at 10–15%.

My predictions are therefore:

- CDU—SPD±FDP: 70%
- CDU—FDP: 10%
- SPD—Left—Green: 15%
- Other (CDU): 3%
- Other (SPD): 2%

Putting Angela Merkel continuing in the role of Chancellor at 83%.

The race to be the Socialist Party candidate is being led by Manuel Valls and Arnaud Montebourg. Based on current polling, Valls is expected to lead in the first round, but more voters for other candidates are expected to split to Arnaud Montebourg in the second round, making it a close run thing.

Predicted candidate:

- Manuel Valls: 53%
- Arnaud Montebourg: 45%
- Other: 2%

Predicted top two in first round:

- Socialist Party candidate and Marine Le Pen (Front National): 3%
- Socialist Party candidate and François Fillon (Republican): 2%
- Socialist Party candidate and Emmanuel Macron (Independent/En Marche): 1%
- François Fillon and Marine Le Pen: 82%
- François Fillon and Emmanuel Macron: 8%
- Marine Le Pen and Emmanuel Macron: 3%
- Other: 1%

Predicted winner:

- François Fillon: 63%
- Marine Le Pen: 27%
- Emmanuel Macron: 7%
- Manuel Valls: 1%
- Arnaud Montebourg: 1%
- Other: 1%

A grab-bag of predictions from most likely to least likely:

- Jeremy Hunt still Health Secretary at end of the year: 90%
- Labour hold Leigh by-election triggered after Andy Burnham elected Mayor of Greater Manchester: 80%
- Labour lose Copeland by-election: 70%
- New Foreign Secretary by end of the year: 60%
- Number of Conservatives in the House of Lords increases from 255 to over 280: 50%
- A serious attempt to unseat Corbyn by end of the year: 40%
- Theresa May conducts significant reshuffle (changes to 4 or more Cabinet positions): 30%
- Simon Stevens stands down as chief executive of NHS England: 20%
- New Labour leader by end of the year: 10%

Funnel plots (or funnel control charts) are a popular way to explore outcomes from organisations of different sizes.

They make it possible to see how much variability can be explained by random chance (small sample errors) and how much by common heterogeneity.

Plotting the unit cost of colonoscopy from different organisations makes it clear that a funnel plot may aid in the interpretation.

I assumed that logarithm of the mean cost from each organisation would be normally distributed (unfortunately we are given the arithmetic mean rather than the geometric mean, so there is likely some bias) as follows:

where is the log(Unit cost) for organisation .

Using maximum likelihood estimation for colonoscopy costs (FZ51Z, FZ52Z and FZ53Z) from 2014-15, I fitted the model, as shown in this funnel plot:

The STATA command I used was:

mlexp (-0.5*( ln(2*_pi*(exp(2*{lnsigma=1})/activity + exp(2*{lntau=-2}))) + (ln_unitcost - {mu})^2/(exp(2*{lnsigma})/activity + exp(2*{lntau})) ))

This produced the following parameter estimates:

Coef. | Std. Err. | 95% Conf. Interval | ||
---|---|---|---|---|

6.363 | 0.011 | 6.343 | 6.384 | |

0.878 | 0.029 | 0.821 | 0.934 | |

0.492 | 0.011 | 0.471 | 0.513 |

We can now show the fitted model for the distribution of ‘true’ unit costs across organisations, :

The quartiles of this distribution are at £416, £580 and £809. These compare to the quartiles of the raw data (as used in the simple method described in the previous post), which are £401, £587 and £868. It is unsurprising that the interquartile range is greater in the raw data, since this includes the source of variance.

Now that we have quantified the uncertainty due to heterogeneity between organisations, we can consider how this relates to the distribution we want to assign to the unit cost in a probabilistic sensitivity analysis.

We need to think from the perspective of the NHS. We are (usually) considering doing more of these procedures, and this has an opportunity cost. We don’t know where the additional procedures will be done, but we might start with the assumption that they will be distributed as they are currently.

As the number of new procedures increases, by the central limit theorem we should see that the average cost of the new procedures approaches the expected value of the distribution (assuming there is a constant marginal cost), which is .

So, perhaps we really want to look at the uncertainty in this, due to model fitting.

nlcom (mean: exp(_b[mu:_cons]+0.5*exp(2*_b[lntau:_cons])))

This gives us a mean cost of £655 (95% CI, £639 to £671), but this is significantly different from the £572 which is the actual sample mean. Calculating a median from the model fit gives £580 (95% CI, £568 to £592).

To investigate how much this is an artefact of the model over-fitting to low activity organisations, I removed organisations with fewer than 50 procedures. The sample mean lowered to £565 (not a great change), and the mean estimated from the model fit dropped significantly to £577 (95% CI, £559 to £595).

The table below summarises the estimates from alternative methods:

Method | Mean | 95% Conf. Interval | |
---|---|---|---|

MLE | £655 | £639 | £671 |

MLE excl. small | £577 | £559 | £595 |

Simple | £572 | £568 | £576 |

Simple excl. small | £565 | £561 | £569 |

It can be seen that the model fitting method leads to higher mean estimates (undesirable) and significantly wider confidence interval estimates (possibly desirable).

Alternative methods of quantifying the uncertainty due to heterogeneity can lead to different estimates. If the results of a decision model are sensitive to a particular unit cost, it is advised that care is taken to consider whether the standard approach gives an adequate estimation of the uncertainty.

]]>NHS reference costs are frequently used when estimating how cost-effective different treatments would be in the NHS.

When economic models are developed for estimating cost-effectiveness, a key part of the process is to quantify the amount of uncertainty in the different parameters, including the costs of procedures. There are three key sources of uncertainty to consider:

- Sampling uncertainty – Some procedures are not performed very frequently, so if there were additional procedures within the same year, our estimate for the average in that year may move.
- Heterogeneity – The cost of performing a procedure may vary systematically according to characteristics of the patient or the organisation. In the published reference costs, the heterogeneity due to organisations can be seen in the organisation level data, which can show significant variability in the average costs (see Figure 1).
- Changing costs over time – The cost of performing a procedure may change over time. The cost of a procedure in the financial year 2014-15 may be estimated with great precision, but this might not be a good estimate of the cost of the same procedure in 2015-16 (even after adjusting for inflation).

A common approach is to estimate the standard error from the lower and upper quartile unit costs from the different organisations.

Let denote the number of organisations providing unit cost estimates. Let and denote the lower and upper quartile unit costs.

My thoughts on this approach, and some alternatives, to follow…

]]>