Very helpful article for what is likely (as the UK election was) a very poll-dominated Presidential campaign. The danger of applying percentage chances to one-off events is, I feel, also apparent in medical statistics - being told you have a 20% chance of dying in an operation (or of a disease) doesn’t really help you. There’s a binary outcome - you’ll either die or you won’t. It’s rather like Ole Peters’ ergodicity coin toss experiment: what the statistical average is across the ensemble isn’t necessarily a good guide of what will happen to you.
Yes I agree with you about medical statistics, often the results of statistical recommendations are more relevant to the GP than to the patient. My GP surgery like many others uses the QRISK algorithm to estimate the probability that a patient will have a cardiovascular event in the next 10 years. The output is then used to recommend whether a patient should take statins. I played around with the online app to calculate my own risk and discovered that one of the biggest changes to the output came about when I put in my previous postcode compared to my current postcode! I doubt my change of address has really had a large effect on my individual risk. What the algorithm is really telling the GP is the population health of their region and clearly if they can improve the statistical results a small amount over those registered to their surgery it is good for them. However as an individual patient the algorithm isn’t as useful an indication as many would think. The GP gets hundreds of coin tosses, the individual patient only gets the one!
Great post. Allow me to also recommend recent article that argues that, even in hindsight, you can't really tell election forecasters from coin flippers: https://osf.io/preprints/osf/6g5zq
Thanks! Chris Cook on Twitter pointed me at the same article, which looks really interesting, and I'm hoping to find time to read it properly soon.
I said to him there that the effect of these high-profile models influencing the outcome concerns me too. We had a bunch of MRP polls in the UK which were projecting a Labour majority in the high 200s or even 300s. The final outcome was 174, and all the final MRPs were above that. We'll never know of course, but it doesn't seem impossible that some people didn't vote because they thought it was a foregone conclusion (or changed their vote because they didn't want to give one party such a dominant share of the seats) https://en.wikipedia.org/wiki/Opinion_polling_for_the_2024_United_Kingdom_general_election#Seat_projections
We'll never know, but I do think this gives political punditry a bit too much credit. Regardless, the fact that it occupies elite thinking is still deeply pernicious.
Geoff Shullenberger highlighted a passage from Tim Shenck's Left Adrift which made it sound like Hillary Clinton's campaign employed a bunch of Nate Silver wannabees who put a lot more faith in their simulations than on evidence on the ground.
Thanks, you just reminded me of one of my favourite articles of all time https://www.politico.com/magazine/story/2016/09/hillary-clinton-data-campaign-elan-kriegel-214215/ "overnight, in some of the few hours that headquarters isn’t whirring with activity, the team’s computers run 400,000 simulations of the fall campaign in what amounts to a massive stress-test of the possibilities on Nov. 8" - I meant at least wait until you've won the election before you're taking a lap of honour because you've decided you are the smartest kid in the class.
Very helpful! I enjoy following the drama and the calculations and this is a useful corrective.
Your coin toss chart makes me think of designing games, which some of my friends and I do as a hobby. It is so tempting to say something is unbalanced, or a particular feature is broken, just because it wins 2-3 times in succession. But in reality that could just be random chance, and you need to play a fair few more to be confident of that (in some cases).
I think to be fair there's probably more information in these models and you can try to check if they are properly calibrated (do 80% of the states which were rated an 80% chance go that way?). But if you only care about the headline result that may not help much!
Definitely meaningless to pretend there's 3 significant digits to these forecasts, inn reality there's not really any point in having more than one.
My "model" right now is to look at a set of different parameters and what the preponderance of them are telling me will happen. Right now I'm operating on about 5-6 parameters, and they're all related to historical comparisons.
Do Americans feel things are going the right way with the incumbent party? How is the Democrat's favorability compared to Trump's when you compare to when he lost and won? How is Trump's standing in critical states now compared to the previous two elections? Etc.
Right now, all of these parameters are pointing to a Trump win.
Add in the fact that Harris is riding on a very risky "Hide the Harris" strategy meant to prevent the voters from finding out who she is, you've got a stacked deck against her. The first (and probably only) debate will reveal how risky this strategy will have been, like the debate with Biden did with their "Hide the Dementia" strategy.
That Silver's model is inclined to support my current conclusion is just a bonus.
as you say the benefit to the individual in terms of risk of death / days of life lost is marginal. though he does leave out risk of life changing impact of a stroke etc which aren’t to be discounted
Very helpful article for what is likely (as the UK election was) a very poll-dominated Presidential campaign. The danger of applying percentage chances to one-off events is, I feel, also apparent in medical statistics - being told you have a 20% chance of dying in an operation (or of a disease) doesn’t really help you. There’s a binary outcome - you’ll either die or you won’t. It’s rather like Ole Peters’ ergodicity coin toss experiment: what the statistical average is across the ensemble isn’t necessarily a good guide of what will happen to you.
Yes, I think that's a very good analogy .. The Law of Large Numbers applies as N tends to infinity, but not so much for N=1.
Yes I agree with you about medical statistics, often the results of statistical recommendations are more relevant to the GP than to the patient. My GP surgery like many others uses the QRISK algorithm to estimate the probability that a patient will have a cardiovascular event in the next 10 years. The output is then used to recommend whether a patient should take statins. I played around with the online app to calculate my own risk and discovered that one of the biggest changes to the output came about when I put in my previous postcode compared to my current postcode! I doubt my change of address has really had a large effect on my individual risk. What the algorithm is really telling the GP is the population health of their region and clearly if they can improve the statistical results a small amount over those registered to their surgery it is good for them. However as an individual patient the algorithm isn’t as useful an indication as many would think. The GP gets hundreds of coin tosses, the individual patient only gets the one!
Great post. Allow me to also recommend recent article that argues that, even in hindsight, you can't really tell election forecasters from coin flippers: https://osf.io/preprints/osf/6g5zq
Thanks! Chris Cook on Twitter pointed me at the same article, which looks really interesting, and I'm hoping to find time to read it properly soon.
I said to him there that the effect of these high-profile models influencing the outcome concerns me too. We had a bunch of MRP polls in the UK which were projecting a Labour majority in the high 200s or even 300s. The final outcome was 174, and all the final MRPs were above that. We'll never know of course, but it doesn't seem impossible that some people didn't vote because they thought it was a foregone conclusion (or changed their vote because they didn't want to give one party such a dominant share of the seats) https://en.wikipedia.org/wiki/Opinion_polling_for_the_2024_United_Kingdom_general_election#Seat_projections
We'll never know, but I do think this gives political punditry a bit too much credit. Regardless, the fact that it occupies elite thinking is still deeply pernicious.
Geoff Shullenberger highlighted a passage from Tim Shenck's Left Adrift which made it sound like Hillary Clinton's campaign employed a bunch of Nate Silver wannabees who put a lot more faith in their simulations than on evidence on the ground.
https://x.com/g_shullenberger/status/1829676306193400173
Thanks, you just reminded me of one of my favourite articles of all time https://www.politico.com/magazine/story/2016/09/hillary-clinton-data-campaign-elan-kriegel-214215/ "overnight, in some of the few hours that headquarters isn’t whirring with activity, the team’s computers run 400,000 simulations of the fall campaign in what amounts to a massive stress-test of the possibilities on Nov. 8" - I meant at least wait until you've won the election before you're taking a lap of honour because you've decided you are the smartest kid in the class.
Very interesting, Oliver.
For readers over the pond, check out the Numberwang sketches on YouTube. Very funny.
This is a very interesting article. Thanks.
Very helpful! I enjoy following the drama and the calculations and this is a useful corrective.
Your coin toss chart makes me think of designing games, which some of my friends and I do as a hobby. It is so tempting to say something is unbalanced, or a particular feature is broken, just because it wins 2-3 times in succession. But in reality that could just be random chance, and you need to play a fair few more to be confident of that (in some cases).
I think to be fair there's probably more information in these models and you can try to check if they are properly calibrated (do 80% of the states which were rated an 80% chance go that way?). But if you only care about the headline result that may not help much!
If you actually understand math, Nate’s models are very simple.
Definitely meaningless to pretend there's 3 significant digits to these forecasts, inn reality there's not really any point in having more than one.
My "model" right now is to look at a set of different parameters and what the preponderance of them are telling me will happen. Right now I'm operating on about 5-6 parameters, and they're all related to historical comparisons.
Do Americans feel things are going the right way with the incumbent party? How is the Democrat's favorability compared to Trump's when you compare to when he lost and won? How is Trump's standing in critical states now compared to the previous two elections? Etc.
Right now, all of these parameters are pointing to a Trump win.
Add in the fact that Harris is riding on a very risky "Hide the Harris" strategy meant to prevent the voters from finding out who she is, you've got a stacked deck against her. The first (and probably only) debate will reveal how risky this strategy will have been, like the debate with Biden did with their "Hide the Dementia" strategy.
That Silver's model is inclined to support my current conclusion is just a bonus.
interesting read on statins here https://sebastianrushworth.com/2022/06/14/should-the-patient-really-get-the-drug/
as you say the benefit to the individual in terms of risk of death / days of life lost is marginal. though he does leave out risk of life changing impact of a stroke etc which aren’t to be discounted