I’m a data nerd and a data cheerleader, but still I fear Bill English’s datatopia
NZ’s new prime minister is a champion of evidence-based policy, the “social investment approach” and open government. So why is fellow data-evangelist Keith Ng warning of a data-democracy cargo cult?
Bill English is the most data nerdy prime minister we have ever had. An ex-Treasury wonk and a champion of open data across government, he’s likely to continue his push for evidence-based, data-driven policy-making from the 9th Floor.As a data nerd, I should love it. I used to dream about this datatopia too. But here’s why it could go terribly wrong. There are two parts to his vision: Evaluation and democratisation. The evaluation part is called the Social Investment approach, which is about “applying rigorous and evidence-based investment practices to social services”… which basically means treating social spending as an investment, and measuring it like you’d measure an investment (ie on a spreadsheet). The democratisation part is about opening up data and making it available to everyone, in order to democratise policy-making. As English has put it:
“Public policy people have this view that everything they do is highly complex and very special. We run university systems just to train people in public policy. But they’re wrong. Policy is now a commodity – you can print world best practice off the internet. You don’t need a department to know it, a 12-year-old can do it.”How could anyone have a problem with rigorous and evidence-based evaluations, or with the democratisation of policy-making?
Tyranny of the null hypothesisThis came up recently in a podcast with political scientist Grant Gordon, on whether an evidence-based approach ought to be used to evaluate conflict interventions. He offered a hypothetical (around 43:20 in): What if the US civil rights movement was assessed like this? The changes in wellbeing for African-Americans don’t show up for years, but racial violence spiked as a direct response to the reforms. From the evidence, it looked like the civil rights reforms wasn’t making people’s lives better and was actually causing violence! So it should be abandoned – right? The civil rights hypothetical is great because we have intuitions about what is just. We can sense (and know with the benefit of hindsight) why it was important, beyond what could’ve been measured at the time. It tells us why we should push through with something despite the lack of evidence; through that, we can understand the limitations of what was measured. But most policies are going to be much lamer: There is insufficient evidence to prove that a thing which should work in theory is, in fact, working. Should we push through that? There are several reasons why there might not be evidence. We could be measuring the wrong things, we could be measuring the right things in the wrong way, or maybe it actually just doesn’t work. But a cornerstone of the scientific method is the null hypothesis: The assumption that nope, that drug doesn’t cure cancer; nope, the Higg Boson isn’t there; and nope, the policy doesn’t work. The null hypothesis is the default – like “innocent until proven guilty”, it’s presumed that things don’t cure cancer until it’s proven that it does. It’s a great standard for science, but it’s a catastrophic principle to apply to government. It’s no coincidence that Big Tobacco are great fans of “evidence-based policy”. Like in this 2012 excerpt from former Philip Morris spokeperson, now National MP, Chris Bishop:
“We support evidence-based regulation of all tobacco products. In particular, we support measures that are effective in preventing young people from smoking. Plain packaging fails this standard because it is not based on sound evidence and will not reduce youth smoking.” (emphasis added)When Bishop said that it wasn’t based on sound evidence, he meant that it was only based on experimental evidence, rather than real world empirical evidence – which didn’t exist because plain packaging wasn’t in place, which shouldn’t go ahead because there was no evidence, which didn’t exist because… and so on. The idea is that nothing ought to be done until evidence shows it really works. Like “it’s better that ten guilty people go free than let one innocent suffer”… except “it’s better that 5000 people per year die of smoking than one innocent intellectual property right suffer”. The tobacco industry’s (mainly in Australia) position kept shifting as the evidence started to come in: “nothing happened in the first year so we’re right”, “sure you have evidence but we commissioned this other piece of evidence which says we’re right”, through to, as always “evidence schmevidence, it’s always been about the principle of intellectual property rights”. So, surprise! They weren’t really interested evidence-based policy after all. The problem isn’t that the tobacco industry is self-interested. It’s that when we pretend governance is science, we’re creating a bias towards accepting the null hypothesis, towards doing nothing. But governance isn’t science. Science is about challenging, acquiring, and testing knowledge. It loses nothing from being uncertain, but false knowledge can lead science astray for years, even centuries. That’s why the scientific method is skeptical and conservative by design. Governance is about making decisions with imperfect knowledge. They have to be best guesses because inaction can be as catastrophic as incorrect action, and because sometimes solid knowledge is hard to come by. When we use the language of science for governance, we’re setting up a hair-trigger (“but there’s no evidence for that”) that favours doing nothing. The best case scenario is that it’s a sincerely inquisitive process that could be abused by bad actors to keep us locked into inaction. The worst case scenario is that this is inaction by design, that it is ideologically driven Conservatism trying to hide behind the language of scientific conservatism.