Talk to forum on Evidence-Based Regulatory Reform
Monash University Law Chambers, December 6, 2010
Some of the most important ideas are deceptively simple. Who, for instance, could dispute that the practice of medicine, the making of government policy or the reform of government regulation should be ‘evidence-based’? I have a mate who’s an academic neurologist and when we’re at our holiday homes and getting our daily exercise toiling up and down a mountain he often tells me about the advent of evidence-based medicine. I was surprised to learn that many of the cures prescribed by doctors often aren’t chosen on the basis of hard, scientific evidence, but rather on the doctor’s habits, on what he was taught at med school, on the doctor’s impression of what other doctors do, on years of casual observation as to what works and what doesn’t, on what might minimise the doctor’s ‘medico-legal’ risk, on what the doctor remembers reading in a medical journal, or even what the doctor was told recently by a visiting drug company rep. In place of this, the medical profession is slowly and painstakingly assessing what it knows from high-quality empirical studies about the efficacy of particular treatments, with treatments and tests graded, starting with the ‘gold standard’ and working down. Doctors who follow these guidelines can expect to avoid much criticism should things go amiss. The role of professional judgment is reduced, but the effectiveness of treatment is greatly enhanced.
Now, who among us could doubt that making medicine more ‘evidence-based’ is a great step forward? I was pondering the idea of writing a radical column advocating that economists and econocrats follow the doctors’ example and make their policy recommendations more evidence-based - less reliance on theory, more on empirics - when I was amazed to see the chairman of the Productivity Commission had given a long speech advocating ‘evidence-based policy-making’. The boss of the most notoriously theoretically pure economic institution is preaching sermons about the need to take more notice of the actual evidence? And I have no doubt that, in his own mind, his commitment to taking more notice of evidence was completely genuine.
The point I want to make is that, while no one could disagree with the proposition that everything we do ought to be more evidence-based - perhaps even choosing our spouse - in practice it ain’t that simple. What I regard as evidence may not be what you regard as evidence. And the conclusions I draw from the evidence may not be those you draw from the same evidence. It may transpire that your response to the evidence was sound and mine wasn’t. But both of us will be able to claim our decisions were evidence-based. So making evidence-based decisions may improve the quality of those decisions on average, but it offers no guarantee that decisions are correct. And even the expectation that paying more attention to evidence will generally improve the quality of decisions should itself be subject to evaluation once we have more evidence about the effectiveness of evidence-based decision-making.
‘Evidence’ is a word with a wonderful air of certainty and assurance. Who could doubt the evidence? Who could argue with the evidence? Yet we know from watching telly that this is just what barristers are paid to do: argue about different interpretations of the evidence. And far more lowly paid juries have the last word on what the evidence proves. We need to think more clearly about what evidence is. I think it’s just facts. In the case of economic questions it’s often statistical facts - the statistical composition of some population; the movement in some variable over time. Maybe the movement in two variables over time, the degree of correlation between them and whether we can be sure that one caused the other, or whether both are caused by some third variable.
But the world abounds with facts, far more than we can assimilate. Some facts (and some statistics) are relevant to the question at hand and some are irrelevant. How do we decide which facts are pertinent and which aren’t? We apply our judgment. The judgment of a professional has been schooled by her professional education; she’s been trained to think along certain lines, her thinking is guided by a ‘model’. A model - which can just as easily be called a theory - is an attempt to understand a complex reality by simplifying that reality: excluding all those factors considered to be of little importance so as to focus on those factors believed to be most important in explaining how things work. All models pick and choose between the facts, all professions rely on models, and each profession’s model tends to be quite different from the others, so that, for example, a lawyer and an economist can draw quite different conclusions from the same set of facts, the same evidence. How successful each profession’s model is at explaining and predicting the way the world works depends on whether it has avoided excluding from its model factors that end up being important in explaining behaviour. In practice, all professions suffer from what I call ‘model blindness’ - a tendency to underrate factors that aren’t in their model, but turn out to be important explanators. I doubt if more emphasis on evidence will overcome the endemic problem of model blindness.
Let me make a quite different point. One reason policy-makers don’t make more use of evidence is that the evidence - particularly statistical evidence - doesn’t exist. And often it doesn’t exist because evidence is expensive to collect. So any genuine push to make decision-making more evidence-based would involve spending a lot more taxpayers’ money on collecting evidence. And collecting more evidence would annoy lots of people. Much valuable potential evidence is possessed by government agencies, but they’re often quite resistant to requests to make it available, lest they provide outsiders with a stick to beat them. Other evidence has to be collected by government agencies from business, but business greatly resents having to fill in forms for the government. It’s likely that, in the mind of business, a fair bit of the push for ‘evidence-based regulatory reform’ involves getting rid of ‘red tape’ otherwise known as evidence.
But I’m supposed to be talking about regulating for sustainability, so let me take a case study and consider the extent to which evidence could throw light on the regulatory decisions to be made. Most of us are agreed that we need to greatly reduce our emissions of greenhouse gases in the next few decades. Economists are agreed that the least-cost way to achieve this is not by prohibiting certain undesirable behaviour or by subsidising certain desirable behaviour, but by ‘putting a price on carbon’. Is this judgment evidence-based? Probably not. It’s such a new area that there isn’t a lot of evidence available, and what evidence there is would be mixed. No, this conclusion comes straight from the economists’ conventional model, which assumes that prices are always the main influence over people’s economic behaviour and that changing prices is always the least-cost way to achieve policy objectives.
But as you probably realise, there are two different ways we could put a price on carbon. We could take the cap-and-trade, emissions trading scheme approach of the Rudd government’s carbon pollution reduction scheme, and limit the quantity of carbon dioxide permitted to be emitted, thus forcing up the price of emissions. Or we could impose a tax on emissions, thus increasing their price and thereby encouraging businesses and households to reduce the quantity of their emissions. As you can see, the two approaches are essentially mirror-image and, in theory, either way should achieve the same result. From about the time the Rudd government committed itself to the trading-scheme approach, the balance of opinion among economists swung in favour of the carbon tax approach. And now with the abandonment of the Rudd scheme, but the recommitment to putting a price on carbon, the nation is facing the choice.
Was the swing in economists’ opinion in favour of a tax evidence-based? I doubt it. Could it have been? Not really; as I say, there isn’t a lot of evidence available. It does seem clear the European Union’s trading scheme has not worked well - but it’s easy to say we can see what the Europeans did wrong and will make sure we avoid making the same mistakes. So on what basis do economists divide between supporting a trading scheme and supporting a carbon tax? Because the model says they’re essentially equivalent, I think it boils down to differing judgments about their political and administrative feasibility.
We should gather more evidence and take more notice of the evidence we have, but don’t imagine this will greatly reduce the role of subjective judgment in the way governments regulate the economy.
