NSW Bureau of Crime Statistics and Research annual symposium
February 18, 2009
When I was at university many years ago I read a fascinating little Pelican by Darrell Huff called, How to Lie with Statistics. It’s proved invaluable in my career as a journalist. In talking about the uses and abuses of crime statistics I could focus on the misdemeanours of the politicians, but instead I’m going to concentrate on something I know a little about, the failings of journalists.
In June last year the bureau put out a report showing no link between the heroin shortage and the rise in the use of amphetamines such as ice. The study was reported on the front page of the MX newspaper with the banner headline ‘Users switch to ice - heroin blitz forces drug change’. MX is Murdoch’s afternoon giveaway paper - I guess you get what you pay for.
On another occasion the bureau gave a newspaper figures showing that the number of eight and nine-year olds coming to the attention of police had fallen from 130 a month to 94 a month over the two years to 2007. The bureau also told the journalist that less than 1 per cent of the population aged eight or nine had some contact with the police. The headline on the journo’s story was ‘Kid Crime Rampage’.
On a third occasion the bureau gave the media figures showing there was no upward or downward trend in knife attacks in Sydney or the rest of NSW. One newspaper’s report of this study attracted headline ‘Stabbings skyrocket as knives plague city’.
A fourth example involves a paper called the Herald. The bureau put out a report showing that the percentage of convicted offenders receiving prison sentences had risen substantially since 1993. The study also showed that prison terms for most offences had increased, as had the proportion of defendants refused bail. The headline on the Herald’s report of the study said, ‘Prison population rises despite lower jailing rates’.
I could quote more examples, but that’s enough. What I want to focus on is how this misuse of crime statistics is brought about and then why it occurs. We’ll start with the how.
I quoted headlines to you and the truth is that the headline on a story heavily influences a reader’s perception of what the story is saying. But headlines are written by sub-editors, not reporters, and sometimes there’s a gap between what the story actually says and what the headline says it says. If there is, most readers won’t notice it. Such gaps can occur for three reasons: because the hard-pressed sub doesn’t accurately comprehend what the story’s actually saying, because the reporter has left some ambiguity in his copy and the sub, who generally knows far less about the topic than the reporter, has jumped the wrong way, or because the sub knowingly writes a headline that makes the story sound more exciting than it actually is. The first two explanations - misunderstandings - are more likely to be the case on broadsheet newspapers; the third - misrepresentation - is more likely to be found in tabloid newspapers. In the case of the Herald story I quoted, the reporter focused on all those offences where the rate of imprisonment had fallen. Since he noted but didn’t highlight that, overall, rates of imprisonment had risen, this left the opportunity for the sub-editor who didn’t read the story as carefully as he should have to conclude from the early paragraphs that the overall rate of imprisonment had fallen.
The interesting question is why the reporter wrote his story in a way that encouraged that error to be made - why he’d focus on the unrepresentative falls rather than the representative rises. I’ll try to answer that when we get to the question of motive - why the media behave the way they do. Perhaps here I should remind you that journalists have to draw the essence from sometimes long and complex reports or events in just an hour or two - under pressure from bosses to make it quick and make it sexy - so it’s not surprising errors and misinterpretations occur.
Now let me give you some relevant background information. Much of the news the media publish comes to them in the form of press releases. All of the bureau’s reports, for instance, are accompanied by a summarising press release. It’s often alleged that the media are so lazy they largely publish uncritically the press releases sent to them by powerful government, business and other interests. In my experience that’s usually not the case; quite the reverse. These days most interest groups seek to use the media to advance their own interests. They employ PR people to put their own spin on the information they release to the media. Most journalists aren’t lazy and they see it as their job to get past the spin, finding the news their audience would like to know about but which the powerful interest would like to conceal. When they receive a report or a press release they think: there’s probably an interesting story in here somewhere, but I’ll have to dig for it; certainly, it won’t be the one the people who put out the press release put at the top of the release. There’s so much spin in the world that many journalists come to the conclusion that everyone’s trying to pull the wool over their eyes. You may regard the bureau as a beacon of independent truth-seeking, but I guess many journalists would suspect it’s just another government agency pumping out bromide about the receding crime wave at the behest of its political masters. There’s a saying in journalism that news is anything somebody somewhere doesn’t want you to know. My guess is that the Herald journalist in question waded through the bureau’s report until he found the bit he thought the NSW Government wouldn’t want people to know: that in the case of five significant offences, rates of imprisonment are going down not up.
Much of the misrepresentation of crime stats arises from statistical misinterpretation. You can misrepresent a time series in a host of obvious ways: by choosing a convenient time period for your comparison, by ignoring random variation (ie failing to ignore outliers), by ignoring seasonal variation (eg the number of assaults peaks in January each year and troughs in May or June), by ignoring base effects (eg saying some crime rate has doubled when it’s gone from 2 a year to 4 a year) and by ignoring the effect of police activity. For instance, when the number of arrests goes up because we’ve got more police on the job arresting people, you call it a crime wave; when the number of arrests goes down you say police aren’t out there on the job countering the crime wave.
The question is whether the journos who commit these statistical crimes are knaves or fools. I couldn’t deny there’s a lot of knavery - journos who know they’re distorting the statistics’ message, but don’t care - but there are more fools than you may imagine. Most journalists are arts-degree types with a very weak grasp on maths and little clue about how to interpret statistical information. If they did understand those things they’d be an economics editor by now. But the question goes deeper: many journalists wouldn’t be sure the diligent performance of their job required them to take account of those statistical niceties. The rules of statistical interpretation aim to ensure the user draws from the stats an accurate or representative picture of the aspect of the world the stats relate to. But that’s simply not the objective of journalism. Journalism pays no heed to the scientific method.
So let’s turn to the question of why the media misuse crime statistics and misrepresent the extent of crime. As the coppers would say, let’s look at motive. Much of the criticism of the media rests on the unspoken assumption that the media’s role is to give us an accurate picture of the world around us. We don’t have first hand experience of much of what’s happening around us and we need the media to inform us.
If that’s the role you think the media play - or should play - I have shocking news. The news media are on about news. What is news worthy? Anything happening out there that our audience will find interesting or important, although the interesting will always trump the important. Paris Hilton is interesting but of no importance; the latest change in the superannuation rules is important but deadly dull - guess which one gets more media overage?
Maybe 99 per cent of what happens in the world is of little interest: 99 per cent of the motorists who crossed the Bridge today made it without incident; someone you’ve never heard of went to work as usual and sold a new ring to someone you don’t know; Australia didn’t declare war on New Zealand . . . the list of uninteresting things that happen is endless. Journalists sort through all the things that happen looking for things they believe their audience will find interesting: the 10-car pile-up on the Bridge, Brad Pitt bought a ring for Angelina Jolie to make up after a fight, the Dutch withdrew their troops from Afghanistan.
When social scientists take a random sample they may examine the sample and discard any outliers that could distort their survey, throwing them on the floor. A journalist is someone who comes along, finds them on the floor and says, ‘these would make a great story’. I happened to be in the Herald’s daily news conference a fortnight ago on the day Kevin Rudd’s $42 billion stimulus package was announced, with all its (then) $950 cash handouts. We discussed searching for a farmer who’d get $950 because he was in exceptional circumstances, $950 because he paid tax last year, $950 because his wife also works, $4750 because he has five school-age kids, and maybe another $950 because one of the kids is doing a training course. And, of course, he’d have a big mortgage, meaning he’d also save $250 a month because of the 1 per cent cut in interest rates announced the same day. Had we found such a person and taken a good photo of him he’d have been all over our front page. The point is that we were search for the most unrepresentative person we could find. Why? Because our readers would have been fascinated to read about him. It’s reasonable to expect the media to be accurate in the facts they report but, even if they are, it’s idle to expect them to give us a representative picture of the world. And that takes me to an even more shocking thought: if the media aren’t on about giving us a representative picture of the world around us, why would journalists bother adhering to the rules of statistical interpretation? Why not highlight a quite unrepresentative statistical comparison if it happens to be the most interesting comparison?
It’s often claimed that the media focus heavily on bad news, often ignoring good news. Guilty as charged. But we do so for a simple reason: we know our audience finds bad news a lot more interesting than good news. So I’m not particularly apologetic for this state of affairs: our failings are the failings of our audience, which are the failings of human nature. Why do people find bad news more interesting than good news? As I’ve written elsewhere (SMH 12.4.2006), I believe the explanation can be found in our evolutionary history. Our brains are hardwired to perpetually scan our environment for threats, and now the chances of our being eaten by a lion have diminished we’re left with a strong appetite for bad news about, for instance, the threat of crime.
Communications research tells us we read much more for reinforcement than enlightenment. While there’s a niche market for columns that challenge the conventional wisdom, and news about some new and unexpected twist in a standard story will be found interesting, journalists know the news that goes down best is the news that confirms people prejudices. Perhaps thanks to the efforts of the media themselves, most people know as a self-evident truth that crime is increasing. Most stories about crime are intended to reinforce that belief.
Let me conclude. The media’s defence against criticism is that their failings are those of their audience; they do what they do because their audience demands it of them. But shouldn’t we hold the media to a higher standard than we hold ourselves? Yes we should. We can expect less crass commercialism and more professionalism. Doctors, for instance, don’t ask patients what disease they want to be told they have and don’t let patients pick the medicine they want prescribed.
And there’s a limit of inaccuracy and sensationalism below which market punishment sets in. Mediums that play too lightly with the truth eventually lose their credibility and their audience’s respect. This means there are checks and balances. Mediums that value their credibility - in commercial as well as ethical terms - often employ commentators who set a high store on making sure their audience isn’t misled, even when those commentators spend a fair bit of time highlighting the media’s own failings and trying to beat down some of the things that get beaten up on the front page. My guess is that, as information overload and infotainment continue to grow, at least the better-educated audience will gravitate to those journalists and journals they perceive to be committed to the search for truth. What’s more, it is possible to be truthful and interesting at the same time. So don’t slit you throat yet.
Read more >>
February 18, 2009
When I was at university many years ago I read a fascinating little Pelican by Darrell Huff called, How to Lie with Statistics. It’s proved invaluable in my career as a journalist. In talking about the uses and abuses of crime statistics I could focus on the misdemeanours of the politicians, but instead I’m going to concentrate on something I know a little about, the failings of journalists.
In June last year the bureau put out a report showing no link between the heroin shortage and the rise in the use of amphetamines such as ice. The study was reported on the front page of the MX newspaper with the banner headline ‘Users switch to ice - heroin blitz forces drug change’. MX is Murdoch’s afternoon giveaway paper - I guess you get what you pay for.
On another occasion the bureau gave a newspaper figures showing that the number of eight and nine-year olds coming to the attention of police had fallen from 130 a month to 94 a month over the two years to 2007. The bureau also told the journalist that less than 1 per cent of the population aged eight or nine had some contact with the police. The headline on the journo’s story was ‘Kid Crime Rampage’.
On a third occasion the bureau gave the media figures showing there was no upward or downward trend in knife attacks in Sydney or the rest of NSW. One newspaper’s report of this study attracted headline ‘Stabbings skyrocket as knives plague city’.
A fourth example involves a paper called the Herald. The bureau put out a report showing that the percentage of convicted offenders receiving prison sentences had risen substantially since 1993. The study also showed that prison terms for most offences had increased, as had the proportion of defendants refused bail. The headline on the Herald’s report of the study said, ‘Prison population rises despite lower jailing rates’.
I could quote more examples, but that’s enough. What I want to focus on is how this misuse of crime statistics is brought about and then why it occurs. We’ll start with the how.
I quoted headlines to you and the truth is that the headline on a story heavily influences a reader’s perception of what the story is saying. But headlines are written by sub-editors, not reporters, and sometimes there’s a gap between what the story actually says and what the headline says it says. If there is, most readers won’t notice it. Such gaps can occur for three reasons: because the hard-pressed sub doesn’t accurately comprehend what the story’s actually saying, because the reporter has left some ambiguity in his copy and the sub, who generally knows far less about the topic than the reporter, has jumped the wrong way, or because the sub knowingly writes a headline that makes the story sound more exciting than it actually is. The first two explanations - misunderstandings - are more likely to be the case on broadsheet newspapers; the third - misrepresentation - is more likely to be found in tabloid newspapers. In the case of the Herald story I quoted, the reporter focused on all those offences where the rate of imprisonment had fallen. Since he noted but didn’t highlight that, overall, rates of imprisonment had risen, this left the opportunity for the sub-editor who didn’t read the story as carefully as he should have to conclude from the early paragraphs that the overall rate of imprisonment had fallen.
The interesting question is why the reporter wrote his story in a way that encouraged that error to be made - why he’d focus on the unrepresentative falls rather than the representative rises. I’ll try to answer that when we get to the question of motive - why the media behave the way they do. Perhaps here I should remind you that journalists have to draw the essence from sometimes long and complex reports or events in just an hour or two - under pressure from bosses to make it quick and make it sexy - so it’s not surprising errors and misinterpretations occur.
Now let me give you some relevant background information. Much of the news the media publish comes to them in the form of press releases. All of the bureau’s reports, for instance, are accompanied by a summarising press release. It’s often alleged that the media are so lazy they largely publish uncritically the press releases sent to them by powerful government, business and other interests. In my experience that’s usually not the case; quite the reverse. These days most interest groups seek to use the media to advance their own interests. They employ PR people to put their own spin on the information they release to the media. Most journalists aren’t lazy and they see it as their job to get past the spin, finding the news their audience would like to know about but which the powerful interest would like to conceal. When they receive a report or a press release they think: there’s probably an interesting story in here somewhere, but I’ll have to dig for it; certainly, it won’t be the one the people who put out the press release put at the top of the release. There’s so much spin in the world that many journalists come to the conclusion that everyone’s trying to pull the wool over their eyes. You may regard the bureau as a beacon of independent truth-seeking, but I guess many journalists would suspect it’s just another government agency pumping out bromide about the receding crime wave at the behest of its political masters. There’s a saying in journalism that news is anything somebody somewhere doesn’t want you to know. My guess is that the Herald journalist in question waded through the bureau’s report until he found the bit he thought the NSW Government wouldn’t want people to know: that in the case of five significant offences, rates of imprisonment are going down not up.
Much of the misrepresentation of crime stats arises from statistical misinterpretation. You can misrepresent a time series in a host of obvious ways: by choosing a convenient time period for your comparison, by ignoring random variation (ie failing to ignore outliers), by ignoring seasonal variation (eg the number of assaults peaks in January each year and troughs in May or June), by ignoring base effects (eg saying some crime rate has doubled when it’s gone from 2 a year to 4 a year) and by ignoring the effect of police activity. For instance, when the number of arrests goes up because we’ve got more police on the job arresting people, you call it a crime wave; when the number of arrests goes down you say police aren’t out there on the job countering the crime wave.
The question is whether the journos who commit these statistical crimes are knaves or fools. I couldn’t deny there’s a lot of knavery - journos who know they’re distorting the statistics’ message, but don’t care - but there are more fools than you may imagine. Most journalists are arts-degree types with a very weak grasp on maths and little clue about how to interpret statistical information. If they did understand those things they’d be an economics editor by now. But the question goes deeper: many journalists wouldn’t be sure the diligent performance of their job required them to take account of those statistical niceties. The rules of statistical interpretation aim to ensure the user draws from the stats an accurate or representative picture of the aspect of the world the stats relate to. But that’s simply not the objective of journalism. Journalism pays no heed to the scientific method.
So let’s turn to the question of why the media misuse crime statistics and misrepresent the extent of crime. As the coppers would say, let’s look at motive. Much of the criticism of the media rests on the unspoken assumption that the media’s role is to give us an accurate picture of the world around us. We don’t have first hand experience of much of what’s happening around us and we need the media to inform us.
If that’s the role you think the media play - or should play - I have shocking news. The news media are on about news. What is news worthy? Anything happening out there that our audience will find interesting or important, although the interesting will always trump the important. Paris Hilton is interesting but of no importance; the latest change in the superannuation rules is important but deadly dull - guess which one gets more media overage?
Maybe 99 per cent of what happens in the world is of little interest: 99 per cent of the motorists who crossed the Bridge today made it without incident; someone you’ve never heard of went to work as usual and sold a new ring to someone you don’t know; Australia didn’t declare war on New Zealand . . . the list of uninteresting things that happen is endless. Journalists sort through all the things that happen looking for things they believe their audience will find interesting: the 10-car pile-up on the Bridge, Brad Pitt bought a ring for Angelina Jolie to make up after a fight, the Dutch withdrew their troops from Afghanistan.
When social scientists take a random sample they may examine the sample and discard any outliers that could distort their survey, throwing them on the floor. A journalist is someone who comes along, finds them on the floor and says, ‘these would make a great story’. I happened to be in the Herald’s daily news conference a fortnight ago on the day Kevin Rudd’s $42 billion stimulus package was announced, with all its (then) $950 cash handouts. We discussed searching for a farmer who’d get $950 because he was in exceptional circumstances, $950 because he paid tax last year, $950 because his wife also works, $4750 because he has five school-age kids, and maybe another $950 because one of the kids is doing a training course. And, of course, he’d have a big mortgage, meaning he’d also save $250 a month because of the 1 per cent cut in interest rates announced the same day. Had we found such a person and taken a good photo of him he’d have been all over our front page. The point is that we were search for the most unrepresentative person we could find. Why? Because our readers would have been fascinated to read about him. It’s reasonable to expect the media to be accurate in the facts they report but, even if they are, it’s idle to expect them to give us a representative picture of the world. And that takes me to an even more shocking thought: if the media aren’t on about giving us a representative picture of the world around us, why would journalists bother adhering to the rules of statistical interpretation? Why not highlight a quite unrepresentative statistical comparison if it happens to be the most interesting comparison?
It’s often claimed that the media focus heavily on bad news, often ignoring good news. Guilty as charged. But we do so for a simple reason: we know our audience finds bad news a lot more interesting than good news. So I’m not particularly apologetic for this state of affairs: our failings are the failings of our audience, which are the failings of human nature. Why do people find bad news more interesting than good news? As I’ve written elsewhere (SMH 12.4.2006), I believe the explanation can be found in our evolutionary history. Our brains are hardwired to perpetually scan our environment for threats, and now the chances of our being eaten by a lion have diminished we’re left with a strong appetite for bad news about, for instance, the threat of crime.
Communications research tells us we read much more for reinforcement than enlightenment. While there’s a niche market for columns that challenge the conventional wisdom, and news about some new and unexpected twist in a standard story will be found interesting, journalists know the news that goes down best is the news that confirms people prejudices. Perhaps thanks to the efforts of the media themselves, most people know as a self-evident truth that crime is increasing. Most stories about crime are intended to reinforce that belief.
Let me conclude. The media’s defence against criticism is that their failings are those of their audience; they do what they do because their audience demands it of them. But shouldn’t we hold the media to a higher standard than we hold ourselves? Yes we should. We can expect less crass commercialism and more professionalism. Doctors, for instance, don’t ask patients what disease they want to be told they have and don’t let patients pick the medicine they want prescribed.
And there’s a limit of inaccuracy and sensationalism below which market punishment sets in. Mediums that play too lightly with the truth eventually lose their credibility and their audience’s respect. This means there are checks and balances. Mediums that value their credibility - in commercial as well as ethical terms - often employ commentators who set a high store on making sure their audience isn’t misled, even when those commentators spend a fair bit of time highlighting the media’s own failings and trying to beat down some of the things that get beaten up on the front page. My guess is that, as information overload and infotainment continue to grow, at least the better-educated audience will gravitate to those journalists and journals they perceive to be committed to the search for truth. What’s more, it is possible to be truthful and interesting at the same time. So don’t slit you throat yet.
