I’ve written in the past about the problem of bad science reporting in the US. The short version: very few American reporters have enough grounding in statistics and the sciences to accurately parse the claims of quantitative research, and as a result they often misrepresent what studies actually say.
This is an indictment of, among other things, university journalism school curricula, which ought to be requiring at some basic coursework in statistical analysis and scientific method. That prominent news agencies aren’t doing enough to remedy the failings of the J schools is obvious and distressing. Americans generally don’t understand stats and science any better than the average cub reporter, so bad reporting can actually do material damage in the lives of those who walk away from the newspaper or evening news with inaccurate conclusions.
A new report on autism rates and the preservative thimerosal has again put this problem in the national spotlight, although with better results overall than we might have expected or even hoped for. Let’s begin with what the researchers themselves said:
“The hypothesis that a modifiable risk factor, such as thimerosal exposure, is a major cause of autism offers the hope for prevention through reduced exposure,” the authors conclude. “Although our analysis of Department of Developmental Services data shows an increase in autism in California despite the removal of thimerosal from most vaccines, we support the continued quest for the timely discovery of modifiable risk factors for autism and related conditions. Continuing evaluation of the trends in the prevalence of autism for children born in recent years is warranted to confirm our findings.”
A quick caveat. I am not asserting a link between thimerosal and autism, nor am I indicting this new research, which seems entirely credible and helpful. I am primarily concerned with the reporting of the results.
This report in the Canadian Press does a pretty good job of getting the story right. (I know, Canada isn’t in the US, but we share a lot of the same issues, so I’m including this report as a mostly good example.)
Autism cases continue to rise after preservative thimerosal removed from vaccine
LOS ANGELES – Autism cases in California continued to climb even after a mercury-based vaccine preservative that some people blame for the neurological disorder was removed from routine childhood shots, a new study found.
Researchers from the state Department of Public Health found the autism rate in children rose continuously during the 12-year study period from 1995 to 2007. The preservative thimerosal hasn’t been used in childhood vaccines since 2001, but is used in some flu shots.
The conclusion offered by the researchers in the next graf is perhaps problematic, though.
Doctors say the latest study adds to existing evidence refuting a link between thimerosal exposure and autism risk and should reassure parents that the disorder is not caused by vaccinations. If there was a risk, they said, autism rates should have dropped between 2004 and 2007.
This is an intuitively convincing conclusion, and is probably accurate. However, I think you could perhaps construct a hypothetical alternate explanation positing complex interactions of multiple factors that would allow you to question whether it’s technically correct to state the results in this fashion. Stick with me, because I’ll present an example of this below.
Let’s look at some publications that muffed the story
Intellihealth.com gaks it completely, despite the fact that the actual findings are clearly contained in the story:
No Thimerosal-Autism Link
The mercury-based preservative thimerosal does not appear to be linked to autism. A study published in the Archives of General Psychiatry looked at autism cases diagnosed in California between 1995 and 2007 and found that the number continued to grow even after thimerosal was removed from vaccine formulations in 2001. In the study, state public health officials used a database of state-funded centers for people with autism and other developmental disorders to calculate the rate of the neurological disease. They found that the prevalence of autism in children aged 3 to 12 grew throughout the study period. The study did not determine why autism cases increased. The researchers say the findings suggest that scientists should explore other possible causes of autism, such as genetic links, The Associated Press reports.
The header is patently inaccurate, although if you added the word “found” to the end it would be correct. The lead then botches the results even further, because saying “we found no link between A and B” is different from saying “A and B aren’t linked,” and the researchers did not and could not have said the latter.
Science Daily does a little better: “Autism cases continued to increase in California after the mercury-containing preservative thimerosal was eliminated from most childhood vaccines, according to a new report. This suggests that exposure to thimerosal is not a primary cause of autism.” This is a defensible interpretation, depending on what you do with the word “primary.” However, as we’ll see below, there’s ample room for a science-savvy reporter to have interrogated the conclusion more thoroughly.
Then there’s WTRF-TV. I have no idea what these people are thinking. Their Web site draws on a fairly good CBS News report, but leads into with a boldfaced statement of utter fiction: “There’s no connection between mercury and Autism…”
The San Mateo Times opted for the splashy and inaccurate approach, as well, leading with the headline “Vaccine ruled out as autism cause” despite the fact that the findings themselves say no such thing.
Why you can’t say there’s no link
When we know a lot more about this complex issue it may be clear that thimerosal has nothing to do with autism and never did, and based on the data before us today you might be safe enough making that bet now. However, science and research are conducted according to stringent rules about what you can claim from a piece of research, and those rules matter a great deal here. For starters, it’s damned near impossible to prove anything, especially a negative. So demonstrating that there is no relationship is tricky, at best.
As for the autism case specifically, let’s look at a hypothetical that illustrates the difficulty in knowing what exactly is at work with thimerosal. We don’t know what causes autism, although it seems likely that there are multiple factors involved (including new research finding a chromosomal flaw that could open the door to greater understanding about possible genetic causes).
For the sake of argument, let’s posit that autism is caused by three factors: A is a controllable factor like thimerosal; B is a genetic factor; and C is an artifact of public awareness and medical diagnostic trends (that is, we’re learning more about the disease and as a result people who wouldn’t have been classified as autistic 20 years ago are today, meaning that real numbers could be steady while diagnoses climb).
We study a five-year period in which A is eliminated, but diagnosed cases continue to increase. This may mean that A is not a factor. However, it could also mean that A is a factor, but that the genetic trigger is spreading through the population rapidly while C is also occurring – industry dynamics are driving an increase in the rate of diagnoses. In this case, the increases associated with B and C might more than mask the decrease associated with A, which would mean that A is a trigger and that had it not been eliminated the observed increase would have been even higher.
Unless these kinds of factors can be controlled, it is not possible for a study to state that A is not a factor, although educated researchers are more than capable of offering informed speculation – which the scientists in this study do.
There is good news on the reporting front
A lot of places got the story right. WNCT.com, and AP reporter Alicia Chang does a nice job. Wired and the Star-Ledger adopt the “primary cause of autism” language in reports that are at least as on-point as the researchers’ own words. So I’m pleasantly surprised to see a number of outlets doing a pretty good job on a complex story.
We’ve seen plenty of bad reporting on research – here’s hoping those who did a good job on this story represent the coming trend instead of a momentary aberration.