I’ve written in the past about the problem of bad science reporting in the US. The short version: very few American reporters have enough grounding in statistics and the sciences to accurately parse the claims of quantitative research, and as a result they often misrepresent what studies actually say.
This is an indictment of, among other things, university journalism school curricula, which ought to be requiring at some basic coursework in statistical analysis and scientific method. That prominent news agencies aren’t doing enough to remedy the failings of the J schools is obvious and distressing. Americans generally don’t understand stats and science any better than the average cub reporter, so bad reporting can actually do material damage in the lives of those who walk away from the newspaper or evening news with inaccurate conclusions.
A new report on autism rates and the preservative thimerosal has again put this problem in the national spotlight, although with better results overall than we might have expected or even hoped for. Let’s begin with what the researchers themselves said:
“The hypothesis that a modifiable risk factor, such as thimerosal exposure, is a major cause of autism offers the hope for prevention through reduced exposure,” the authors conclude. “Although our analysis of Department of Developmental Services data shows an increase in autism in California despite the removal of thimerosal from most vaccines, we support the continued quest for the timely discovery of modifiable risk factors for autism and related conditions. Continuing evaluation of the trends in the prevalence of autism for children born in recent years is warranted to confirm our findings.”
A quick caveat. I am not asserting a link between thimerosal and autism, nor am I indicting this new research, which seems entirely credible and helpful. I am primarily concerned with the reporting of the results.
This report in the Canadian Press does a pretty good job of getting the story right. (I know, Canada isn’t in the US, but we share a lot of the same issues, so I’m including this report as a mostly good example.)
Autism cases continue to rise after preservative thimerosal removed from vaccine
LOS ANGELES – Autism cases in California continued to climb even after a mercury-based vaccine preservative that some people blame for the neurological disorder was removed from routine childhood shots, a new study found.
Researchers from the state Department of Public Health found the autism rate in children rose continuously during the 12-year study period from 1995 to 2007. The preservative thimerosal hasn’t been used in childhood vaccines since 2001, but is used in some flu shots.
The conclusion offered by the researchers in the next graf is perhaps problematic, though.
Doctors say the latest study adds to existing evidence refuting a link between thimerosal exposure and autism risk and should reassure parents that the disorder is not caused by vaccinations. If there was a risk, they said, autism rates should have dropped between 2004 and 2007.
This is an intuitively convincing conclusion, and is probably accurate. However, I think you could perhaps construct a hypothetical alternate explanation positing complex interactions of multiple factors that would allow you to question whether it’s technically correct to state the results in this fashion. Stick with me, because I’ll present an example of this below.
Let’s look at some publications that muffed the story
Intellihealth.com gaks it completely, despite the fact that the actual findings are clearly contained in the story:
No Thimerosal-Autism Link
The mercury-based preservative thimerosal does not appear to be linked to autism. A study published in the Archives of General Psychiatry looked at autism cases diagnosed in California between 1995 and 2007 and found that the number continued to grow even after thimerosal was removed from vaccine formulations in 2001. In the study, state public health officials used a database of state-funded centers for people with autism and other developmental disorders to calculate the rate of the neurological disease. They found that the prevalence of autism in children aged 3 to 12 grew throughout the study period. The study did not determine why autism cases increased. The researchers say the findings suggest that scientists should explore other possible causes of autism, such as genetic links, The Associated Press reports.
The header is patently inaccurate, although if you added the word “found” to the end it would be correct. The lead then botches the results even further, because saying “we found no link between A and B” is different from saying “A and B aren’t linked,” and the researchers did not and could not have said the latter.
Science Daily does a little better: “Autism cases continued to increase in California after the mercury-containing preservative thimerosal was eliminated from most childhood vaccines, according to a new report. This suggests that exposure to thimerosal is not a primary cause of autism.” This is a defensible interpretation, depending on what you do with the word “primary.” However, as we’ll see below, there’s ample room for a science-savvy reporter to have interrogated the conclusion more thoroughly.
Then there’s WTRF-TV. I have no idea what these people are thinking. Their Web site draws on a fairly good CBS News report, but leads into with a boldfaced statement of utter fiction: “There’s no connection between mercury and Autism…”
The San Mateo Times opted for the splashy and inaccurate approach, as well, leading with the headline “Vaccine ruled out as autism cause” despite the fact that the findings themselves say no such thing.
Why you can’t say there’s no link
When we know a lot more about this complex issue it may be clear that thimerosal has nothing to do with autism and never did, and based on the data before us today you might be safe enough making that bet now. However, science and research are conducted according to stringent rules about what you can claim from a piece of research, and those rules matter a great deal here. For starters, it’s damned near impossible to prove anything, especially a negative. So demonstrating that there is no relationship is tricky, at best.
As for the autism case specifically, let’s look at a hypothetical that illustrates the difficulty in knowing what exactly is at work with thimerosal. We don’t know what causes autism, although it seems likely that there are multiple factors involved (including new research finding a chromosomal flaw that could open the door to greater understanding about possible genetic causes).
For the sake of argument, let’s posit that autism is caused by three factors: A is a controllable factor like thimerosal; B is a genetic factor; and C is an artifact of public awareness and medical diagnostic trends (that is, we’re learning more about the disease and as a result people who wouldn’t have been classified as autistic 20 years ago are today, meaning that real numbers could be steady while diagnoses climb).
We study a five-year period in which A is eliminated, but diagnosed cases continue to increase. This may mean that A is not a factor. However, it could also mean that A is a factor, but that the genetic trigger is spreading through the population rapidly while C is also occurring – industry dynamics are driving an increase in the rate of diagnoses. In this case, the increases associated with B and C might more than mask the decrease associated with A, which would mean that A is a trigger and that had it not been eliminated the observed increase would have been even higher.
Unless these kinds of factors can be controlled, it is not possible for a study to state that A is not a factor, although educated researchers are more than capable of offering informed speculation – which the scientists in this study do.
There is good news on the reporting front
A lot of places got the story right. WNCT.com, and AP reporter Alicia Chang does a nice job. Wired and the Star-Ledger adopt the “primary cause of autism” language in reports that are at least as on-point as the researchers’ own words. So I’m pleasantly surprised to see a number of outlets doing a pretty good job on a complex story.
We’ve seen plenty of bad reporting on research – here’s hoping those who did a good job on this story represent the coming trend instead of a momentary aberration.
Categories: Journalism, Media/Entertainment, Science/Technology
Progressive Culture | Scholars & Rogues 
Nicely done!
I once put together a training package to sell to editorial management of, mostly, newspapers (though other news outlets were welcome to buy, as well). The training was on simple statistics, research techniques, samlings, etc, and how to discern if some study was trying to manipulate journalists to report inaccurate findings/conclusions. Having worked at a firm where the research department’s sole purpose was to do low-cost, non-valid surveys to get ink in places like the Wall Street Journal (which they regularly did), I thought the product would be snapped up.
I got not a single taker. Not one. Puzzled, I called a few newspapers to ask why. I was told that validity didn’t matter. If someone said something, and it seemed interesting, they would report it. Since the paper was reporting accurately that someone else said it, they had no ethical responsibility to probe further.
Woe. Woe.
I’m not surprised that the newspapers said that. I AM surprised that you wasted your time trying. Must have been a younger, more naive man than you are now…. 🙂
Well, Doc, I really don’t think it’s all that naive to believe that most people don’t go to work every day thinking, “How can I do a bad job and screw up today.”
Newspapers may be an exception.
Sorry Denny.
No, the naive part is thinking that they identify this behavior as screwing up. They really don’t. In some cases they simply don’t see it as a big deal – a lot of reporters would read this piece and dismiss me as a egghead buried in meaningless minutiae.
You have no idea how close to home this hits. Excellent work.
Thanks. In addition to the obvious reporting issues, I’m close to people who have autistic children and siblings, so while I’m hardly an expert on that subject I do notice when stories come across the transom and I tend to care about them being right.
In “News & Numbers: A guide to reporting statistical claims and controversies in health and other fields,” authors Lewis Cope and the late Victor Cohn identify four principal problems beyond the lack of basic training in statistics that Sam bemoans:
“We sometimes overstate and oversimplify.”
“We work fast, sometimes too fast, with severe limits on the space or airtime we may fill.”
“We too often omit essential perspective, context, or background.”
“We are influenced by intense competition and other pressures to tell the story first and tell it most dramatically.”
That was written nearly 20 years ago. None of those factors has improved.
Cope and Cohn’s book addresses provides precisely the kind of knowledge and analytical skills needed to allow a good reporter to tell a story housed in science. Newspaper editors can buy it at Amazon for $35.99 a pop. They could pay for their reporters’ memberships in the Society for Environmental Journalism ($45), the National Association of Science Writers ($75) and Investigative Reporters & Editors ($60). They could send their reporters to workshops and conventions.
It’s not that expensive to provide good reporters with the kind of tools Sam is arguing for. But good science writing is not prized by editors of American newspapers any more, because it fails to draw sufficient advertising support.
There was a time when more than 100 newspapers published weekly full-page science sections.
Where are they now? If the newspaper business wants to defend itself against charges it has dumbed down discourse, it should not trot out its performance in science writing.
After all, it hasn’t even addressed the four problems with which Cope and Cohn opened their book.
Denny says: “None of those factors has improved.”
Congratulations. You win the award for understatement of the week.
As someone who’s blogged on science topics and garnered precious few eyeballs (Mmmmm, eyeballs…) in the process, I understand the publishers decisions. It sucks, and I think it’s the wrong decision, but I do understand it.
I wonder how much of that is related to a public that doesn’t want to read, that wants everything right-now-if-not-a-minute-ago, and wants to be pandered to. Science reporting doesn’t lend itself to pandering – it takes time and energy to understand what you’re reading. Sometimes you have to go out and do some research before you understand everything the reporter’s telling you (although the best writing gives you everything you need to understand it at at least a basic level), and it takes time and energy to write it up effectively.
In our current culture, time is money and research is money and the reporters’ tools are money, and we’ve got to keep those profit margins above 25%. That leaves us with people like me and innumerable other bloggers who do this stuff for free out of love or obsession and the rare newspaper that can afford to have science reporters on staff (like Andrew Revkin at the NYTimes).
Well, folks, it seems to me that we’re hardly talking about JUST hard science writing, here. The media puslish all kinds of things about the social sciences that’s just plain dumb. For instance, do you remember when Newsweek published a survey with the line, “A woman is more likely to be killed by a terrorist than to be married after age 30.” Or when Shere Hite’s egregiously awful research on sex in the office generated headline about how “70% of people reported having sex at work.”
I don’t know about you, but I started to wonder what was wrong with me that I wasn’t getting invited.
All it takes to get a paper to print ridiculously awful “research” is a press release. You send it, they print it. PR firms everywhere know this.
I try not to be evil, but in my career I sometimes do that PR thing. And knowing what I know about these issues makes it easier to play reporters. Like banjos.
It’s bad enough having to fight my own ethical battles, but the state of journalism these days means I sometimes have to find ways of spotting the reporter a few points just to make it semi-fair.
The issues not addressed: vaccine providers are allowed to use up their stocks of thimerosal, which I understood is several years worth, and when checking I found some offices had trouble getting enough thimerosal free vaccine or only provided them when asked specifically. Both of these problems cloud how much exposure remains and how valid any assumption is so soon after the technical cut off date.
A good observation by Anonymous in #12. The effects of mercury in children’s vaccines also may not be an immediate trigger, but may become apparent only after several years.
Also, and this has not been mentioned in anything I have seen: Does the removal of Thimerosal help to isolate the real source, which is the vaccine itself. In particular, the MMR combined “cocktail” of vaccines that is being pushed as the most economical way to get children vaccinated has also been identified as a possible source. There are other factors as well that may be at play, such as the fact that the whole series of shots required or recommended for babies, infants and toddlers (close to 40 by age 5?) may be pushing those small bodies past a threshold that triggers autism. To immediately fall back on”genetic” sources as so many of the reports did seems an almost knee-jerk response to exonerating the pharmaceutical industry for their production and marketing practices.
Incidentally, the MMR vaccine can be administered in three separate shots, requiring three office visits and a lot of hassle to get the doctor to order them and then not being reimbursed by insurance because it is not covered that way in most cases.
Don’t forget that even without mercury in the vaccines, kids (all of us, really) are being exposed to more and more mercury all the time.. Fish are begining to be fairly toxic, and that’s a problem.
While they might try to say “Thimerosal” isn’t doing it, to help boost the sales of drugs by Big Pharma, that doesn’t mean there’s no mercury link (some are suggesting it’s purely genetic). By pretending that a specific derivative “might not” have anything to do with a specific condition only serves to benefit those looking to push a specific vector… That doesn’t help society at large.
I’ve also not read the study or the methods taken, but I wonder if the “Thimerosal free” vaccines used were actually -tested- for the presence of the vector or if it was taken at the “word” of the companies providing the chemicals. Testing for variables is something -much- different than letting the fox tell you he’s not really eating a chicken.
In journalism, nothing has more value than the truth. So the overall value of journalism is eroded every time the journalist swings for the fences at the expense of getting it right. To say, “There’s no connection between mercury and Autism” sounds like a scoop. It makes some people right and some people wrong. It’s definitive. It’s the journalistic equivalent of staring down that guy in whose face you just dunked. But that doesn’t mean the journalist has done his or her job. Sometimes that requires nuance. And who needs that? Except maybe all of us.
Personally, I’d like nothing better than for someone to come along and either completely debunk the Thimerosal/Mercury/Autism link or prove it beyond a doubt. But until then I hope the information is disseminated professionally and responsibly.
This article highlights the tip of several icebergs, perhaps the most important of which is the way journalists are being unwittingly used by not only Big Corps. as PR agents, but by their own paper’s parent Corp. Can you imagine the reaction of the owners’ of a large rag which printed the above story more truthfully (“the data from CA as presented do not seem consistant with a thimerosal-autism link, but further study will be needed to clarify the point”), knowing full well that not only do the makers of vaccines contribute heavily to the advertising budget, but also that any story which paints the sacred cow of vaccination in a questionable light will offend the rank and file of most professional orgs in the field of medicine as well as tick off the people in DHHR?
In the words of the infamous F Lee Bailey, “Let’s not cloud this issue with the facts”.
The contention that autism cannot be caused by thimerosal in vaccines, because it has been removed from them and the rates of autism in California have not decreased with that removal, would be comforting except for one very uncomfortable fact. Thimerosal has not, actually, been removed from the vaccines that children are receiving. I recently did a bit of research into the question of thimerosal content in flu vaccines. While the information I was able to obtain was limited, the convoluted answers that I received were enlightening in and of themselves. I make no claims as to the scientific validity of my blog, which I decided to call Adventures at the Flu Fair after seeing flu shot clinics being marketed as ‘Fairs’
http://www.wideopenwest.com/~r_nemeth/clinic_timeline.htm
CNN claims to have obtained an answer to this question
http://www.cnn.com/2007/HEALTH/10/31/flu.hm.flu.shot/index.html
and they say that only six percent of individual doses of flu shots for this season are ‘mercury free’. If one is to presume that the rest have levels of thimerosal that could be of concern, then, eight long years after the Department of Health and Human Services said that thimerosal should and would be removed from vaccines, the vast majority of flu vaccines in the US still contain the mercury preservative.
Most of the articles I read have a great deal to say about the influence of genes on the development of autism. I am not surprised that genes would affect the development of this devastating condition–they have some affect on how each and every one of us reacts to each and every thing we are exposed to in our environment.
I believe that the incidence of autism in all of it’s forms has skyrocketed in recent decades—I’m pretty sure even the CDC no longer denies this. What the article fails to address is that it is impossible to explain such a ballooning in incidence simply by looking at genetic factors, and not environmental ones as well. (If the environment seems to be affecting genes, journalists these days seem to me to be loath to write the word ‘environment’, but will merely refer to a genetic cause.) If I remember my high school biology correctly, genes are combined, male and female, and passed on from one generation to the next, which means that the process of change in populations is a slow one that can occur only with the passage of many generations. This spike in autism that has parents so concerned right now, however, has occurred in the space of a decade and a half. Such swift change is not possible only via the natural recombination of genes that occurs when two parents have sex. It seems to me that some kind of damage is being done to the genetic makeup, or simply damage to the organism.
JS O’Brien posted here, about an experience he had with a newspaper journalist, ‘Since the paper was reporting accurately that someone else said it, they had no ethical responsibility to probe further.’
Here’s some reporting on the Karen McCarron story, by the Associated Press, that I find particularly interesting (http://ap.google.com/article/ALeqM5gzi4G83F97PxaZn6ctuLeZ9l5tkwD8U63A0O1):
‘McCarron said she ignored God’s warnings by listening to doctors and having Katie vaccinated, then believed the vaccinations caused the autism, said Glenmullen, who works at Harvard University and came to her conclusion by reading another psychiatrist’s interviews, reviewing medical records and hearing other testimony.’
Does Karen McCarron actually believe that God warned her to kill her child? I don’t know, I can’t tell from this story, but I’m willing to bet that a lot of people will jump to that conclusion. I would love to be a fly on the wall in that courtroom, though.
Robin – According to reports I’ve read, mercury-based preservatives are only allowed to be used in flu vaccines, rather than in nearly every vaccine made in the 1990s. So yes, you’re right that the claims Dr. Slammy are talking about are erroneous, but it actually strengthens his original argument – that most of the journalists and pundits reporting on this subject have neither studied their science nor done their due diligence to ensure they got their facts correct.
Here’s some more interesting reporting. This quote
McCarron, a former pathologist, testified she felt responsible for Katie’s autism because she allowed the child to get vaccinated.
is one that I found in a January 12 Associated Press story on this website
http://ap.google.com/article/ALeqM5gzi4G83F97PxaZn6ctuLeZ9l5tkwD8U453JG0
The AP story also talks of a videotaped confession that was made of McCarron sitting next to her husband in the hospital after a suicide attempt. It is during that confession that her words about vaccines were allegedly spoken.
Good luck trying to find any mention of the word ‘vaccine’ on google, today, though, if you do either a web or a news search on the words ‘karen+mccarron’. I don’t know, maybe there’s something wrong with my browser.
i think more money needs to go towards autism than anything else.