Science Blog: Clinical trials: Unfavorable results often go unpublished

I have a continuing interest in patients understanding the statistics that are used to recommend treatments to doctors - and thus to us. So I was intrigued by this post Wednesday at Science Blog:

Clinical trials: Unfavorable results often go unpublished

Trials showing a positive treatment effect, or those with important or striking findings, were much more likely to be published in scientific journals than those with negative findings, a new review from The Cochrane Library has found.

"This publication bias has important implications for healthcare. Unless both positive and negative findings from clinical trials are made available, it is impossible to make a fair assessment of a drug's safety and efficacy," says lead researcher, Sally Hopewell of the UK Cochrane Centre in Oxford, UK.

The international team of researchers carried out a systematic review of all the existing research in this area. In addition to showing that negative results were published less often, they found that if these results were eventually published, they would take between one and four more years to appear in journals than studies showing positive results. ...

As often happens, there's more to learn from the comments than in the original post:

"No one will publish a paper about an experiment that gave negative results. The problem is that negative results could as important as positive ones (so maybe other researcher won't try the same thing again, for example). I hope the publication industry will soon disappear, and that the strengths and paradigm of the Internet will finally be used also for scientific articles."

"The worst part of this very understandable human trait to publish only successes is this: How can we learn from failures if we never hear about them?"

"I think you could argue that publication bias negatively affects the entirety of scientific research, not just clinical trials. As someone who works in the business, I wanted to mention the PHARMA Code, the code of ethics for the industry. Regarding publication it's pretty clear: you must ATTEMPT to publish findings, significant or not. Now, whether a journal editor wants to publish non-significant results is another story entirely."

"I think it is unfair to blame the journal editors. ... The researchers (in any grant-dependent field) may be required by a grantor or a code of ethics to attempt publication, but my guess is that they don't work very hard to produce a high quality manuscript when all they have to discuss are unsuccessful trials or nul results."

My takeaway: it's even sketchier than I thought to presume "If it was important information, we'd have read about it." The process of preparing and publishing articles is fraught with potholes and pitfalls.

I'm not saying we should ditch journals. We should, though, be conscious of what they are and aren't. Certainly not an inherently authoritative source of information – despite the best efforts of their editors.

p.s. Where did I learn of that post? In an online patient-to-patient community – patients empowering and informing each other. Gotta love the Internet and e-patients!

"Health reporters should have higher standards" (Sci.Am.)

Scientific American has a good new post, Health reporters should have 'higher standards,' commentary says. It's about a piece in today's New England Journal of Medicine on the "Pitfalls of Health Care Journalism" by NPR's Susan Dentzer.

Well folks, you heard it here first. :)

11/15/08: Making Sense of Health Statistics

11/17/08: Part 2

12/1/08: HealthNewsReview.org: a great learning resource

Just between us, I can't tell you how pleased I am that having just started this exploration in February, I've gotten to the point where I'm noticing (and writing about) issues when (or even before) the big cats do.

Florence Nightingale, the passionate statistician

From Science News:

When Florence Nightingale arrived at a British hospital in Turkey during the Crimean War, she found a nightmare of misery and chaos. Men lay crowded next to each other in endless corridors. The air reeked from the cesspool that lay just beneath the hospital floor. There was little food and fewer basic supplies.

By the time Nightingale left Turkey after the war ended in July 1856, the hospitals were well-run and efficient, with mortality rates no greater than civilian hospitals in England, and Nightingale had earned a reputation as an icon of Victorian women. Her later and less well-known work, however, saved far more lives. She brought about fundamental change in the British military medical system, preventing any such future calamities. To do it, she pioneered a brand-new method for bringing about social change: applied statistics.

HealthNewsReview.org: a great learning resource

My volume of recent posts on evidence and statistics (here, here, here, here) indicates how important I think it is that e-patients 'E'ducate themselves about 'E'vidence, as Sarah Greene recently wrote. Last week Ted Eytan steered me to HealthNewsReview, which scrutinizes news coverage of health stories. Newspapers and TV are covered.

When you're new to a subject, as I am to scrutinizing evidence, there's nothing better than this to help you learn. A few examples:

Elderly fare well in open-heart surgery (Associated Press) got two stars of a possible five:

This story failed to provide balance for a reader to understand the risks and benefits of this line of treatment, what other options are available, or the costs involved. Other than gaining the insight that the outcomes are better today than they were in 1989, the reader did not learn much beyond the fact that surgical treatment of coronary conditions may be an option for those in their 80s and 90s. Full review

A single test to detect many winter ailments (Wall Street Journal) got a perfect five stars:

... a good job of presenting accurate, comprehensive information ... the test is presented in context of a key health issue: the under-diagnosis of the flu in vulnerable populations, such as children, versus the inappropriate and unneeded prescription antibiotics in cases where they are ineffective in treating the flu. This story takes a balanced approach in presenting the evidence supporting the pros and cons of the test. It could have been improved by stating that it is not necessary for most routine febrile illnesses in the outpatient setting. Full review

A search for keyword "statins" shows that almost all stories on statins (at least the ones the site reviewed) are poor. Example, from the LA Times 11/10/08, Statins may benefit healthy people too:

This story fails to be skeptical about claims of self-interested researchers. Rather than pushing back against exaggerated claims of benefits, safety and guideline changes, the story magnifies them.

Another review I spotted dinged an article for talking only about relative risk reduction, not raw numbers. That's consistent with the Making Sense paper, as I wrote earlier.

Ted Eytan is a pretty nifty MD. Clearly believes in teaching patients anything they want to know about.

Making Sense of Health Statistics, part 2

Well, wouldn't you know it??

A perfect example of Saturday's post just arose: Today’s NY Times discusses a “large new study” of Crestor, a statin, involving 17,800 patients. Well, let's take what we learned the other day.

The Times editorial reports that Crestor has dramatic benefits - 54% fewer heart attacks, etc. And the writer correctly asks, “Who should take statins?”

But these “relative risk reduction” numbers (percent reduction) are exactly what Making Sense warns against: what are the raw numbers? We don't know, so from what they wrote, we can't tell whether there's improvement for one person in five or one person in 5,000.

This is not to say we shouldn’t use statins. The whole point is that the Times piece doesn’t give us enough information to know.

And Making Sense argues that without such information, the whole concept of informed consent is a fiction.

Making Sense of Health Statistics

John Grohol, Psy.D., is founder and publisher of PsychCentral, a pioneering community of e-patients. After he read my post the other day about evidence-based medicine, he sent me a paper worth reading: Helping Doctors and Patients Make Sense of Health Statistics.

This is relevant to the e-patient movement because as you and I become more responsible for our own healthcare, we need to be clearer about what we're reading. Plus, it appears we could be more vigilant about what our own professional policymakers are thinking.

The paper is 44 pages, but even the first few will open your eyes to how statistically illiterate most of us are - and that includes MDs.

Consider this question, which was given to 160 gynecologists:

Assume the following information about the women in a region:

• The probability that a woman has breast cancer is 1%
• If a woman has breast cancer, the probability that she tests positive is 90%
• If a woman does not have breast cancer, the probability that she nevertheless tests positive is 9% (false-positive rate)

A woman tests positive. She wants to know whether that means that she has breast cancer for sure, or what the chances are. What is the best answer?

1. The probability that she has breast cancer is about 81%.
2. Out of 10 women with a positive mammogram, about 9 have breast cancer.
3. Out of 10 women with a positive mammogram, about 1 has breast cancer.
4. The probability that she has breast cancer is about 1%.

21% of them got the right answer (#3, 1 chance in 10). 60% guessed way too high, the other 19% guessed #4. (That's 10 times too low).

The paper presents numerous other examples of statistical illiteracy (an example of "innumeracy"), misunderstandings of data that lead to serious unintended policy consequences. My personal favorite is the opening item about Rudy Giuliani's assertion that he's lucky to have gotten prostate cancer here instead of under the UK's "socialized" medical system. It's not because I don't like Giuliani - it's that his own misunderstanding of the data he was quoting led him to advocate something that had nothing to do with his actual odds. He himself would have been harmed if he'd been guided by his own best advice. And he's not alone in that.

The paper proposes uncomplicated ways to improve our comprehension. First among them is to stop talking in percentages and talk instead in raw numbers. Phrased that way, the same three facts that were given to the gynecologists is much clearer:

• Ten out of every 1,000 women have breast cancer
• Of these ten women with breast cancer, 9 test positive
• Of the 990 without breast cancer, 89 nevertheless test positive.

With this view, 87% got it right. (Of the 98 women who tested positive, only 9 actually have cancer: about 1 in 10.)

Another example echoed what The End of Medicine said about Lipitor. (Without Lipitor, 1.5% of the control group had a coronary event; with Lipitor, about 1% still had one.) A 1995 alert in the UK warned that certain oral contraceptives doubled the risk of blood clots in the lung or leg. Understandably, many women stopped taking the pill; within three years, 13,000 more abortions were performed, reversing five years of decline, and there was a matching increase in live births.

What was the risk that led to this? In raw numbers, one woman in 7,000 has such a blood clot anyway; with this pill, one more blood clot happened.

The irony in this case is that both abortion and childbirth carry more risk of clots than the pill itself. In other words, one benefit of the pill is that it avoids the risk of clots associated with the end of any pregnancy.

So although the number presented ("double the risk") was absolutely accurate, the real clinical impact wasn't nearly as absolute.

This is a taste of what's in the first few pages. It gets dry in places but even the first few pages are compelling and informative - and at no point does it require that you be a mathematician. The explanation of Giuliani's error is particularly good.

Thanks to the good Doctor John for the link.

Continued in part 2

Evidence-based medicine

On the fringes of medical knowledge, lives are at stake and medicine doesn't have the answers yet. What do you do?

As I've recently studied the nature of healthcare today, one thing I've learned about is evidence-based medicine. It's a discipline whose intent, at least in part, is to correct what you might call "medical superstition" - overprescribing certain treatments for no reason other than an individual doctor's preferences or superstitions.

Excellent researchers, now at Dartmouth, discovered widely varying practices, such as a fourfold difference in rate of certain surgeries (from tonsillectomies to hysterectomies) in some regions, even after correcting for differences in population. The discipline of evidence-based medicine is to prescribe treatments based on evidence that they make a difference, not based on local doctors' personal favorites.

What I'm also learning, though, is that the discipline has shortcomings. For one thing, not all evidence of effectiveness means something should be prescribed a lot. The End of Medicine cites Lipitor, the cholesterol drug (a "statin"). We spend $25 billion a year on statins. There's statistically significant evidence that it helps - a 35% reduction in coronary events. But that same evidence, if examined closely, shows that it only makes a difference for 0.5% of the population.

Specifically: 1.59% of the placebo group had a coronary event, but 1.03% of those who got a statin had one anyway. (n=19,243. Study=ASCOT-LLA.)

Looked at a different way: if you're over 60 with cholesterol over 240, you have a 51% chance of coronary disease sometime before you die. But 49% still don't. Which group are you in? Nobody knows: we're at the fringes of knowledge.

This reminds me of the situation with my cancer treatment, high dosage Interleukin-2 (HDIL-2). Depending on which study you read, it only works on 7%, 13%, or 20% of patients. At my hospital it works on 20%, and my team said that's largely because they've gotten better at predicting who it won't work on, so they don't even try. But still, only one in five responds.

End says we spend $25 billion a year on statins; this 2005 article says 12 million of us are on Lipitor, not to mention other statins. The 35% decrease is enough to make it justifiable to insurance companies and doctors. Think what else we could do with $25 billion a year.

Another limitation of evidence-based medicine is that if it's used as the gating criterion for using a treatment, it blocks many things that could be useful if you're in need now, and the firm evidence you need now has not yet been developed - or has been developed, and hasn' t been published yet. (See "the lethal lag time" in Chapter 5 of the e-patients white paper.)

Or it's been published and your doctor hasn't seen it yet. (Tens of thousands of peer-reviewed studies are published every year. Who can keep up?)

This comes up time after time in the book Anticancer, which I mentioned the other day. Sample quote from a woman at a breast cancer conference: "If we wait for you epidemiologists to decide what's what, we'll all be dead! We need to make our choices now."

This is not to say that evidence-based medicine is wrong. It's a valid method, but it needs to be understood for what it is, not swallowed blindly.

Books that make you dumb? I don't think so.

Time for another statistics lesson. I'm not the world's greatest statistics whiz (not like the super-geek on TV's "NUMB3RS" show), but that's part of my point: you don't have to be a super-geek to detect major mistakes in statistics that come your way.

This isn't a rant - it's just an interesting example of what to be careful about, with a little entertainment along the way.

I just got an email with something that, on the face of it, is fascinating:

Some one matched up the most popular books in Facebook college groups with average SAT scores at colleges to see what people commonly read at different intelligence levels. http://booksthatmakeyoudumb.virgil.gr/

Nice graphics, and a decent explanation about his method. On the face of it, pretty interesting.

But there's this thing about statistics: you've got to be careful about (at least) three things:

1. When you see a pattern, are you really seeing a pattern you can count on, or is it just a momentary coincidence? (If the first two people to walk into your office are men, does that mean only men will walk in today?)
   
   
2. Even when you do see a pretty reliable pattern, can you be reasonably sure it means what you think it means? (A relationship between the behaviors of two variables is called a correlation, but that doesn't mean you can say one caused the other. A famous example: for some years there was a correlation between wolverine population and the number of sunspots. Did either cause the other? Not likely, and besides, who could tell? The lesson: Similar behavior of two figures could just be a coincidence.)
   
   
3. Finally, you've got to be really careful about whom you actually measured. (If you interview people who are hanging out in skid row bars at 2 a.m., you may reach some interesting conclusions about the opinions of people in skid row bars at 2 a.m., but you can't say they're conclusions about people in general.)
   

Returning to the email: this guy saw patterns in which books were favorites at colleges with different average SAT scores. Addressing #1, he correctly didn't count colleges with very little data. But he blew it on #2, when he titled the page "books that make you dumb," revealing a pretty massive fixation on one aspect of the whole picture, and flying in the face of his assurance that "I know correlation doesn't equal causation."

And besides, on #3, he doesn't even mention the gross sampling error of making an assertion about the book, based on data from Facebook readers who read it AND who participate in listing their favorites. Example 1: the Jesuit scholars at Boston College are highly intellectual, and I imagine that if they ranked their favorite books, the Holy Bible would rank high; but I doubt the Jesuits are ranking books on Facebook, and the Bible ranks among the lowest on this guy's charts.

Example 2: if some book actually made many people so brilliant they ditched Facebook, those people would disappear from this ranking entirely, and all that would remain would be the people who completely didn't get it. And, that book would show up as "making people dumb."

Besides, there's the whole issue of whether SATs are any indication of smartness, not to mention which type of smartness (Gardner's Multiple Intelligences).

He woud have been better off titling it BooksThatLowAndHighSATSchoolFacebookMembersLove.

This isn't just an academic issue - these errors can lead us to drive off a cliff. When we think we see something, and we don't, then with the best of intentions we can make serious mistakes in our conclusions, our policy decisions and our life choices.

For prettier statistics, omit inconvenient people.

Occasionally I’ll use this bully pulpit for a rant. The two top rantables on my agenda right now are statistics and silos. This time it’s statistics.

I’m irked because I keep seeing a mistake that blows the kneecaps off any well-intentioned effort to improve policy by looking at statistics. People need to be aware of it, spot it, and cry “BS!” when it rears its head.

Earlier this week, in Paul Levy’s blog I got into a discussion in the comments section of a post. Frequent and knowledgeable contributor Barry Carol had wondered if high health care spending around here might be caused in part by a large supply of hospital beds and specialists locally. I said, in part:

I'm intrigued with Barry's observation. (I don’t have an opinion – I don’t know the data he cites; I’m just intrigued.) Is it accurate to say the *cause* is too many beds? Or is it that more are available, so it's possible to give someone the care they need? [I then recounted a story of my father’s care in his final decade, where the hospital staff only seemed to become competent when it was time to kick him out.]

If motorists were spending lots of money on fixing flats, would we say the problem is that we have so many tire repair shops? It's not a perfect analogy, but it's worth looking at. Some cultures think women are the cause of rape, because if there weren't all those women, there wouldn't be all those rapes.

I feel strongly that any statistics about costs and outcomes in a system should have an accountant's note specifying what proportion of the population goes without coverage in that system, so they don’t even have an outcome. Until we get honest about that, all we're doing is chasing a bubble under the blanket.

There’s the rub, the itchy spot. In cases like this, the goal of statistical analysis is to better understand things, particularly to know what a batch of data does or doesn’t represent so we can predict the best way to approach future situations.

And if we don't know what those statistics left out, we don't know what we'd be getting ourselves into by relying on them. We cannot rely on findings until we know what cases were and weren't included.

Increasingly, what might be getting omitted is you. Or someone you love.

As the boomers age, and their decades of productivity and home buying convert to decades of home selling and health costs (who, me?), this is gonna be a big skull-knocking issue. There will be claims about which system works better, with all kinds of statistics being flung around like monkey dung. (Sorry, but monkeys do fling dung when they’re fighting, and when policymakers start fighting, they fling statistics, claiming they're proving reality.)

For health policy, all kinds of claims can be made with good statistical support – but you damn well better ask who got left out, making the picture prettier, whether it was intentional or not.

Personal story: in Massachusetts insurers must price all group policies the same, without considering who’s in the group; New Hampshire has no such law. My wife and I started 2007 with insurance at her job in NH. Without warning, in June her (small) employer’s group rate went up 60%. Why? Because she had turned 60. Young people generally incur lower health costs, so in most states a company can choose to be competitive by selectively offering lower rates to more attractive groups. But when she turned 60, the entire company’s rates went up 60%.

I work in Mass., and it turns out we could get equivalent coverage from my employer (from the same insurer! See my next post) for 40% less.

Now here’s the killer: in NH the disenfranchised can find themselves in real trouble, as policies evolve and unattractive individuals are increasingly isolated. Next personal story: I know a healthy, athletic 20-something whose coverage was costing $2,300 per year (for one person) because she has a minor murmur that’s never caused a symptom, but she wasn’t in a big group. Now she works for a big company, so she’s swallowed up into a big group and gets group rates.

What is the justification for this???

I also know two young families who simply go without coverage because there’s no room for it in their budget. Statistically they are of course counted in the 46 million uninsured – but I say they should also be factored somehow into the total cost of health care, including what it WOULD cost to provide the care they don’t get but would if they could. (Which brings us back to Barry's point about how many hospital beds we have.)

Worse, while excluding those cases, you can bet that the insurance companies (all of them) talked about how good their rates are, and they mean it. (I would - I'm in marketing, and when I believe my company is doing a super job, you bet I say so.) But again, I say you can’t talk about costs and outcomes without specifying whom you’ve excluded.

Final first-hand story: some years ago, when self-employed in NH, I myself found that I couldn’t afford health insurance, because at the time things had evolved to where almost all the AIDS patients in the state were in the category “not a member of any group” – same as me. So any statistics about insurance prices in that state at that time would have been a fat load of crap – flingable crap.

Overlooking the inconvenient people isn’t limited to health care costs. Consider the following, from the US Dept of Labor’s Bureau of Labor Statistics (BLS):

• Unemployment statistics don’t include everyone who wants a job but can’t find one. Once your unemployment benefits run out, they simply stop counting you. You don’t even exist as a problem anymore, as far as the BLS is concerned. I cannot figure out a legitimate reason for this.
  
  
• There are no statistics for people who eventually gave up on their previous career and are now working for half their previous pay. People in that situation are, again, simply not counted as a concern.
  
  
• Nor are there statistics for the loss of benefits. Employers certainly pay less for no-benefit or feeble-benefit jobs, but if you or I change to a job with no benefits, it doesn’t even make a dent in the pretty statistics.
  
  
• Worst of all, the “jobs created” statistics are a cruel joke. When a full-time job with benefits is carved up into three part-time jobs with no benefits, the BLS counts it as job growth. (I called my Senator’s office and had them check it out; a senior BLS statistician got back to me and confirmed it.)
  
  

This is insane. It's as if King Solomon chopped up 1,000 babies and declared a population explosion.

What is wrong with these people?? In May of 2006 an erudite observer in the New York Times remarked with surprise about the 200,000 “new jobs” that had been created in April: “employment [is] doing well, yet core inflation has remained remarkably subdued." Remarkable indeed, until you know what they're calling “job creation."

As I say, until we get honest about this, all we’re doing is chasing a bubble around under the blanket. With the best intentions, we'll make misguided policy decisions. And believe you me, policy has impact at the personal level. The time will come when you (or a loved one) is the bubble everyone wants to chase away. Do whatever you can to stop this crap. Now. Wake up! And wake others up.