Thursday, January 22, 2009

Science Blog: Clinical trials: Unfavorable results often go unpublished

I have a continuing interest in patients understanding the statistics that are used to recommend treatments to doctors - and thus to us. So I was intrigued by this post Wednesday at Science Blog:

Clinical trials: Unfavorable results often go unpublished

Trials showing a positive treatment effect, or those with important or striking findings, were much more likely to be published in scientific journals than those with negative findings, a new review from The Cochrane Library has found.

"This publication bias has important implications for healthcare. Unless both positive and negative findings from clinical trials are made available, it is impossible to make a fair assessment of a drug's safety and efficacy," says lead researcher, Sally Hopewell of the UK Cochrane Centre in Oxford, UK.

The international team of researchers carried out a systematic review of all the existing research in this area. In addition to showing that negative results were published less often, they found that if these results were eventually published, they would take between one and four more years to appear in journals than studies showing positive results. ...
As often happens, there's more to learn from the comments than in the original post:
"No one will publish a paper about an experiment that gave negative results. The problem is that negative results could as important as positive ones (so maybe other researcher won't try the same thing again, for example). I hope the publication industry will soon disappear, and that the strengths and paradigm of the Internet will finally be used also for scientific articles."

"The worst part of this very understandable human trait to publish only successes is this: How can we learn from failures if we never hear about them?"

"I think you could argue that publication bias negatively affects the entirety of scientific research, not just clinical trials. As someone who works in the business, I wanted to mention the PHARMA Code, the code of ethics for the industry. Regarding publication it's pretty clear: you must ATTEMPT to publish findings, significant or not. Now, whether a journal editor wants to publish non-significant results is another story entirely."

"I think it is unfair to blame the journal editors. ... The researchers (in any grant-dependent field) may be required by a grantor or a code of ethics to attempt publication, but my guess is that they don't work very hard to produce a high quality manuscript when all they have to discuss are unsuccessful trials or nul results."
My takeaway: it's even sketchier than I thought to presume "If it was important information, we'd have read about it." The process of preparing and publishing articles is fraught with potholes and pitfalls.

I'm not saying we should ditch journals. We should, though, be conscious of what they are and aren't. Certainly not an inherently authoritative source of information – despite the best efforts of their editors.

p.s. Where did I learn of that post? In an online patient-to-patient community – patients empowering and informing each other. Gotta love the Internet and e-patients!

15 comments:

  1. Dave,
    I admit the folks at Cochrane have decided to dredge this up again, and I don't blame them. BUT, it's been well known and intransigent for years. This is bit of a strawman coming from Cochrane. The standard approach to meta analysis and guideline establishment includes finding all the unpublished and negative data out there (and that's what Cochranes do for a living!) I admit, hearing about negative studies as part of meta analysis is not optimal, but at least it's something. It's also disingenous to say this is ALL due to a culture of valuing only positive studies, or "successfull" studies. Many important negative studies have been published. There are two problems, 1) It's a lot harder to present negative data convincingly from a purely statistical perspective (it's easier to show difference than "sameness") 2) we are relying on our expert community to decide what is important.

    ReplyDelete
  2. Hi Dan -

    Why "I admit"?

    I thought about the subject for some time before I decided what to say. I decided I can't imagine asking for full-blown articles about tests that failed, unless the results were really extraordinary. So I chose not to "rage against the machine" or imply there's something crooked.

    But from my perspective as a self-educating e-patient, I've been exploring what constitutes a good reliable source of information. What's increasingly clear to me is that we shouldn't assume journals include all the relevant information we might want.

    Commenter Fred Bortz said he thinks there are registries of clinical trials so failures can be found. I don't know if that's true, but it seems like it would be a great idea - there could be a (very large) Web list of failed trials. As someone else said, this would at least let other researchers know something's been tried, perhaps eliminating some waste.

    ReplyDelete
  3. Many of the folks in the Cushing's community desperately want a cure for the disease other than major surgery. Right now there is none. There are a couple of stop-gap measures that work for some (ketoconazole, etc.)but they are temporary due to their effects on the liver and other parts of the body.

    There have been and are on-going multiple trials with several drugs which we are told are "promising", but yet we later find out they are no longer testing them or they aren't viable. We don't always know why.

    I think what Dave says is very pertinent. And perhaps the information really is out there, but not as easily found as the promising trials/information/research. I know the following do record any trials. I'm just not sure if ALL trials/research are reported somewhere. Is it voluntary? Or do all have to be reported.

    Those funded by the NIH and other government monies certainly are recorded. Is all data accessible? I'm not sure.

    The International Federation of Pharmaceutical Manufacturers and Associations put out a statement in November asserting their commitment to transparency.

    NIH records funded trials here:

    Also pertinent are these (I'm doing down and dirty links...sorry)

    http://www.centerwatch.com/clinical-trials/results/db/

    http://www.who.int/ictrp/en/

    http://clinicaltrials.gov/

    The big question: IF all data is out there, but only "positive" data get to the journals, then how does one extrapolate the other? And even more key, is how does one do that relatively easily?

    ReplyDelete
  4. All,

    It's great to hear from Robin, blogger of the well respected Survive The Journey blog. Her blog was one of the finalists in the way-cool MedGadget Best Medical Blog awards this year, and for good reason. She knows her stuff and writes well. Check out some of her posts.

    And if you don't know what Cushing's Syndrome is, and the consequences of people not being aware of it, get a tissue and read her post Martha's Story.

    ReplyDelete
  5. Published on www.brainblogger.com:

    A Failed Attempt to Improve Misperceived Greatness: The ENHANCE Trial

    While it seems that pharmaceutical company sponsors of clinical trials usually end up with results that clearly favor their meds studied in their trial, there are rare exceptions, and Merck and Schering proved that with their disappointing ENHANCE Trial, which many have heard about through the media not long ago. The drugs studied were Vytorin, which was compared with Zocor
    Vytorin is a combination med for high cholesterol and contains Merck’s Zocor, which is now generic, and Schering’s Zetia, which works differently than Zocor, which is one of many statin drugs. Both Vytorin and Zetia are co-promoted by Merck and Schering. So, several years ago, an outcomes study was initiated to prove superiority of Vytorin over Zocor. The trial was named the ENHANCE trial, and possibly this trial was initiated because Zocor is generic now, and not a priority from a profit paradigm of its creator.
    After several years passed, a disappointment arrived for the sponsors of this trial, which was first brought to the attention of Schering in March of 2007, yet the results existed since the spring of 2006, I believe upon information and belief.
    The disappointment is that Vytorin lacked anticipated benefit or superiority over Zocor. Since about 1 million scripts were written for both Vytorin and Zetia every week in 2007, combined with what I believe was about 5 billion in revenue for these two drugs that year, this was a problem for the drug makers, meaning a fear of shareholder reaction. Perhaps for Schering in particular, it was more of a calamity, since over half of their profits and earnings were from these two drugs with Schering, I understand.
    Being the responsible corporations both companies are, of course, alterations occurred after such events were discovered that fractured numerous rules and regulations with clinical trials, possibly in illegal and unethical tactics.
    The trial sponsors delayed the release of the trial results for secrecy reasons, it has been speculated. Results from the trial existed, yet were not disclosed at the time of their discovery. After several months of possessing these trial results that were only known to the manufacturers, they created or implemented some atrocious tactics to improve the trial’s unimpressive results following the original results of this ENHANCE study. At the end of 2007, the companies changed the primary endpoint of the trial, which is what the results were measured upon during the entire course of the trial. Sort of like sorting cards to make a good hand not dealt to you. Anyway, since their deliberate concealment of these trial results was clearly wrong, to respond to those who asked where the results were actually as they had been anticipated for quite some time, and while such trial manipulation was occurring and results were being kept secret, Schering stated that continued data analysis from the trial results was the etiology for the delay.
    With clinical trials, case report forms are used to record data from the trials, and are created in a manner where further analysis is not normally necessary, as such forms are quite clear and often not subject to interpretation as implied by the trial sponsors, one could conclude. So at the end of 2007, both Merck and Schering got the attention of relevant government officials who contacted both companies regarding this ENHANCE trial due to such suspicions on the facts known and presented, and an investigation began into the activities of both companies regarding this trial at that point.
    This became a catalyst for the ENHANCE trial results to be finally released at the beginning of 2008, which caught the attention of major media organizations, as expected. In the spring of 2008, a very large cardiology meeting was held, where the audience was told, I understand, to stick with statins due to this trial’s lack of outcomes for Vytorin, when the ENHANCE trial was discussed at this meeting. Furthermore, it has been said that a cardiologist at this meeting also suggested that a moratorium should occur with the utilization of Vytorin by prescribers, since statins are much less expensive, and are highly regarded, as they have been available for a couple of decades, starting with Mevacor in the 1980s. Of course and as expected, Merck and Schering were not pleased, nor were they surprised at the review of Vytorin at this particular meeting. The following month after this cardiology meeting, Schering’s earnings dropped by 48 percent, as I recall. Also during much of this year, Schering in particular blamed the media for amplifying the situation regarding the ENHANCE trial.
    Now, these cholesterol drugs promoted by Merck and Schering, Zetia and Vytorin, were aggressively marketed in a number of ways, including investing I believe about 200million dollars in 2007 for DTC ads for these products. To add to this, and soon after both meds were launched, reps from both companies made inferences to doctors about outcomes regarding plaque accumulation and how Vytorin was superior in that area, which, of course, this ENHANCE trial proved it is in fact not the case whatsoever. It did not matter, apparently, to both Merck and Schering that such claims were is entirely void of proof, which is not unique to any pharma rep, in my opinion. No remorse or regret from the makers of these drug makers, either, which did not shock many. Yet what is known now is that these companies, as stated by other researchers, performed junk science with their deliberate manipulation of this ENHANCE trial using such tactics. Also, last year, Zetia and Vytorin had about 20 percent of the cholesterol lowering market. It does not seem that there will be an increase of this percentage because of this scandal. Possibly if they presented the truth, the future of these meds might be better than what is anticipated presently.
    Worst of all regarding this ENHANCE trial scandal is the harm caused to both doctors and patients. The ENHANCE trial concerned and confused both of these participants in the health care system. Furthermore, it’s likely they were devastated by being so clearly misled by the marketing of both Merck and Schering regarding the false benefits of Vytorin they were led to believe by the companies that promoted them- the health care providers in particular.
    This whole situation is another example of the progressively frequent discovery of corruption of the scientific method by placing profits over the well-being of patients, which harms the well being of patients. In addition, most were shocked by Merck behaving in such a way in particular because of what use to be their excellent reputation as an ethical pharmaceutical company. And this alone shows the progression and infiltration of such damaging ethical atrophy that desperately needs to be stopped and corrected for the sake of others- for everyone.
    Don’t just say something. Have something to say- to the right people, with conviction and with others who share your views.
    “Try not to become a man of success, but rather try to become a man of value.” --- Albert Einstein
    Dan Abshear

    Author’s note: What you have read is based upon information and belief. Thank you

    ReplyDelete
  6. Dan,

    We met at the forum the other day. Good post here. If you are familiar with Edward Tufte he is a world renowned speaker on statistics and information. He says something to the effect, "The most exciting scientific research is rarely ever hear from a year later. This is due to cherry picking."

    At the forum we encounter this form time to time. When you tell a speaker their scientific data will be posted online after their talk, most are very happy about it. But some suddenly get very nervous and refuse. Now, some have a legitimate claim about unpublished data, but this is very rare. Most use the issue of unpublished data as a way to cover their bad cherry picking of data and the false hype they have created.

    Good science develops over time and rarely ever starts off with a bang.

    ReplyDelete
  7. Great to see you here, Mike! (Folks, this is Mike Young of the Center for Integration of Medicine and Innovative Technology. Do I love what they're up to, or what?? I met him at a session of theirs that I attended in Boston last week.)

    Of course I love Tufte. But I only know him for his work on data graphics - didn't know he's commented on the data itself, particularly sleight-of-hand. Are you saying people are bold enough to publish or present data that's essentially a sham? Cherry-picked to show the best cases, not the overall picture? THAT would be a whole different kettle of fish.

    ReplyDelete
  8. Not exactly that people are presenting sham data in their research, rather presenting just the beautiful pieces. "Beautiful Evidence."

    This typically happens early on in the pilot research stages and is humanly innocent. It is done out of the desire for primary funding so that their science can have good and meaningful impact on patient care. Many clinicians have a feeling that their science is good and that it will truly work out in the end, even if the pilot data is mixed. The belief is that if they can just get the money, they can get the science working later on. They have faith in their science working out, rather than looking at the pilot facts.

    This leads to selecting which data they show upfront and this creates "excited science." People all buzzing with hope and energy. Stem Cells had this early on and today it is a much more realistic science of authentic progress.

    Mike

    ReplyDelete
  9. Good point – my wording suggested sham data when I really meant sham conclusions. I should be more careful.

    I've got nothing against saying "promising early results" but IMO it MUST be couched in "too early to draw conclusions."

    I understand "humanly innocent" but IMO this difference is what makes us scientists, relatively speaking: the ability to know when we do and don't have something that's reasonably reliable.

    ReplyDelete
  10. Hi Dave! OH MY GOSH! I read your call for patient-speakers and my heart started pounding. I thought OH MY GOSH! THAT IS WHAT I SHOULD BE DOING! I have lived 24 years in Canada and 24 years in USA needed constant medical care for multiple auto-immune problems in both countries, and I could write a book about the comparisons and needs. The problem is, I am not well enough to do it...yet! And in Canada, with the health care at #35 on the World Health Organization quality of care list, it looks as if Canada's treatments will be VERY SLOWWWW. I think you have just lit a fire in me, and I am going to work to improve my health enough to be an advocate for HUMANE health care. The USA and Canada think they are the best of the world in everything, but we are #33 and #35 or so on the quality of country's health care list! see http://www.photius.com/rankings/healthranks.html
    WE NEED TO BE THE GOLD STANDARD here in N. America, for both public health care and private health care! Best wishes to you, KEEP IT UP!! I know I will!
    Sheila

    ReplyDelete
  11. Hey, you Bluebirdy person, where ya been??

    Everyone, this is a woman with a tremendous will to overcome adversity, to persevere. One of the things I love about e-patientness is that the Internet lets somebody like her, with limited physical capacity, have a great big freakin' empowered microphone.

    That's empowering, equipped, and enabled - and that's e.

    ReplyDelete
  12. Aww you remember me! Cool! I literally thank God that I can live in the age of the Internet, and do research and counsel friends from my bed. It is the great equalizer. I think people would be shocked to know the adventurous things I do world-wide from my bed. People who are too sick to leave home can reach very influential people from home, through the internet. I have made some changes locally in our abusive hospital, so maybe its time for me to move on to bigger things and help more people through changing the medical system. Someone just dedicated a whole post to me. Wow was I shocked and humbled! Dave if you want to see it its at
    http://melavilaalarilla.blogspot.com/2009/02/promise.html. He has a number of blogs, one of them reviews other blogs, and he reviewed my whole blog site about 6 months ago. I need to dig through his blogs to find the URL of that article though. I want to thank you Dave for making people aware and keeping this subject in the public eye and having a forum for use to feel empowered.
    Sheila/Bluebirdy

    ReplyDelete
  13. Thank you for your posting.
    Part of clinical research trials is getting all information out as soon as the results are ready. As to why a research facility would not follow suit on this I am not sure, so it is difficult to comment on that aspect.
    I did like the comment regarding the importance of filing even negative findings...it might keep other research facilities copying the same test.

    ReplyDelete

Your comments will be posted after they are reviewed by the moderator.