Duped by dark chocolate: Anatomy of a weight loss hoax

Did you hear the news that dark chocolate is a "weight loss accelerant"? It's scientific research, so it must be true. (It's not.) It was all a grand hoax and the "researcher" fooled millions. Online scams are ubiquitous, but what made this one different was that, allegedly, there really was an actual diet study.

But if there was, in fact, an actual diet study involving dark chocolate, how was it a hoax?

I'll explain how John Bohannon, a science journalist with a Ph.D. in microbiology (of bacteria), deliberately used shoddy "science" to scam the media and betray the public trust and why this should matter to you.

According to Bohannon, he conducted a three-week diet study and then randomly assigned 16 people (although the research and press release didn't disclose this number) to one of three diet groups:

Why the "study" was a scam:

1.  Far too few people in the experiment ("sample size"). For a study testing three different diets, 16 participants is the equivalent of about five people per diet. This type of experiment needs way, WAY more people than that (and needs to be tracked for longer than 21 days). Would you recommend that the world's population of 7 billion people start eating dark chocolate based on the weight loss of five random people over a three-week period?

In terms of "good research practices," Bohannon took matters from bad to worse by comparing the three diets in a myriad of different ways.  He reported finding a "significant" difference among the groups: the chocolate-eating low-carb group lost five pounds 10 percent faster than the low-carb-only group.

2.  Lies, damned lies, and statistics. When researchers compare groups, they have to decide whether any differences are due to chance (dumb luck) or real, meaningful differences between the groups (like amount of weight lost for each diet). Bohannon and colleagues "concluded" that the differences among the diets were not due to chance alone; i.e., the chocolate, not chance, explained the differences.

But he rigged the game. Bohannon concluded that the chocolate dieters lost weight faster than the other two diets by using what's called "p-hacking." I'll spare you the math, but it's the equivalent of rolling two dice, betting you're going to roll a three (a 2 and a 1), and then reporting that you did, in fact, roll a three (low probability). BUT, in reality, by doing so many different comparisons in his study, Bohannon had basically just kept rolling the dice until he finally rolled a three. By my calculations, based upon Bohannon's account of the scam, he did the equivalent of rolling the two dice not once, but 16 times (in which case he could expect to find a "significant difference," most of the time, just by chance alone).

3.  Pay to publish. Scientific journals are scholarly "magazines" that publish research articles. Good thing such obvious bunk could never get published in a scientific journal, right?

Wrong. He submitted his article to a very scientific-sounding online "journal" that charges scientists to publish their research findings (a highly unusual for reputable scientific journals). Bohannon reportedly paid the hefty fee, and his article was then published with nary a comma change.

4.  Get the spin out. Bohannon asked someone in on the scam to issue a carefully written press release and the story took off. The media covered the story in 18 different countries, and millions of people eager for a chocolate weight loss miracle were duped.

Why you should care:

Aside from the sticky ethical questions raised by this "study," Bohannon's sting raises an important issue for the general public: How can you trust your science and nutrition news and information?

The field of journalism is in flux, and the public has become increasingly accustomed to getting their news for free. With ever-declining paid subscriptions, most media outlets have become reliant on advertisers for a revenue stream.

Advertisers pay for "clicks" or page views. This can mean less available resources with which to adequately compensate science writers, who are skilled enough to critically evaluate research. And, if a headline proclaiming the discovery of dark chocolate as a "weight loss accelerant" is going to generate Web traffic, there's tremendous incentive to quickly and easily recycle a press release into a news story, and minimal incentive not to do so.

Tips for getting your facts without spin.

Stacey C. Cahn, Ph.D. is associate professor of clinical psychology at the Philadelphia College of Osteopathic Medicine. She specializes in eating disorders, obesity, body image and cognitive-behavioral therapy. She wishes to acknowledge Brian Bober's helpful contribution to this article.

----------

Read more Goal Getter for healthy eating, weight loss and more.