Savvy Supplement Shopper is a blog by CSPI senior staff scientist David Schardt. This blog aims to parse the good from the bad from the truly awful in the supplement aisle.
If you watch TV, you’ve probably seen an ad for the dietary supplement Prevagen. That’s the jellyfish protein that supposedly improves your memory. Consumers have spent more than $165 million on it over the past 10 years.
What did they get for all that money? Maybe not very much.
Quincy Bioscience, the Wisconsin company that manufactures it, claims that Prevagen “has been clinically proven to improve memory.”
But that’s not what the company’s own study showed.
According to the Federal Trade Commission (FTC), which charged Quincy Bioscience with false and deceptive advertising last January, the company study found that Prevagen was no more effective than a placebo at improving any of the nine cognitive skills, including memory, that the company measured.
So how can the company claim otherwise?
Because Quincy Bioscience went back and sliced and diced the results, performing more than 30 additional analyses of the data, and came up with three results that were statistically significant (and more than 27 results that weren’t).
It’s these three positive results that Quincy Bioscience touts as evidence that Prevagen is a “breakthrough” for helping with “mild memory problems associated with aging.”
What Quincy Bioscience did is called a “post hoc” analysis
This is an after-the-fact, unplanned exploration of the data to see if anything else of interest happened in the trial. Some might call it a fishing expedition. Scientists do this all the time, but with a big caveat: post hoc results are considered tentative, not conclusive. Before they’re accepted as valid outcomes, they need to be confirmed by additional studies.
That’s because random events happen all the time in scientific studies. Some of them may seem statistically significant, but they’re flukes and not the result of cause-and-effect. And the more post hoc analyses you do (like the more than 30 Quincy Bioscience did), the more likely you’ll encounter these chance results.
Scientists guard against accepting them as real by setting a high bar for statistical significance and by not accepting post hoc findings until they’ve been tested again.
But if Quincy Bioscience tested Prevagen again, it hasn’t said so.
So, Prevagen has been promoted all these years based on tentative results that have never been confirmed
The FTC finally challenged Quincy Bioscience’s claims in court this year, but this September a federal judge in New York dismissed the case in favor of the company. He ruled that the government had not shown that the company’s claims were wrong, just that there was an increased risk that they might be wrong. Of course, Quincy Bioscience hadn’t shown that its post hoc results were real, either.
It’s a terrible decision that invites supplement companies to massage and squeeze unsuccessful studies until they can extract something statistically significant out of them to use to sell their supplements.
It further tilts the playing field in favor of supplement companies and makes it even harder for consumers to figure out which supplements are truly worth buying and which are not.
Fortunately, FTC announced this month that it would appeal the judge’s decision. Let’s hope that good science prevails this time.
Meanwhile, Quincy Bioscience, how about taking some of the more than $165 million consumers have spent on your supplement and do another study to see if Prevagen really does anything for memory?
Here’s a link to the Federal Trade Commission’s announcement charging Quincy Bioscience with false and deceptive advertising:
https://www.ftc.gov/news-events/press-releases/2017/01/ftc-new-york-state-charge-marketers-prevagen-making-deceptive
Here’s a link to the government’s formal complaint against Prevagen:
https://www.ftc.gov/system/files/documents/cases/quincy_bioscience_complaint-filed_version.pdf
Here’s a link to the judge’s decision to dismiss the case:
http://adlawaccess.lexblogplatformthree.com/wp-content/uploads/sites/137/2017/09/Pacer45-Main-Document915579ba-c738-4944-8d51-9f37d2e8ef75.pdf