You’ve seen on many websites and “As Seen On TV” commercials: a claim that they have studies that prove the effectiveness of their product. Not so fast! BrightReviews examines the tactics marketers use and expose them in this article we call “Anatomy of a Clinical Study.”
#1 – Trust Us: It’s Clinically Proven!
Sometimes companies don’t even bother with actual proof; instead they’ll just say their product is “clinically proven” to work. Perhaps they’ll have some kind of seal, maybe with a “doctor recommended” or even have the rod-and-snakes symbol associated with medicine.
These images have been clinically proven to say “clinically proven.”
What you should know: Just because something says it’s proven, doesn’t mean it is. In other words, proof needs PROOF. It’s also important to note that the two-snake symbol as illustrated above is different from the rod of Aciephus (below right) which has only one snake and used by the American Medical Association.
Can you spot the difference?
#2 – They Give Incomplete Information
Some companies will take the extra step of adding some more information to their claims, like 78% of people noticed a big improvement or 85% said they felt better after using it. There may even be a fancy chart showing these results.
75% of people think this chart means something.
What you should know: There’s still not enough information about the study. For example, who conducted it? How many participants? (We’ll go into the criteria for what makes a good clinical study in a bit.) Finally, you may note the language of these types of studies say “participants reported” or “patients said” – this means the results were based on the patients’ own personal assessment, not on any changes in condition observed by the conductors of the study.
#3 – They Point to a Study That Doesn’t Back Their Claims
Another tactic is to cite a clinical study, but misrepresent its findings. For example, on the Brain Storm Elite website one of the studies they point to about the benefits of Panax Ginseng expressly states: “Currently, there is a lack of convincing evidence to show a cognitive enhancing effect of Panax ginseng in healthy participants ...” Another brain supplement site AlphaZXT cites a report that doesn’t even include information on the benefits of phosphatidylcholine; instead it seems to be an observational study on primate brain growth.
What you should know: By including a link to a clinical study, they hope it adds some legitimacy to their claims. But usually they are counting on you to either not click on the report, or not read it very carefully.
What You Can Do
We’ll admit we have no scientific or medical training and probably neither do you. And many clinical studies are filled with jargon that is difficult to understand. Still, there are some very basic things you can do to at least analyze the general claims of the study and see if it has any validity.
Where was the study done?
A good clinical trial will tell you where it took place. Was it in the United States in a reputable university or medical facility? Was it performed in some other country? Was it performed in-house by the company that also makes the product or paid for by the company but done by an independent lab? While it’s possible a study can be legitimate if done or paid for by the sponsoring company, it’s usually a red flag that it’s biased.
Generally speaking, the United States has the most stringent requirements for GCP (Good Clinical Practice) outlined in this document. Other countries have similar documents but require less scrutiny.
How many people were involved?
Again, we don’t claim to be scientists or doctors, but a good rule of thumb for a study is number of participants. No matter what the results, a study of a handful of people may show promise for a certain claim, but should be studied with a larger (read: thousands) of participants.
Is it a double-blind trial with a control group?
A double-blind trial is a standard that means both subject and experimenter don’t know who is actually in the test group and who is in the control group. This way, any biases can be eliminated and compared to the control, who are given a fake version or placebo. Results are then measured against the “placebo effect.”
Is It Peer-Reviewed?
After a clinical trial or experiment is conducted, it is essential that both the methodology and subsequent results are evaluated by an independent party. Like we have a “jury of peers” similarly doctors and scientists have their “peers” (persons with equal training) that look over studies to ensure greater reliability in the results. This fun graphic from Understanding Science explains it simply:
Image credit: Understanding Science
Peer reviewed papers are usually published in a scholarly journal that has “peer reviewed” somewhere on its masthead (look for it!) They also have a handy thing called an abstract, which summarizes key findings of the report.
Are Results Too Good To Be True?
Finally, you may want to ask yourself that clichéd question: “are the claims of this (pill/brace/cream) too good to be true?” And then if the answer is “yes”, remind yourself of the second part of that adage: “then the benefits probably aren’t real.”
As an article in The Guardian points out, in spite of advances in technology or health care, radical medical cures are still few and far between. After all, there is still no cure for the common cold!
We hope this brief overview of the different ways companies use (and abuse) the clinical study process to try to sell you stuff. Now you have a better idea of what to look for when evaluating these claims.