Welcome to another edition of ASK BILL!
Bill has been overwhelmed by the number of requests that he receives from people asking him a variety of questions related to science and scams that circulate on the Internet and elsewhere. To manage his time effectively, Bill has decided to do a weekly fun and educational "Ask Bill" segment.
Every Wednesday, Bill will choose one question from his e-mails and answer another science or hoax question. Get your questions into him ASAP.
For this week's ASK BILL, the question is:
How does a regular Joe know a good scientific study from a bad one?
-------------------------------------------------------------------------------------------------------------
This is a very good question. For laypersons (and even those knowledgeable in the area), trying to find out if a certain study is significant or not can be very confusing. It is not always a simple straight answer as there are many factors to consider. None of the following by themselves is necessarily a sure sign of a bad study, but they can definitely be used for the final consideration, especially if the study has multiple infractions.
The first thing I look at is where the study was published. This is usually a good first indication. If a study is published in a respected scientific journal, its odds of being a good one increases. Of course some bad studies still do get published (some are deliberate hoaxes testing the reliability of said journals and their process), but in time they are often retracted. Searching on Google, the top journals can give you decent resources such as:
http://www.scimagojr.com/journalrank.php and
https://en.wikipedia.org/wiki/List_of_scientific_journals.
So how does a study get into a respected journal? There is usually a process of evaluation (part of the scientific method) and review. They are often checking the methodology of the study. It's also good to know what type of study it is.
Is it just a review study looking at certain literature? I find these types are only ideal for proposing further studies and not necessarily for drawing any solid conclusion. Now this shouldn't be confused with a meta-analysis, which looks more in-depth at a larger selection of studies to try to determine a reasonable consensus and include more complex data algorithms and thus hold more weight.
Is the study relying on people's reporting/memory recall (which is often fallible, inaccurate, or prone to bias?) Studies that rely on people reporting from memory how they ate 5 years ago (or even 5 months ago) is problematic.
Was it done with experiments on actual people? Many studies are done just in petri dishes or maybe in just certain animals. While these give some good starting points for pursing further studies, they can't always be relied on to discern the reactions within the human body, which is quite different then a petri dish and rats, for example.
How many people were used? A small number of people used in a study can contain a lot of “noise” in regards to more closely representing what can happen in the general public. A study done on 10,000 people is definitely more robust than one done on 10. For example, you could get 3 out of 10 with a reaction and thus conclude a 30% effect. But in the 10,000 study one could find only 100, which then is only 1%.
Was the study blinded and have controls? Although not always possible, being randomized, blinded and having controls to compare to can increase the robustness of the research. Having it where participants do not know if they are getting a placebo or not is ideal, and even more ideal if the researchers are also blind to that fact as well (usually relying on an impartial 3rd party to keep track of that info for later review).
Did the study list conflicts of interest? A good study will list any conflicts of interest that could bias their results. It's about open honesty. One that hides any conflict raises serious red flags. To find a conflict of interest, one may have to do a bit of searching to discover the conflict of interest.
Has it been peer reviewed? With science part of the process is having your research critiqued. This can be others checking your math or even better seeing if when they repeat your experiment, can they replicate the results. This type of review of a study done by competent peers is the reason some studies have been retracted from journals. They found serious flaws in the data, conclusions and reproducibility of the studies. This is why good science can sometimes take a while, but it eventually corrects itself.
I find that media reports on science studies can sometimes misrepresent or exaggerate what is said in an actual study. Headlines are usually done to grab your attention by asserting a certain point like “chocolate can help with_______ study says.” What is often the case is a certain ingredient was found to have an interesting result (quite often in that petri dish we talked about earlier) and that more study is needed. So don't take headlines at face value. It's always a good idea to look at the actual study to see its conclusions (if any), or you could just ASK BILL :) (or any other friend who is proficient in finding out that info).
There are many more factors to consider as well, but just knowing some of the above, one can start to get an understanding of what makes a good study and will hopefully be able to disregard some “bad science.” I hope that answers your question.
-------------------------------------------------------------------------------------------------------------
*For over 30 years, Bill has been a professional magician and has traveled all across Canada, performing for all ages. Along with his passion for entertaining, Bill is an educator and life-long learner. He continues to study biology, psychology, neuroscience and chemistry. Bill has also written many articles on science and scams for various blogs, newspapers and other publications.
Visit his business site, Nuvo Entertainment: www.nuvoentertainment.com
