As we become more research literate, we recognize and even come to expect citations in articles of our professional magazines and throughout textbooks and teaching materials. We may go so far as to scan the references at the end to make sure the sourced information is current by checking the publication date.
Rarely do we explore the references further, trusting that the author practiced due diligence in selecting sources that represent the best available evidence. But what if the author found a convenient reference that supported the theory he or she is presenting, but it came from a third-hand source and the overarching evidence is to the contrary?
It is time to take an active approach and scrutinize the references our authors and educators are using. I postulate that if all professional articles and educational materials cite quality references, the caliber of massage education and our body of knowledge will improve. Outdated theories may finally fall by the wayside. In turn, we will get more evidence-based information, making it less challenging for us to offer evidence-informed massage and bodywork sessions to our clients. In the process of critically evaluating the citations, we hone our research literacy skills.
In a previous column, we learned that all research is not created equally (“Levels of Evidence,” March/April 2010, page 116). We can apply the same level of scrutiny to citations. Go beyond checking publication dates and investigate whether the information cited is someone’s interpretation of research results or a direct reporting of the research. If the source is an indirect quote, find the original study and check the accuracy of the quote and level of evidence of the original data. See if the data reflects the prevailing theories, has been cited in other research, and has been replicated in future studies. It is time to take a closer look at the citations of our professional articles and textbooks and uncover the story behind the authors’ statements.
What Makes a Good Reference?
There are three data points to investigate when scrutinizing citations that help determine if the quoted information comes from a reputable source:
• Is the reference a primary, secondary, or tertiary source of information?
• What level of evidence is the information based on: a randomized control trial, a case report, or something in between?
• Was there due diligence in investigating whether the data is representative of the best available evidence?
Primary, Secondary, and Tertiary Sources
Sources of information are considered primary, secondary, or tertiary depending on originality and proximity. Is the work original, or is the author commenting on the work of others? How close is the author to the source—is this a firsthand account or somewhat removed from the research? The distinctions are not rigid; it is not always easy to distinguish between a primary, secondary, or tertiary source of information.1, 2, 3
A primary reference includes original research. In scientific literature, a primary source of information is the original publication of a scientist’s new data, results, and theories. An author citing this level of information is required to interact with the research and extract information directly.4, 5
A secondary source represents an accounting of the primary data. The original information has been interpreted, analyzed, synthesized, or evaluated by a second party. In a research review, for example, the author restates the original data with a discerning eye, synthesizing a collection of data from a variety of like studies.6, 7
A meta-analysis is a bit different than a research review. Even though it is a culmination and summary of many like-research studies, it also presents original theories and combines data in such a way that new data is presented. Therefore, it can be considered a primary source of information rather than a secondary source.
A tertiary source is based on secondary information, providing an overview of topics. Typical tertiary sources include indexes, encyclopedias, databases, etc. Textbooks can be considered tertiary information, unless they rely exclusively on primary information.8, 9 A tertiary source is a retelling of information summarized by someone other than the original author—in other words, it is third-hand information.
A common pitfall in citing sources is that an author will cite a textbook—at best a secondary source of information—rather than looking up the reference and drawing his or her own conclusions. When you cite a primary source, you demonstrate that you engaged with the original data and considered its hypothesis, methods, and results in context, therefore making the quote more substantial. When you cite a textbook, you are assuming that the author is accurately representing the original research and not quoting it out of context to support a specific theory.
Levels of Evidence
The level of evidence, or the grade we place on the research design, depends on several factors such as generalizability or power, bias, and
the presence of comparison or
control groups.
Studies that provide information applicable to a large group of people are considered to have greater power than small studies that cannot be generalized beyond the subject pool. A study with 500 subjects, for example, is more likely to provide data that is transferable to people over and above those studied verses a study with one subject.10
Controlling for bias is about limiting the possibility that research results could be caused by chance or by something other than the intervention. For example, a well-designed research study does not permit the participants to select the protocol they will receive. If given a choice between massage and acupuncture, for example, and people already believe massage will work for them, choosing massage as their intervention can influence the treatment’s success. They believe they picked the best treatment and may skew reporting as a result. Research has shown that selection bias can boost the power of the treatment.11
A control group or placebo is used to measure the body’s natural ability to heal. It is always possible that the subject would have gotten better anyway over the course of time, and this must be accounted for in a study. This is not a fail-safe measure. Placebos have been known to advance healing. In fact, often the evidence demonstrates no statistical difference between the protocol and the placebo.12, 13
When gathering primary references, select research that represents a higher level of evidence than other studies on similar topics with comparative results. Another option is to provide multiple sources for one statement. Citing more than one source for a single piece of information will allow you to cite a wider range of evidence and include lower level studies without diminishing the overall quality of references.
Due Diligence
It’s important that authors cite the best available evidence. Authors must investigate both the data that supports their theories and the data that contradicts their theories. Then, the author can determine if the information being presented is representative of the current prevailing findings. Consider whether the data has been reproduced in quality studies or is an obscure idea or a theory that has been refuted in more contemporary studies. Other alternatives are possible: that the author’s theory has yet to be properly challenged by academic debate or that the topic is controversial. If the latter is so, full disclosure is recommended.
Full investigation requires some legwork on the part of the author or educator. Because our profession did not develop in an academic environment, we are not accustomed to questioning authority, as it were. We trusted the expertise of our teachers and mentors and our education was steeped in traditions and customs demonstrated to be effective over time. But we have new information about the underlying mechanisms of basic tenants taught 10 years ago, rendering some of our theories and practices invalid. Through scientific studies, we know massage in general, and lymph drainage in particular, does not spread cancer14, 15 and massage does not enhance the removal of lactic acid from muscles.16, 17 More theories need to be challenged by research, and safer, more effective approaches to treatment identified. By challenging our educators to research the research, our knowledge base and skill level can expand.
Challenging Citations, Improving Our Knowledge
Let’s put our new skills to the test. Pull out the March/April issue of Massage & Bodywork magazine. Turn to the article titled, “Working With Whiplash, Part 1” (page 109).
The author, Til Luchau, did a fine job of referencing his text. He wrote a thoughtful and thorough article on the signs, symptoms, and treatment of whiplash. Because his approach is sound and his delivery is well-documented, I felt confident asking Mr. Luchau if I could use the one less-than-stellar source he cited as an example for this article. It is a perfect opportunity for us to practice researching the research. Our goal is to gain better knowledge about the topic—whiplash—and learn to critique our sources of information, not to criticize Mr. Luchau. He graciously agreed to let us use this citation as a learning experience (see Til Luchau’s response on page 121).
Citation 6 supports a statement made early in the article on page 109, second column, first paragraph:
“…whiplash patients can be involved in legal or insurance difficulties, which may complicate or even hinder recovery.” The citation includes the direct quote from the source and is credited to:
Brian Grottkau, MD, writing in the New England Journal of Medicine (348, no. 14 (April 3, 2003): 1413–14) about Andrew Malleson’s Whiplash and Other Useful Illnesses.
As an educator on whiplash and massage since 1988, I am familiar with much of the literature on whiplash. I recognized the theory that litigation hinders recovery and have found it controversial at best. Many studies have successfully demonstrated the opposing theory—that litigation and recovery have no bearing on each other. I also know that many studies that support the former theory cite something known as the “Lithuanian study,” a project with suspect methods that have been criticized for more than a decade, although I did not know if that study had any influence on the statement made by Mr. Luchau.
I decided to investigate this citation. Using the previously proposed structure for evaluating citations, this is what I found.
Primary, Secondary, and Tertiary Sources
The first step in researching the research is to plug the citation into a research database like PubMed and see if it is available. Using the citation above, I went directly to the New England Journal of Medicine and discovered that the reference is a book review. The good news is that it is a book review in a peer-reviewed journal, which gives it good marks. The bad news is that a book review of a textbook is far removed from the original source and is therefore considered a tertiary source of information.
The guideline for determining if the textbook is secondary or tertiary requires us to look at the references provided in the back of the textbook. I found Andrew Malleson’s book, Whiplash and Other Useful Illnesses, almost page for page, on Google books.18 In searching the 62 pages of references, many are primary sources, but some are newspaper articles, magazine articles, and nonfiction.19, 20, 21 The textbook is not an original account of information, and it does not exclusively cite primary sources of information. Therefore, the textbook itself is a tertiary source. More importantly, we find it would have been relatively easy to go to a more direct source and quote the book and not a book review.
Do we stop there? Malleson, the author of the textbook, took the information about litigation hindering recovery from an original research study. The textbook is 544 pages long, potentially making it difficult to find the original research study where the quote was sourced. Luckily, Google books provides a search engine, allowing for a search of the text. I typed in the keywords “litigation hinders recovery” and got exactly what I was looking for. Three short pages were highlighted, and after a quick read I had all the information I needed. I was just steps away from quoting the original source: the infamous Lithuanian study, as suspected. From here, I went to PubMed, typed in “Schrader H and whiplash” and read the abstract of the original research, including his most recent study.22
Levels of Evidence
Once we get our hands on the original research, we can assess the level of evidence of the study methods. In PubMed, I discovered that the original study was written in Norwegian, but the abstract was available in English. There is enough information available in the abstract, however, to assess the level of evidence. The sample size was 202, a moderate number of subjects, but had disproportionately more men than women, more than most whiplash studies. It was a retrospective study, taking place up to three years post-accident, and data was collected by a questionnaire. A control group was identified. The study was reproduced three years later with similar results. Strictly on the merits of its study design, I would place this study in the mid-range level of evidence.
Due Diligence
Mr. Luchau exercised full disclosure in stating that there is widespread disagreement on diagnosis, treatment, and even terminology of injuries referred to as whiplash. But he did not extend that disclosure to the controversy surrounding the statement about litigation hindering recovery. I did a simple search of PubMed, typing in the keywords “litigation and whiplash.” The search resulted in 44 citations. Of the 44, nine specifically stated that litigation hindered recovery, 16 found no correlation between litigation and recovery, and four did not address the issue in question. The other citations did not have abstracts or full texts available, so could not be tallied (search conducted on March 28, 2010). The most recent article on the subject, a meta-analysis, reviewed all articles on the subject and concluded that there is no correlation between litigation and recovery.23
While Mr. Luchau meant no harm and clearly had no difficulty finding sources to support his statement, this is an example of how researching the research can help avoid perpetuating information that potentially undermines our clients’ faith and trust in us when treating complex conditions such as whiplash.
Til Luchau Responds
Thank you to Diana Thompson for her work to raise the standard of our professional writing. Regarding tertiary sources, I do think that it can be useful, credible, and appropriate to reference non-primary sources, such as when it is the secondary or tertiary source’s own interpretation or framing of primary information that the author wishes to reference. In my own research for the article, I found the same Google Books source she cites, but decided that Dr. Grottkau’s statements in his New England Journal of Medicine review were clearer and more relevant to my article—and although made in the context of a book review, were phrased as statements in their own right. However, her point about primary sources generally having more academic weight and credibility is very well taken, and will certainly influence my future citation choices.
Although Ms. Thompson disputes my statement that “… legal or insurance difficulties may complicate or even hinder recovery” from whiplash, I do stand behind this assertion, at least as long as it includes the conditional “may.” Most of us have probably worked with clients whose legal or insurance complications are as great a pain in the neck as their physical injuries. Raising our sensitivity to the burden that these challenges can place on recovery was the point of the phrase quoted. I suspect that Ms. Thompson and I would find ourselves on the same side of any controversy about this subject, in agreement with each other about the primacy of the client’s experience and needs.
A licensed massage practitioner since 1984, Diana Thompson has created a varied and interesting career out of massage: from specializing in pre- and postsurgical lymph drainage to teaching, writing, consulting, and volunteering. Her consulting includes assisting insurance carriers on integrating massage into insurance plans and educating researchers on massage therapy theory and practice to ensure research projects and protocols are designed to match how we practice. Contact her at soapsage@comcast.net.
Notes
1. Available at www.lib.umd.edu/guides/primary-sources.html (accessed May 2010).
2. Available at http://library.uwsp.edu/guides/webtutorials/primary.htm (accessed May 2010).
3. University of St. Thomas, “Primary, Secondary, and Tertiary Sources.” Available at www.stthomas.edu/libraries/research/tutorials/basic/sourcetype.html (accessed May 2010).
4. Available at http://library.uwsp.edu/guides/webtutorials/primary.html (accessed May 2010).
5. University of St. Thomas, “Primary, Secondary, and Tertiary Sources.”
6. Available at www.lib.umd.edu/guides/primary-sources.html (accessed May 2010).
7. Available at http://library.uwsp.edu/guides/webtutorials/primary.htm (accessed May 2010).
8. Available at www.lib.umd.edu/guides/primary-sources.html (accessed May 2010).
9. University of St. Thomas, “Primary, Secondary, and Tertiary Sources.”
10. Colorado State University, Writing @ CSU, “Glossary of Key Terms.” Available at http://writing.colostate.edu/guides/research/glossary/ (accessed May 2010).
11. S. Sidani et al., “Influence of Treatment Preferences on Validity: A Review,” Canadian Journal of Nursing Research 41, no. 4 (December 2009): 52–67.
12. C.M. Hughes et al., “Reflexology for the Treatment of Pain in People with Multiple Sclerosis: a Double-Blind Randomized Sham-Controlled Clinical Trial,” Multiple Sclerosis 15, no. 11 (November 2009): 1329–38.
13. L. Kimber et al., “Massage or Music for Pain Relief in Labour: A Pilot Randomized Placebo Controlled Trial,” European Journal of Pain 12, no. 8 (November 2008): 961–9.
14. K.M. Wesa and B.R. Cassileth, “Is There a Role for Complementary Therapy in the Management of Leukemia?” Expert Review of Anticancer Therapy 9, no. 9 (September 2009): 1241–9.
15. Lacomba M. Torres et al., “Effectiveness of Early Physiotherapy to Prevent Lymphoedema After Surgery for Breast Cancer: Randomized, Single Blinded, Clinical Trial,” British Medical Journal 340 (January 12, 2010): b5396.
16. R. Ogai et al., “Effects of Petrissage Massage on Fatigue and Exercise Performance Following Intensive Cycle Pedaling,” British Journal of Sports Medicine 42, no. 10 (October 2008): 834–8.
17. A. Moraska, “Sports Massage. A Comprehensive Review,” Journal of Sports Medicine & Physical Fitness 45, no. 3 (September 2005): 370–80.
18. Andrew Malleson, “Whiplash and Other Useful Illnesses.” Available at http://books.google.com/books?id=naDghccZhEwC&printsec=frontcover&dq=whiplash+and+other+useful+illnesses&cd=1#v=onepage&q=&f=false (accessed May 2010).
19. E. Loundsberry, “Doctor, 73, Lauded as Charitable, Caring, Gets Jailed for Mail Fraud,” Philadelphia Inquirer (August 31, 1998).
20. J. Lyons, “It’s Not a Wonderful Situation,” Forbes (February 4, 1991): 90–1.
21. J.E. Mack, Alien Encounters, Human Encounters with Aliens (New York: Simon & Schuster).
22. H. Schrader et al., “Examination of the Diagnostic Validity of ‘Headache Attributed to Whiplash Injury’: A Controlled, Prospective Study.” Available at www.ncbi.nlm.nih.gov/pubmed/17038037 (accessed May 2010).
23. N.M. Spearing and L.B. Connelly, “Is Compensation ‘Bad for Health’? A Systematic Meta-Review,” Injury (January 7, 2010).