“…Russel Lyons who posits that Christakis' and Fowler's work is a great example of statistical illiteracy, and that the conclusion drawn from their data, that obesity is socially contagious, is severely flawed and can't be made.”
Blogger Yoni Freedhoff, MD, in his blog “Weighty Matters” (1)
Scholarly blogs are one of the most prominent information sources for altmetrics and are reported in the main altmetric services (e.g., ImpactStory.org, altmetric.com). National Geographic, the Nature Group, Scientific American, and the PLOS (Public Library of Science) journals all have science blogging networks. Scholarly blogs have been defined as “blogs written by academic experts that are dedicated in large part to scientific content” (p. 171) (2). This definition is rather vague, because of the difficulty defining an “expert” and “scholarly content.” A scholarly blog can also be defined by its platform (e.g., SciLogs, Scienceblogs.com), by the media outlet hosting it (e.g., Scientific American, The Guardian), by the affiliation or education of its blogger(s), by its contents, discipline, links to other blogs (blog roll) or any combination of the above. A blog by a single graduate student posting about subjects related to her research can be as ‘scholarly’ as a blog by several experienced researchers posting across disciplines.
A number of studies have implicated blog coverage as an indicator of scholarly impact. The commercial firm altmetric.com’s data from July 2011 up to January 1st, 2013 has been used to study the association of potential altmetric sources (Twitter, Facebook wall posts, research highlights, blogs, mainstream media, forums, Google+, LinkedIn, Pinterest, question and answer sites, and Reddit) and Web of Science (WoS) citations (3). The blogs in the sample came from the Nature.com blogging network and the blogging aggregators ResearchBlogging (RB) and ScienceSeeker. The study compared the number of times an article was covered in blogs (they calculated each altmetric source separately), with two articles that have also received a mention in an altmetric source (not necessarily a blog), one published shortly before the article in question and the second published shortly after. The authors concluded that “In summary, there is clear evidence that three altmetrics (tweets, FbWalls, blogs) tend to associate with citations at the level of individual journals” (p. 4).
Another study of altmetric.com data (4) looked at altmetric mentions of articles (with DOIs - Digital Object Identifiers) in various metric sources (reference managers excluded) from July 2011 to mid-October 2013, and correlated them with articles indexed in WoS and the citations they accumulated in 2012 (in part of the analysis the corpus was used in full, in another part only the July-December 2011 data, to allow a full year of citations). They found a relatively strong correlation between blog and news outlet mentions and citations. A factor analysis found that blog and news outlet mentions belonged to one dimension, while other altmetrics (Twitter, Google+ and Facebook walls) belonged to another. This aligns with Taylor and Plume (5), who studied altmetric.com data from the last four months of 2013. Taylor and Plume classified the altmetric data sources into four categories: social activity (e.g., Facebook, Twitter), scholarly activity (e.g., bookmarking on Mendeley), scholarly commentary (e.g., blogs, F1000Prime) and mass media coverage. They found that between the top 0.5% of articles in each category, the highest chance of overlap was between mass media coverage and scholarly commentary.
A small-scale study (6) looked at the effect of blog post coverage on 16 clinical pain PLOS ONE articles. The blog posts were published in the blog BodyinMind.org, which had at that time over 2,500 unique views per week, and were disseminated by social media (RB, Twitter, Facebook, LinkedIn). In the week after the blog post coverage of each article, there were on average about 3 additional downloads of the article per day and 12 additional HTML views. The authors did not find a correlation between Scopus citations a year after the blog post publications and social media metrics or HTML views, but did find a moderate correlation between PDF downloads and citations.
The structured blog citation
Scholarly bloggers often comment on material from peer-reviewed journals, but unlike authors of peer-reviewed articles, they are not obligated to reference their sources in a formal way. Despite this, scientific bloggers have mentioned in interviews that they would have liked to use references in a similar way to the way that they cite in scholarly articles (7).
The aggregator ResearchBlogging.org (RB) was built to answer this need. Launched in late 2008, it aggregates blog posts referring specifically to peer-reviewed research. It is a self-selecting aggregator that allows bloggers to refer to peer-reviewed research in an academic citation format. Bloggers discussing peer-reviewed research can register with the aggregator and after they mark relevant posts in their blog, these posts appear on the aggregator site, giving one-stop access to a variety of research reviews from different authors. The site's human editors ensure that blogs submitted to the aggregator follow its guidelines and are of appropriate quality. RB already has an altmetric role; it currently serves as one of the article level metrics (ALM) displayed for each article in the journal PLOS ONE (8). By the end of 2011, RB had more than 1,230 active blogs and about 27,000 posts (9). These posts seem to be a transitional phase between traditional scholarly discourse and rapid, informal blog writing - a scientometric Archaeopteryx.
The first study of RB, which looked at its Chemistry category, found that most blog posts were about current research and came from high-impact niche journals as well as prestigious multidisciplinary journals (10). Similar results were also found in subsequent studies (9), (11), (12) for other RB categories. Bloggers prefer to cover articles from top multidisciplinary journals, the most popular being (in alphabetical order) Nature, PLOS ONE, Proceedings of the National Academy of Sciences of the United States of America (PNAS) and Science. Most of the posts aggregated in RB are written in English. The bloggers classify their posts into pre-defined categories, the most popular categories being Biology, Health Sciences, Neuroscience and Psychology (9), (11).
RB (see Figure 1) was the data source for our blog study of the association between blog coverage and traditional citations (13). We took a different approach than (3) and (4), not taking into account the number of times an article was covered in blogs, but only whether it was covered or not. We compared journal articles from 2009 and 2010 which were covered in blog posts from the same year (i.e., a 2009 article covered in a 2009 post, a 2010 article covered in a 2010 post) with the general population of articles from the same journal in the same year, to see if these articles received a higher number of citations in the years after their publication in comparison to articles from the same journal and year not covered in blogs. In 2009 a total of 58% (7 out of 12 journals), and in 2010 a total of 68% (13 out of 19 journals) of journals published articles covered by blogs that attracted more citations than articles from the same journal and year that were not covered by blogs. The most striking difference in medians was between articles covered by the New England Journal of Medicine (NEJM) in 2009 (172) and NEJM articles from 2009 which were not covered in blogs (76). We also found an association between coverage of the NEJM articles in blogs and their coverage in the Reuters and New York Times websites. Twenty-one out of the 26 NEJM articles in our 2009 sample (81%) and 20 out of 38 (53%) NEJM sample articles in 2010 were covered by Reuters, the New York Times, or both. This aligns with the findings of (5) as well as those presented in (4). News coverage has been known to correlate with a higher level of citations (14), and it is a possibility that the higher level of citations that many articles covered in blogs enjoy reflects the bloggers’ tendency to choose articles covered by mass media. We cannot tell if this tendency comes from the direct influence of mass media coverage on its scholarly blogger consumers, or if the bloggers’ tastes simply align with those of the mass media.
Figure 1 - a typical RB post snippet.
In conclusion
There is evidence that blog coverage of scholarly articles associates with increased visibility and impact. Unfortunately, there are a number of obstacles that might limit the use of blog posts as an altmetric source. First, only a small percentage of articles is covered in blogs (e.g., 1.9% of the articles studied in (4)).
Second, the definition of “scholarly blogs” and the decision about which blog data to use is problematic. When relying on certain aggregators or networks for blog data we miss the impact of articles covered by blogs outside the data collection range. The coverage problem is not specific to blogs, or even to altmetrics, but extends to bibliometric databases, which also have to choose which sources to index.
Third, there is a lack of sustainability. While most peer-reviewed journals enjoy professional archiving and printed copies, blogs can close down or move without leaving a trace (except perhaps in archive.org and similar sites). For blog-derived data to be reliable, they have to be better indexed and archived.
If peer-reviewed journals citations are “frozen footprints,” (15, abstract) then citations in blogs, and altmetrics in general, are footprints in quicksand. In spite of these limitations, we consider blogs to be an especially promising altmetric source.
The effort required to write a blog post (assuming it isn’t spam or computer-generated) is much greater than the effort needed to tweet, “like” or bookmark an article. Scholarly blogging at its best can be a type of post-publication peer-review, scholarly commentary or citizen journalism and its presence can be used as an impact indicator.
References
(1) Freedhoff, Y. (2011, June 20) “Obesity is contagious, or is it? A sober second look at obesity and social networks.” [BlogPost]. Retrieved from http://www.weightymatters.ca/2011/06/obesitys-contagious-or-is-it-sober.html
(2) Puschmann, C., Mahrt, M. (2012) “Scholarly blogging: A new form of publishing or science journalism 2.0?” In: A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 171-182). Düsseldorf: Düsseldorf University Press.
(3) Thelwall, M., Haustein, S., Larivière, V. & Sugimoto, C. (2013) “Do altmetrics work? Twitter and ten other social Web services”, PLOS ONE, Vol.8, No. 5, e64841.
(4) Costas, R., Zahedi, Z. & Wouters, P. (2014) “Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective”. ArXiv preprint arXiv:1401.4321.
(5) Taylor, M. & Plume, A. (2014) “Party papers or policy discussions: an examination of highly shared papers using altmetric data”, Research Trends, Issue 36, 17-20.
(6) Allen, H.G., Stanton, T.R., Di Pietro, F. & Moseley, G.L. (2013) “Social media release increases dissemination of original articles in the clinical pain sciences”, PLOS ONE, Vol. 8, No. 7, e68914.
(7) Kjellberg, S. (2010) “I am a blogging researcher: Motivations for blogging in a scholarly context” First Monday, Vol. 15, No. 8.
(8) PLOS ONE. (n.d.) “Article-level metrics information”, Retrieved from http://www.plosone.org/static/almInfo
(9) Fausto, S., Machado, F.A., Bento, L.F., Iamarino, A., Nahas, T.R. & Munger, D.S. (2012) “Research blogging: Indexing and registering the change in science 2.0”, PLOS ONE, Vol.7, No. 12, e50109.
(10) Groth, P. & Gurney, T. (2010) “Studying scientific discourse on the Web using bibliometrics: A chemistry blogging case study”. Presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC. Retrieved from http://wiki.few.vu.nl/sms/images/9/9c/Websci10-FINAL-29-4-2010f.pdf
(11) Shema, H., Bar-Ilan, J. & Thelwall, M. (2012) “Research blogs and the discussion of scholarly information”, PLOS ONE, 7(5), e35869.
(12) Shema, Bar-Ilan & Thelwall (in press) “How is research blogged? A content analysis approach”, Journal of the Association for Information Science and Technology.
(13) Shema, H., Bar-Ilan, J. and Thelwall, M. (2014) “Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics.” Journal of the Association for Information Science and Technology. Advanced online publication doi: 10.1002/asi.23037
(14) Phillips, D.P., Kanter, E.J., Bednarczyk, B. & Tastad, P.L. (1991) “Importance of the lay press in the transmission of medical knowledge to the scientific community”, New England Journal of Medicine, Vol. 325, No. 16, pp. 1180–1183.
(15) Cronin, B. (1981) “The need for a theory of citing”, Journal of Documentation, Vol. 37, No.1, pp. 16–24.