The visibility of a research paper extends beyond the journal in which the paper appears, and today includes press coverage, discussion boards, blog posts, post-publication reference managers and downloads. To best represent the visibility and influence of individual articles, the scientific community is turning toward article-level metrics, which in the broadest sense are metrics that apply at the article level rather than the journal level. The field has rapidly expanded, and in addition to the more academic tracking related to citation and downloads, there is now the ability to track the more social aspects of how articles are used, shared and discussed whether via CiteULike, Facebook or Twitter. Article-level metrics provide granular information on how individual articles are used.
Article-level metrics in practice currently come in two flavors: ALMs and altmetrics. ALMs refer to the suite of metrics, from social to academic, with a focus on academic tracking and altmetrics refers to the suite of metrics, from social to academic, with a focus on social tracking. Authors, publishers, funders and patients turn to one or another, or both, depending on need. PLOS ALMs provide metrics across a suite of sources that includes HTML, XML, and PDF downloads, PubMed Central use, Scopus, Web of Science and CrossRef citations, CituULike and Mendeley reference management services, Researchblogging.org and Nature Blogs, and social media platforms Twitter and Facebook.
Authors Engage
Researchers use article-level metrics not only for their own interest in tracking an article’s distribution and use, but also actively, to showcase the influence of their work in tenure and promotion packages. When scientist Steve Pettifer at the University of Manchester, UK, applied for promotion in 2013 he incorporated article-level metrics, rather than just citations or downloads, into his promotion package to best showcase the influence of his 2008 PLOS Computational Biology review article, still the most viewed and downloaded review to be published in any PLOS journal. Although it’s not possible to know if the metrics helped him get the promotion, “I’m definitely a convert,” he says, according to his interview in Nature. For Emilio Bruna, professor, Department of Wildlife Ecology & Conservation, University of Florida, active use of Impactstory metrics was a positive experience. “I included Impactstory data in my portfolios for promotion to full professor and selection to UF’s Academy of Distinguished Teaching Scholars,” he says. Both applications were successful.
ALMs are an “intuitive way for non-scientists to understand the relevance and interest of seemingly esoteric research,” says Andrew Farke, expert in dinosaur evolution and Augustyn Family Curator of Paleontology in Claremont, California. For a privately funded researcher “metrics can be helpful to show supporters of our research program the impact that is made by their contributions,” he says. ALMs benefit readers of scientific literature at all levels by helping to guide them to the most important and influential work among the overwhelming amount of scientific literature published today, both in Open Access and subscription journals. Those leveraging ALMs and the inherent post-publication filtering of articles include Graham Steel, Patient Advocate and Open Access proponent. “ALM’s are a valuable tool that we now have,” he says. “I prioritize my reading in part based on ALMs.”
Publishers Respond
PLOS was one of the earliest publishers to offer ALMs on all articles. As authors increasingly want to track the influence of their work and use article-level metrics, additional publishers have responded. Open Access publishers BioMed Central, PeerJ, Copernicus Publications, Frontiers, eLife and the Public Knowledge Project (PKP) all publish metrics with their articles. Journals published with the PKP’s Open Journal System often operate independently, with “few resources and in developing regions,” says Juan Pablo Alperin, Researcher with the PKP and PhD candidate in Education, Stanford University. Article-level metrics not only help authors better track their impact but also “help authors better understand their audiences,” he says.
Article-level metrics are not just for Open Access journals. Subscription-based publishers see the value in them as well. Springer publishes more than 2,200 journals, approximately 325 of them Open Access. The company “is changing from a sole focus on the journal impact factor to providing multiple metrics” to authors and editors, says Martijn Roelandse, Publishing Editor, Neurosciences, according to The Chronicle of Higher Education. Springer’s ALMs include CrossRef to measure citations and its own download statistics to track use. The growing list of subscription publishers that provide article-level metrics includes The Rockefeller University Press, Cell Press, Nature Publishing Group and John Wiley & Sons.
Publishers are joining forces, as well, to innovate and leverage their individual ALM experiences. As of June 2014, CrossRef Labs, a project of CrossRef and PLOS, has indexed 748,254 articles with PLOS’ ALM application, available through an open API, for the purpose of improving technical aspects prior to industry-wide scale up of ALM use. PLOS was a pioneer in the development of ALM applications, and as a strong advocate of Open Access encourages others to develop additional tools through the shared API.
Funders and Institutions Determine Value
Joining publishers in the shift away from use of journal-based metrics as a surrogate measure of individual research article quality and impact are funders and institutions. The momentum of this shift is seen in the large increase of signatories to the Declaration of Research Assessment (DORA) statement, originally drafted by publishers and editors of scholarly journals in December 2012. As of June 2014, 18 months since inception, the declaration has 10,668 individual and 467 organizational signatories around the globe, increases of nearly 6,800 and 470 percent, respectively. DORA signatories acknowledge that improvements in research assessment, including article-level metrics, are critical to “increase the momentum toward more sophisticated and meaningful approaches” to evaluate research. Institutional funders of the statement include US-based Howard Hughes Medical Institute, UK-based National Health and Medical Research Council and France-based INSERM.
The Gordon and Betty Moore Foundation believes that traditional metrics such as citation count “do not adequately capture the impact of research,” says Vicki Chandler, Chief Program Officer, Science, of the Moore Foundation. “ALMs are part of a growing set of scholarly output measures that we are paying attention to.” Funders turn to ALMs on a case-by-case basis for a more comprehensive understanding of how their grant funds influence research, promotions, education and policy, to satisfy government requirements to make research outcomes more transparent, and to monitor the broader impact of the grants they distribute. In the UK, The Research Excellence Framework, a system for assessing research quality in UK higher education institutions that has influence over funding decisions, allows scientists to use altmetrics to demonstrate the social impact of their research in their reports. The Wellcome Trust specifically states in its Open Access Policy that “it is the intrinsic merit of the work,” rather than the particular journal the work appears in, that should be considered in funding decisions. ALMs, in part, help establish this intrinsic merit.
Progress in ALM adoption is also taking place at research institutions. The University of Pittsburgh uses the dashboard of ALM aggregator PlumX to display the impact of researchers from a cross-section of departments throughout the university, facilitating total institute and departmental metrics. The university is working to increase use and practical application of ALMs through lunch and learn trainings, with one recent seminar focused on “Using Altmetrics to Demonstrate Scholarly Impact.” Trainings include “toolbox tips” on best practices for library colleagues to use with the broader University of Pittsburgh community. Practices such as this propel use of ALMs forward.
Businesses Invest and Organizations Collaborate
Article-level metrics make good business too. Digital Science, launched by MacMillan Publishers, became an investor in the startup Altmetric.com in 2012. The growing list of organizations and companies outside the publishing industry that incorporate ALMs in their business offerings include Mendeley, CrossRef and Impactstory. Since August 2013, Impactstory has more than doubled its user base, says Stacy Konkiel, Director of Marketing & Research. Most recently, commercial publishing firm EBSCO Information Services acquired Plum Analytics in January 2014 for its PlumX database that aggregates altmetrics.
Industry collaborations on article-level metrics, promoted by interest in ALMs, are moving forward to improve technical and practical usability standards. Efforts such as NISO’s Alternative Assessment Metrics Initiative and CrossRef Labs’ pilot with PLOS ALMs indicate that standard organizations are working together to explore, identify, and advance metrics standards and best practices. The annual PLOS ALM conferences, funded by the Alfred P. Sloan Foundation, are growing in number of attendees, diversity of organizations represented and overall popularity. In 2013, 86 participants attended the PLOS ALM Workshop, up 72 percent from the 50 attendees in 2012, representing the research, publishing, funding and technology industries in Europe, Canada, and the United States. New to the conference in 2013 was Annual Reviews, which each year publishes 46 extensive review volumes focused on disciplines within the Biomedical, Life, Physical, Social Sciences and Economics. As for 2012 and 2013, PLOS will organize the 2014 ALM Workshop planned for December 4-5 2014. A parallel event in London this year is also in the works.
Patient Communities Become Users
Even the patient community wants to understand ALMs and how they might benefit from their use. In 2012 the Health Research Alliance (HRA), a national consortium that now numbers 57 private foundations and public charities that fund biomedical research, sent a representative to the meeting in order to understand ALMs “as a tool to help assess the career progress of funded investigators,” as an “indicator of progress toward a treatment goal” and even for donor cultivation purposes, says Kate Ahlport, Executive Director of HRA. Patient organizations and disease foundations such as Autism Speaks attended the 2013 ALM conference as did the Patient-Centered Outcomes Research Institute, PCORI, an organization mandated by President Obama’s Affordable Care Act with a focus on improving healthcare delivery and outcomes.
Harnessing Momentum
The peer-reviewed research paper remains a central research output that informs research assessment. Those currently using article-level metrics have the opportunity to track the impact and use of their work post-publication and also to promote, post-publication, the visibility of their work within the research community and beyond to the larger public. The challenge for ALM developers is to convert passive observers to active users. Efforts underway to increase and improve practical use of ALMs ultimately will increase the benefit to, and use by, those putting in the hard work to generate the manuscripts. These examples show how article-level metrics help paint the evolving picture of a published work over time. The suite of metrics, from social to academic, provides a comprehensive assessment of an article’s influence, both immediate and over time. Those who embrace the use of these metrics have a deeper understanding of the impact and influence of individual scholarly works. As ALM tools improve, greater adoption and practical use are certain to follow.