Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics
---
paper_title: Correlation and Interaction Visualization of Altmetric Indicators Extracted From Scholarly Social Network Activities: Dimensions and Structure

paper_content:

Background: Citation counts for peer-reviewed articles and the impact factor of journals have long been indicators of article importance or quality. In the Web 2.0 era, growing numbers of scholars are using scholarly social network tools to communicate scientific ideas with colleagues, thereby making traditional indicators less sufficient, immediate, and comprehensive. In these new situations, the altmetric indicators offer alternative measures that reflect the multidimensional nature of scholarly impact in an immediate, open, and individualized way. In this direction of research, some studies have demonstrated the correlation between altmetrics and traditional metrics with different samples. However, up to now, there has been relatively little research done on the dimension and interaction structure of altmetrics. Objective: Our goal was to reveal the number of dimensions that altmetric indicators should be divided into and the structure in which altmetric indicators interact with each other. Methods: Because an article-level metrics dataset is collected from scholarly social media and open access platforms, it is one of the most robust samples available to study altmetric indicators. Therefore, we downloaded a large dataset containing activity data in 20 types of metrics present in 33,128 academic articles from the application programming interface website. First, we analyzed the correlation among altmetric indicators using Spearman rank correlation. Second, we visualized the multiple correlation coefficient matrixes with graduated colors. Third, inputting the correlation matrix, we drew an MDS diagram to demonstrate the dimension for altmetric indicators. For correlation structure, we used a social network map to represent the social relationships and the strength of relations. Results: We found that the distribution of altmetric indicators is significantly non-normal and positively skewed. The distribution of downloads and page views follows the Pareto law. Moreover, we found that the Spearman coefficients from 91.58% of the pairs of variables indicate statistical significance at the .01 level. The non-metric MDS map divided the 20 altmetric indicators into three clusters: traditional metrics, active altmetrics, and inactive altmetrics. The social network diagram showed two subgroups that are tied to each other but not to other groups, thus indicating an intersection between altmetrics and traditional metric indicators. Conclusions: Altmetrics complement, and most correlate significantly with, traditional measures. Therefore, in future evaluations of the social impact of articles, we should consider not only traditional metrics but also active altmetrics. There may also be a transfer phenomenon for the social impact of academic articles. The impact transfer path has transfer, or intermediate, stations that transport and accelerate article social impact from active altmetrics to traditional metrics and vice versa. This discovery will be helpful to explain the impact transfer mechanism of articles in the Web 2.0 era. Hence, altmetrics are in fact superior to traditional filters for assessing scholarly impact in multiple dimensions and in terms of social structure. [J Med Internet Res 2013;15(11):e259]
---
paper_title: Health research: measuring the social, health and economic benefits

paper_content:

Despite intense interest in defining the social, health and economic impacts of health research investments globally[1][1]–[5][2] and in Canada[6][3]–[10][4] as proof of value-for-money, no validated method for measuring return on investments yet exists. Until now, issues of complexity combined
---
paper_title: Article-Level Metrics and the Evolution of Scientific Impact

paper_content:

Formally published papers that have been through a traditional prepublication peer review process remain the most important means of communicating science today. Researchers depend on them to learn about the latest advances in their fields and to report their own findings. The intentions of traditional peer review are certainly noble: to ensure methodological integrity and to comment on potential significance of experimental studies through examination by a panel of objective, expert colleagues. In principle, this system enables science to move forward on the collective confidence of previously published work. Unfortunately, the traditional system has inspired methods of measuring impact that are suboptimal for their intended uses.
---
paper_title: A Look at Altmetrics and Its Growing Significance to Research Libraries

paper_content:

This document serves as an informational review of the emerging field and practices of alternative metrics or altmetrics. It is intended to be used by librarians and faculty members in research libraries and universities to better understand the trends and challenges associated with altmetrics in higher education. It is also intended to be used by research libraries to offer guidance on how to participate in shaping this emerging field.
---
paper_title: Usage Bibliometrics

paper_content:

Scholarly usage data provides unique opportunities to address the known shortcomings of citation analysis. However, the collection, processing and analysis of usage data remains an area of active research. This article provides a review of the state-of-the-art in usage-based informetric, i.e. the use of usage data to study the scholarly process.
---
paper_title: How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications

paper_content:

In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readerships for 62.6 % of all the publications studied, other sources only provide marginal information. In terms of relation with citations, a moderate spearman correlation (r = 0.49) has been found between Mendeley readership counts and citation indicators. Other possibilities and limitations of these indicators are discussed and future research lines are outlined.
---
paper_title: What is societal impact of research and how can it be assessed? a literature survey

paper_content:

Since the 1990s, the scope of research evaluations becomes broader as the societal products outputs, societal use societal references, and societal benefits changes in society of research come into scope. Society can reap the benefits of successful research studies only if the results are converted into marketable and consumable products e.g., medicaments, diagnostic tools, machines, and devices or services. A series of different names have been introduced which refer to the societal impact of research: third stream activities, societal benefits, societal quality, usefulness, public values, knowledge transfer, and societal relevance. What most of these names are concerned with is the assessment of social, cultural, environmental, and economic returns impact and effects from results research output or products research outcome of publicly funded research. This review intends to present existing research on and practices employed in the assessment of societal impact in the form of a literature survey. The objective is for this review to serve as a basis for the development of robust and reliable methods of societal impact measurement.
---
paper_title: Validating online reference managers for scholarly impact measurement

paper_content:

This paper investigates whether CiteULike and Mendeley are useful for measuring scholarly influence, using a sample of 1,613 papers published in Nature and Science in 2007. Traditional citation counts from the Web of Science (WoS) were used as benchmarks to compare with the number of users who bookmarked the articles in one of the two free online reference manager sites. Statistically significant correlations were found between the user counts and the corresponding WoS citation counts, suggesting that this type of influence is related in some way to traditional citation-based scholarly impact but the number of users of these systems seems to be still too small for them to challenge traditional citation indexes.
---
paper_title: The power of altmetrics on a CV

paper_content:

CON T E N T S NEX T PAGE > NEX T ART I C L E > < PRE V I OUS PAGE EDITOR’S SUMMARY Alternative metrics demonstrate the value and influence of scholars’ work apart from traditional citation counts and can enhance the impact of a CV. Altmetrics provide additional, supplementary information and can balance misleading metrics tied to particular journals. More timely than traditional metrics, altmetrics quickly reveal the impact of recent work and add authority to different types of scholarly products not captured as articles. Altmetrics can capture social media references that escape traditional metrics and reflect public engagement prompted by scholarly writing. The availability of altmetrics expands publishing opportunities to include new venues and stimulates innovative strategies for evaluating research. When included in a CV altmetrics must be accurate, clear and meaningful.
---
paper_title: Web citation data for impact assessment: A comparison of four science disciplines

paper_content:

The number and type of Web citations to journal articles in four areas of science are examined: biology, genetics, medicine, and multidisciplinary sciences. For a sample of 5,972 articles published in 114 journals, the median Web citation counts per journal article range from 6.2 in medicine to 10.4 in genetics. About 30% of Web citations in each area indicate intellectual impact (citations from articles or class readings, in contrast to citations from bibliographic services or the author's or journal's home page). Journals receiving more Web citations also have higher percentages of citations indicating intellectual impact. There is significant correlation between the number of citations reported in the databases from the Institute for Scientific Information (ISI, now Thomson Scientific) and the number of citations retrieved using the Google search engine (Web citations). The correlation is much weaker for journals published outside the United Kingdom or United States and for multidisciplinary journals. Web citation numbers are higher than ISI citation counts, suggesting that Web searches might be conducted for an earlier or a more fine-grained assessment of an article's impact. The Web-evident impact of non-UK/USA publications might provide a balance to the geographic or cultural biases observed in ISI's data, although the stability of Web citation counts is debatable.
---
paper_title: Altmetric: enriching scholarly content with article-level discussion and metrics

paper_content:

Scholarly content is increasingly being discussed, shared, and bookmarked online by researchers. Altmetric is a start-up that focuses on tracking, collecting, and measuring this activity on behalf of publishers; here we describe our approach and general philosophy. Over the past year we have seen sharing and discussion activity around approximately 750,000 articles. The average number of articles shared each day grows by 5–10% a month. We look at examples of how people are interacting with papers online and at how publishers can collect and present the resulting data to deliver real value to their authors and readers.
---
paper_title: Is there currently a scientific revolution in scientometrics?

paper_content:

The author of this letter to the editor would like to set forth the argument that scientometrics is currently in a phase in which a taxonomic change, and hence a revolution, is taking place. One of the key terms in scientometrics is scientific impact which nowadays is understood to mean not only the impact on science but the impact on every area of society.
---
paper_title: Social Scholarship: Reconsidering Scholarly Practices in the Age of Social Media.

paper_content:

This conceptual exploration inquires, what is scholarship reconsidered in the age of social media? How ought we to conceptualize social scholarship-a new set of practices being discussed in various disciplines? The paper offers a critical examination of the practical and policy implications of reconsidering scholarship in light of social media's affordances toward a conceptualization of social scholarship. For each dimension of Boyer's original framework, we explain its epistemologies and practices. Next, we take a critical approach to inquiring how each dimension, reconsidered through the lens of social scholarship values and social media affordances, might be envisioned today. This exploration provides concrete examples of how scholars might enact social scholarship with what benefits and challenges. [ABSTRACT FROM AUTHOR]
---
paper_title: Using social bookmarks and tags as alternative indicators of journal content description

paper_content:

Qualitative journal evaluation cumulates content descriptions of single articles. Articles are either represented by author–generated keywords, professionally indexed subject headings, automatically extracted terms or, as recently introduced, by reader–generated tags as used in social bookmarking systems. The study presented here shows that different types of keywords each reflect a different perspective on documents and that tags can be used in journal evaluation to represent a reader–specific view. After providing a broad theoretical background and literature review, methods for extensive automatic term cleaning and calculation of term overlaps are introduced. The efficiency of tags and other metadata for journal content description is illustrated for one particular journal.
---
paper_title: Research Blogging: Indexing and Registering the Change in Science 2.0

paper_content:

Increasing public interest in science information in a digital and 2.0 science era promotes a dramatically, rapid and deep change in science itself. The emergence and expansion of new technologies and internet-based tools is leading to new means to improve scientific methodology and communication, assessment, promotion and certification. It allows methods of acquisition, manipulation and storage, generating vast quantities of data that can further facilitate the research process. It also improves access to scientific results through information sharing and discussion. Content previously restricted only to specialists is now available to a wider audience. This context requires new management systems to make scientific knowledge more accessible and useable, including new measures to evaluate the reach of scientific information. The new science and research quality measures are strongly related to the new online technologies and services based in social media. Tools such as blogs, social bookmarks and online reference managers, Twitter and others offer alternative, transparent and more comprehensive information about the active interest, usage and reach of scientific publications. Another of these new filters is the Research Blogging platform, which was created in 2007 and now has over 1,230 active blogs, with over 26,960 entries posted about peer-reviewed research on subjects ranging from Anthropology to Zoology. This study takes a closer look at RB, in order to get insights into its contribution to the rapidly changing landscape of scientific communication.
---
paper_title: A new look at evidence of scholarly citation in citation indexes and from web sources

paper_content:

A sample of 1,483 publications, representative of the scholarly production of LIS faculty, was searched in Web of Science (WoS), Google, and Google Scholar. The median number of citations found through WoS was zero for all types of publications except book chapters; the median for Google Scholar ranged from 1 for print/subscription journal articles to 3 for books and book chapters. For Google the median number of citations ranged from 9 for conference papers to 41 for books. A sample of the web citations was examined and classified as representing intellectual or non-intellectual impact. Almost 92% of the citations identified through Google Scholar represented intellectual impact — primarily citations from journal articles. Bibliographic services (non-intellectual impact) were the largest single contributor of citations identified through Google. Open access journal articles attracted more web citations but the citations to print/subscription journal articles more often represented intellectual impact. In spite of problems with Google Scholar, it has the potential to provide useful data for research evaluation, especially in a field where rapid and fine-grained analysis is desirable.
---
paper_title: Altmetrics: Value all research products

paper_content:

A new funding policy by the US National Science Foundation represents a sea-change in how researchers are evaluated, says Heather Piwowar.
---
paper_title: Altmetrics in the wild: Using social media to explore scholarly impact

paper_content:

In growing numbers, scholars are integrating social media tools like blogs, Twitter, and Mendeley into their professional communications. The online, public nature of these tools exposes and reifies scholarly processes once hidden and ephemeral. Metrics based on this activities could inform broader, faster measures of impact, complementing traditional citation metrics. This study explores the properties of these social media-based metrics or "altmetrics", sampling 24,331 articles published by the Public Library of Science. ::: We find that that different indicators vary greatly in activity. Around 5% of sampled articles are cited in Wikipedia, while close to 80% have been included in at least one Mendeley library. There is, however, an encouraging diversity; a quarter of articles have nonzero data from five or more different sources. Correlation and factor analysis suggest citation and altmetrics indicators track related but distinct impacts, with neither able to describe the complete picture of scholarly use alone. There are moderate correlations between Mendeley and Web of Science citation, but many altmetric indicators seem to measure impact mostly orthogonal to citation. Articles cluster in ways that suggest five different impact "flavors", capturing impacts of different types on different audiences; for instance, some articles may be heavily read and saved by scholars but seldom cited. Together, these findings encourage more research into altmetrics as complements to traditional citation measures.
---
paper_title: Citation Analysis in Twitter: Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences

paper_content:

This paper investigates Twitter usage in scientific contexts, particu- larly the use of Twitter during scientific conferences. It proposes a methodology for capturing and analyzing citations/references in Twitter. First results are presented based on the analysis of tweets gathered for two conference hashtags.
---
paper_title: Using altmetrics for assessing research impact in the humanities

paper_content:

The prospects of altmetrics are especially encouraging for research fields in the humanities that currently are difficult to study using established bibliometric methods. Yet, little is known about the altmetric impact of research fields in the humanities. Consequently, this paper analyses the altmetric coverage and impact of humanities-oriented articles and books published by Swedish universities during 2012. Some of the most common altmetric sources are examined using a sample of 310 journal articles and 54 books. Mendeley has the highest coverage of journal articles (61 %) followed by Twitter (21 %) while very few of the publications are mentioned in blogs or on Facebook. Books, on the other hand, are quite often tweeted while both Mendeley's and the novel data source Library Thing's coverage is low. Many of the problems of applying bibliometrics to the humanities are also relevant for altmetric approaches; the importance of non-journal publications, the reliance on print as well the limited coverage of non-English language publications. However, the continuing development and diversification of methods suggests that altmetrics could evolve into a valuable tool for assessing research in the humanities.
---
paper_title: New opportunities for repositories in the age of altmetrics

paper_content:

For institutional repositories, alternative metrics reflecting online activity present valuable indicators of interest in their holdings that can supplement traditional usage statistics. A variable mix of built-in metrics is available through popular repository platforms: Digital Commons, DSpace and EPrints. These may include download counts at the collection and/or item level, search terms, total and unique visitors, page views and social media and bookmarking metrics; additional data may be available with special plug-ins. Data provide different types of information valuable for repository managers, university administrators and authors. They can reflect both scholarly and popular impact, show readership, reflect an institution's output, justify tenure and promotion and indicate direction for collection management. Practical considerations for implementing altmetrics include service costs, technical support, platform integration and user interest. Altmetrics should not be used for author ranking or comparison, and altmetrics sources should be regularly reevaluated for relevance.
---
paper_title: Who reads research articles? An altmetrics analysis of Mendeley user categories

paper_content:

Little detailed information is known about who reads research articles and the contexts in which research articles are read. Using data about people who register in Mendeley as readers of articles, this article explores different types of users of Clinical Medicine, Engineering and Technology, Social Science, Physics, and Chemistry articles inside and outside academia. The majority of readers for all disciplines were PhD students, postgraduates, and postdocs but other types of academics were also represented. In addition, many Clinical Medicine articles were read by medical professionals. The highest correlations between citations and Mendeley readership counts were found for types of users who often authored academic articles, except for associate professors in some sub-disciplines. This suggests that Mendeley readership can reflect usage similar to traditional citation impact if the data are restricted to readers who are also authors without the delay of impact measured by citation counts. At the same time, Mendeley statistics can also reveal the hidden impact of some research articles, such as educational value for nonauthor users inside academia or the impact of research articles on practice for readers outside academia.
---
paper_title: F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations

paper_content:

F1000 is a postpublication peer review service for biological and medical research. F1000 recommends important publications in the biomedical literature, and from this perspective F1000 could be an interesting tool for research evaluation. By linking the complete database of F1000 recommendations to the Web of Science bibliographic database, we are able to make a comprehensive comparison between F1000 recommendations and citations. We find that about 2% of the publications in the biomedical literature receive at least one F1000 recommendation. Recommended publications on average receive 1.30 recommendations, and more than 90% of the recommendations are given within half a year after a publication has appeared. There turns out to be a clear correlation between F1000 recommendations and citations. However, the correlation is relatively weak, at least weaker than the correlation between journal impact and citations. More research is needed to identify the main reasons for differences between recommendations and citations in assessing the impact of publications.
---
paper_title: What Can Article-Level Metrics Do for You?

paper_content:

Article-level metrics (ALMs) provide a wide range of metrics about the uptake of an individual journal article by the scientific community after publication. They include citations, usage statistics, discussions in online comments and social media, social bookmarking, and recommendations. In this essay, we describe why article-level metrics are an important extension of traditional citation-based journal metrics and provide a number of example from ALM data collected for PLOS Biology.
---
paper_title: An Introduction to Social Media for Scientists

paper_content:

Online social media tools can be some of the most rewarding and informative resources for scientists-IF you know how to use them.
---
paper_title: Research Blogging: Indexing and Registering the Change in Science 2.0

paper_content:

Increasing public interest in science information in a digital and 2.0 science era promotes a dramatically, rapid and deep change in science itself. The emergence and expansion of new technologies and internet-based tools is leading to new means to improve scientific methodology and communication, assessment, promotion and certification. It allows methods of acquisition, manipulation and storage, generating vast quantities of data that can further facilitate the research process. It also improves access to scientific results through information sharing and discussion. Content previously restricted only to specialists is now available to a wider audience. This context requires new management systems to make scientific knowledge more accessible and useable, including new measures to evaluate the reach of scientific information. The new science and research quality measures are strongly related to the new online technologies and services based in social media. Tools such as blogs, social bookmarks and online reference managers, Twitter and others offer alternative, transparent and more comprehensive information about the active interest, usage and reach of scientific publications. Another of these new filters is the Research Blogging platform, which was created in 2007 and now has over 1,230 active blogs, with over 26,960 entries posted about peer-reviewed research on subjects ranging from Anthropology to Zoology. This study takes a closer look at RB, in order to get insights into its contribution to the rapidly changing landscape of scientific communication.
---
paper_title: A Look at Altmetrics and Its Growing Significance to Research Libraries

paper_content:

This document serves as an informational review of the emerging field and practices of alternative metrics or altmetrics. It is intended to be used by librarians and faculty members in research libraries and universities to better understand the trends and challenges associated with altmetrics in higher education. It is also intended to be used by research libraries to offer guidance on how to participate in shaping this emerging field.
---
paper_title: How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications

paper_content:

In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readerships for 62.6 % of all the publications studied, other sources only provide marginal information. In terms of relation with citations, a moderate spearman correlation (r = 0.49) has been found between Mendeley readership counts and citation indicators. Other possibilities and limitations of these indicators are discussed and future research lines are outlined.
---
paper_title: A multi-metric approach for research evaluation

paper_content:

Background information is provided about the Web 2.0 related term altmetrics. This term is placed in the context of the broader field of informetrics. The term influmetrics is proposed as a better term for altmetrics. The importance of considering research products and not just scientific publications is highlighted. Issues related to peer review and making funding decisions within a multi-metric approach are discussed and brought in relation with the new metrics field.
---
paper_title: Altmetrics: Value all research products

paper_content:

A new funding policy by the US National Science Foundation represents a sea-change in how researchers are evaluated, says Heather Piwowar.
---
paper_title: A Look at Altmetrics and Its Growing Significance to Research Libraries

paper_content:

This document serves as an informational review of the emerging field and practices of alternative metrics or altmetrics. It is intended to be used by librarians and faculty members in research libraries and universities to better understand the trends and challenges associated with altmetrics in higher education. It is also intended to be used by research libraries to offer guidance on how to participate in shaping this emerging field.
---
paper_title: Citation time window choice for research impact evaluation

paper_content:

This paper aims to inform choice of citation time window for research evaluation, by answering three questions: (1) How accurate is it to use citation counts in short time windows to approximate total citations? (2) How does citation ageing vary by research fields, document types, publication months, and total citations? (3) Can field normalization improve the accuracy of using short citation time windows? We investigate the 31-year life time non-self-citation processes of all Thomson Reuters Web of Science journal papers published in 1980. The correlation between non-self-citation counts in each time window and total non-self-citations in all 31 years is calculated, and it is lower for more highly cited papers than less highly cited ones. There are significant differences in citation ageing between different research fields, document types, total citation counts, and publication months. However, the within group differences are more striking; many papers in the slowest ageing field may still age faster than many papers in the fastest ageing field. Furthermore, field normalization cannot improve the accuracy of using short citation time windows. Implications and recommendations for choosing adequate citation time windows are discussed.
---
paper_title: Coverage and adoption of altmetrics sources in the bibliometric community

paper_content:

Altmetrics, indices based on social media platforms and tools, have recently emerged as alternative means of measuring scholarly impact. Such indices assume that scholars in fact populate online social environments, and interact with scholarly products in the social web. We tested this assumption by examining the use and coverage of social media environments amongst a sample of bibliometricians examining both their own use of online platforms and the use of their papers on social reference managers. As expected, coverage varied: 82 % of articles published by sampled bibliometricians were included in Mendeley libraries, while only 28 % were included in CiteULike. Mendeley bookmarking was moderately correlated (.45) with Scopus citation counts. We conducted a survey among the participants of the STI2012 participants. Over half of respondents asserted that social media tools were affecting their professional lives, although uptake of online tools varied widely. 68 % of those surveyed had LinkedIn accounts, while Academia.edu, Mendeley, and ResearchGate each claimed a fifth of respondents. Nearly half of those responding had Twitter accounts, which they used both personally and professionally. Surveyed bibliometricians had mixed opinions on altmetrics' potential; 72 % valued download counts, while a third saw potential in tracking articles' influence in blogs, Wikipedia, reference managers, and social media. Altogether, these findings suggest that some online tools are seeing substantial use by bibliometricians, and that they present a potentially valuable source of impact data.
---
paper_title: Do Altmetrics Work? Twitter and Ten Other Social Web Services

paper_content:

Altmetric measurements derived from the social web are increasingly advocated and used as early indicators of article impact and usefulness. Nevertheless, there is a lack of systematic scientific evidence that altmetrics are valid proxies of either impact or utility although a few case studies have reported medium correlations between specific altmetrics and citation rates for individual journals or fields. To fill this gap, this study compares 11 altmetrics with Web of Science citations for 76 to 208,739 PubMed articles with at least one altmetric mention in each case and up to 1,891 journals per metric. It also introduces a simple sign test to overcome biases caused by different citation and usage windows. Statistically significant associations were found between higher metric scores and higher citations for articles with positive altmetric scores in all cases with sufficient evidence (Twitter, Facebook wall posts, research highlights, blogs, mainstream media and forums) except perhaps for Google+ posts. Evidence was insufficient for LinkedIn, Pinterest, question and answer sites, and Reddit, and no conclusions should be drawn about articles with zero altmetric scores or the strength of any correlation between altmetrics and citations. Nevertheless, comparisons between citations and metric values for articles published at different times, even within the same year, can remove or reverse this association and so publishers and scientometricians should consider the effect of time when using altmetrics to rank articles. Finally, the coverage of all the altmetrics except for Twitter seems to be low and so it is not clear if they are prevalent enough to be useful in practice.
---
paper_title: What is societal impact of research and how can it be assessed? a literature survey

paper_content:

Since the 1990s, the scope of research evaluations becomes broader as the societal products outputs, societal use societal references, and societal benefits changes in society of research come into scope. Society can reap the benefits of successful research studies only if the results are converted into marketable and consumable products e.g., medicaments, diagnostic tools, machines, and devices or services. A series of different names have been introduced which refer to the societal impact of research: third stream activities, societal benefits, societal quality, usefulness, public values, knowledge transfer, and societal relevance. What most of these names are concerned with is the assessment of social, cultural, environmental, and economic returns impact and effects from results research output or products research outcome of publicly funded research. This review intends to present existing research on and practices employed in the assessment of societal impact in the form of a literature survey. The objective is for this review to serve as a basis for the development of robust and reliable methods of societal impact measurement.
---
paper_title: Five challenges in altmetrics: A toolmaker's perspective

paper_content:

Jean Liu is data curator and blog editor at Altmetric LLP, and Euan Adie is the founder. They can be reached at http://altmetric.com. D riven by the development of new tools for measuring scholarly attention, altmetrics constitute a burgeoning new area of information science. It is an exciting time to be involved in the field since there are so many opportunities to contribute in innovative ways. We develop altmetrics tools and related services at Altmetric LLP, a small London-based start-up founded in 2011 [1]. Like all developers of new altmetrics tools, we frequently encounter challenges in defining what should be measured, accurately collecting attention from disparate sources and making sense of the huge amount of compiled data. We outline five of these challenges in this piece, illustrating them with examples from our experience. It is worth noting that the altmetrics community as a whole comes together regularly to discuss these and other issues, with two open workshops held in 2012 and more planned for the future.
---
paper_title: Do Altmetrics Work? Twitter and Ten Other Social Web Services

paper_content:

Altmetric measurements derived from the social web are increasingly advocated and used as early indicators of article impact and usefulness. Nevertheless, there is a lack of systematic scientific evidence that altmetrics are valid proxies of either impact or utility although a few case studies have reported medium correlations between specific altmetrics and citation rates for individual journals or fields. To fill this gap, this study compares 11 altmetrics with Web of Science citations for 76 to 208,739 PubMed articles with at least one altmetric mention in each case and up to 1,891 journals per metric. It also introduces a simple sign test to overcome biases caused by different citation and usage windows. Statistically significant associations were found between higher metric scores and higher citations for articles with positive altmetric scores in all cases with sufficient evidence (Twitter, Facebook wall posts, research highlights, blogs, mainstream media and forums) except perhaps for Google+ posts. Evidence was insufficient for LinkedIn, Pinterest, question and answer sites, and Reddit, and no conclusions should be drawn about articles with zero altmetric scores or the strength of any correlation between altmetrics and citations. Nevertheless, comparisons between citations and metric values for articles published at different times, even within the same year, can remove or reverse this association and so publishers and scientometricians should consider the effect of time when using altmetrics to rank articles. Finally, the coverage of all the altmetrics except for Twitter seems to be low and so it is not clear if they are prevalent enough to be useful in practice.
---
paper_title: Five challenges in altmetrics: A toolmaker's perspective

paper_content:

Jean Liu is data curator and blog editor at Altmetric LLP, and Euan Adie is the founder. They can be reached at http://altmetric.com. D riven by the development of new tools for measuring scholarly attention, altmetrics constitute a burgeoning new area of information science. It is an exciting time to be involved in the field since there are so many opportunities to contribute in innovative ways. We develop altmetrics tools and related services at Altmetric LLP, a small London-based start-up founded in 2011 [1]. Like all developers of new altmetrics tools, we frequently encounter challenges in defining what should be measured, accurately collecting attention from disparate sources and making sense of the huge amount of compiled data. We outline five of these challenges in this piece, illustrating them with examples from our experience. It is worth noting that the altmetrics community as a whole comes together regularly to discuss these and other issues, with two open workshops held in 2012 and more planned for the future.
---
paper_title: The evaluation of research by scientometric indicators

paper_content:

Introduction Basic categories of scientometrics Classification of the indicators of evaluative scientometrics Publication growth in science Scientific eminence of journals: the Garfield Factor and the Current Contribution Index The ageing of scientific information Scientometric indicators for the assessment of publications Reference strategy: model of manifested communication through publications Frequency of and strength of motives in referencing: the Reference Threshold Model Research contribution and share of credit of individual authors Standards in scientometric assessments Scientometric assessments: application of scientometrics for the purposes of science policy Institutionalisation of Scientific Information: a scientometric model (ISI-S model) Conclusions.
---
paper_title: The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits

paper_content:

Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describes the opportunities and limits and the advantages and disadvantages of using percentiles in bibliometrics. We also address problems in the calculation of percentiles and percentile rank classes for which there is not (yet) a satisfactory solution. It will be hard to compare the results of different percentile-based studies with each other unless it is clear that the studies were done with the same choices for percentile calculation and rank assignment.
---
paper_title: Disciplinary differences in Twitter scholarly communication

paper_content:

This paper investigates disciplinary differences in how researchers use the microblogging site Twitter. Tweets from selected researchers in ten disciplines (astrophysics, biochemistry, digital humanities, economics, history of science, cheminformatics, cognitive science, drug discovery, social network analysis, and sociology) were collected and analyzed both statistically and qualitatively. The researchers tended to share more links and retweet more than the average Twitter users in earlier research and there were clear disciplinary differences in how they used Twitter. Biochemists retweeted substantially more than researchers in the other disciplines. Researchers in digital humanities and cognitive science used Twitter more for conversations, while researchers in economics shared the most links. Finally, whilst researchers in biochemistry, astrophysics, cheminformatics and digital humanities seemed to use Twitter for scholarly communication, scientific use of Twitter in economics, sociology and history of science appeared to be marginal.
---
paper_title: Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature

paper_content:

Data collected by social media platforms have been introduced as new sources for indicators to help measure the impact of scholarly research in ways that are complementary to traditional citation analysis. Data generated from social media activities can be used to reflect broad types of impact. This article aims to provide systematic evidence about how often Twitter is used to disseminate information about journal articles in the biomedical sciences. The analysis is based on 1.4 million documents covered by both PubMed and Web of Science and published between 2010 and 2012. The number of tweets containing links to these documents was analyzed and compared to citations to evaluate the degree to which certain journals, disciplines, and specialties were represented on Twitter and how far tweets correlate with citation impact. With less than 10% of PubMed articles mentioned on Twitter, its uptake is low in general but differs between journals and specialties. Correlations between tweets and citations are low, implying that impact metrics based on tweets are different from those based on citations. A framework using the coverage of articles and the correlation between Twitter mentions and citations is proposed to facilitate the evaluation of novel social-media-based metrics.
---
paper_title: How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects

paper_content:

Evaluative bibliometrics is concerned with comparing research units by using statistical procedures. According to Williams (2012) an empirical study should be concerned with the substantive and practical significance of the findings as well as the sign and statistical significance of effects. In this study we will explain what adjusted predictions and marginal effects are and how useful they are for institutional evaluative bibliometrics. As an illustration, we will calculate a regression model using publications (and citation data) produced by four universities in German-speaking countries from 1980 to 2010. We will show how these predictions and effects can be estimated and plotted, and how this makes it far easier to get a practical feel for the substantive meaning of results in evaluative bibliometric studies. An added benefit of this approach is that it makes it far easier to explain results obtained via sophisticated statistical techniques to a broader and sometimes non-technical audience. We will focus particularly on Average Adjusted Predictions (AAPs), Average Marginal Effects (AMEs), Adjusted Predictions at Representative Values (APRVs) and Marginal Effects at Representative Values (MERVs).
---
paper_title: Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature

paper_content:

Data collected by social media platforms have been introduced as new sources for indicators to help measure the impact of scholarly research in ways that are complementary to traditional citation analysis. Data generated from social media activities can be used to reflect broad types of impact. This article aims to provide systematic evidence about how often Twitter is used to disseminate information about journal articles in the biomedical sciences. The analysis is based on 1.4 million documents covered by both PubMed and Web of Science and published between 2010 and 2012. The number of tweets containing links to these documents was analyzed and compared to citations to evaluate the degree to which certain journals, disciplines, and specialties were represented on Twitter and how far tweets correlate with citation impact. With less than 10% of PubMed articles mentioned on Twitter, its uptake is low in general but differs between journals and specialties. Correlations between tweets and citations are low, implying that impact metrics based on tweets are different from those based on citations. A framework using the coverage of articles and the correlation between Twitter mentions and citations is proposed to facilitate the evaluation of novel social-media-based metrics.
---
paper_title: Do Altmetrics Work? Twitter and Ten Other Social Web Services

paper_content:

Altmetric measurements derived from the social web are increasingly advocated and used as early indicators of article impact and usefulness. Nevertheless, there is a lack of systematic scientific evidence that altmetrics are valid proxies of either impact or utility although a few case studies have reported medium correlations between specific altmetrics and citation rates for individual journals or fields. To fill this gap, this study compares 11 altmetrics with Web of Science citations for 76 to 208,739 PubMed articles with at least one altmetric mention in each case and up to 1,891 journals per metric. It also introduces a simple sign test to overcome biases caused by different citation and usage windows. Statistically significant associations were found between higher metric scores and higher citations for articles with positive altmetric scores in all cases with sufficient evidence (Twitter, Facebook wall posts, research highlights, blogs, mainstream media and forums) except perhaps for Google+ posts. Evidence was insufficient for LinkedIn, Pinterest, question and answer sites, and Reddit, and no conclusions should be drawn about articles with zero altmetric scores or the strength of any correlation between altmetrics and citations. Nevertheless, comparisons between citations and metric values for articles published at different times, even within the same year, can remove or reverse this association and so publishers and scientometricians should consider the effect of time when using altmetrics to rank articles. Finally, the coverage of all the altmetrics except for Twitter seems to be low and so it is not clear if they are prevalent enough to be useful in practice.
---
paper_title: A multi-metric approach for research evaluation

paper_content:

Background information is provided about the Web 2.0 related term altmetrics. This term is placed in the context of the broader field of informetrics. The term influmetrics is proposed as a better term for altmetrics. The importance of considering research products and not just scientific publications is highlighted. Issues related to peer review and making funding decisions within a multi-metric approach are discussed and brought in relation with the new metrics field.
---
paper_title: Five challenges in altmetrics: A toolmaker's perspective

paper_content:

Jean Liu is data curator and blog editor at Altmetric LLP, and Euan Adie is the founder. They can be reached at http://altmetric.com. D riven by the development of new tools for measuring scholarly attention, altmetrics constitute a burgeoning new area of information science. It is an exciting time to be involved in the field since there are so many opportunities to contribute in innovative ways. We develop altmetrics tools and related services at Altmetric LLP, a small London-based start-up founded in 2011 [1]. Like all developers of new altmetrics tools, we frequently encounter challenges in defining what should be measured, accurately collecting attention from disparate sources and making sense of the huge amount of compiled data. We outline five of these challenges in this piece, illustrating them with examples from our experience. It is worth noting that the altmetrics community as a whole comes together regularly to discuss these and other issues, with two open workshops held in 2012 and more planned for the future.
---
paper_title: Evaluating altmetrics

paper_content:

The rise of the social web and its uptake by scholars has led to the creation of altmetrics, which are social web metrics for academic publications. These new metrics can, in theory, be used in an evaluative role, to give early estimates of the impact of publications or to give estimates of non-traditional types of impact. They can also be used as an information seeking aid: to help draw a digital library user's attention to papers that have attracted social web mentions. If altmetrics are to be trusted then they must be evaluated to see if the claims made about them are reasonable. Drawing upon previous citation analysis debates and web citation analysis research, this article discusses altmetric evaluation strategies, including correlation tests, content analyses, interviews and pragmatic analyses. It recommends that a range of methods are needed for altmetric evaluations, that the methods should focus on identifying the relative strengths of influences on altmetric creation, and that such evaluations should be prioritised in a logical order.
---
paper_title: A Look at Altmetrics and Its Growing Significance to Research Libraries

paper_content:

This document serves as an informational review of the emerging field and practices of alternative metrics or altmetrics. It is intended to be used by librarians and faculty members in research libraries and universities to better understand the trends and challenges associated with altmetrics in higher education. It is also intended to be used by research libraries to offer guidance on how to participate in shaping this emerging field.
---
paper_title: How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications

paper_content:

In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readerships for 62.6 % of all the publications studied, other sources only provide marginal information. In terms of relation with citations, a moderate spearman correlation (r = 0.49) has been found between Mendeley readership counts and citation indicators. Other possibilities and limitations of these indicators are discussed and future research lines are outlined.
---
paper_title: Assessing non-standard article impact using F1000 labels

paper_content:

Faculty of 1000 (F1000) is a post-publishing peer review web site where experts evaluate and rate biomedical publications. F1000 reviewers also assign labels to each paper from a standard list or article types. This research examines the relationship between article types, citation counts and F1000 article factors (FFa). For this purpose, a random sample of F1000 medical articles from the years 2007 and 2008 were studied. In seven out of the nine cases, there were no significant differences between the article types in terms of citation counts and FFa scores. Nevertheless, citation counts and FFa scores were significantly different for two article types: "New finding" and "Changes clinical practice": FFa scores value the appropriateness of medical research for clinical practice and "New finding" articles are more highly cited. It seems that highlighting key features of medical articles alongside ratings by Faculty members of F1000 could help to reveal the hidden value of some medical papers.
---
paper_title: Scientific bloggers under the altmetric microscope

paper_content:

In this paper we present a preliminary altmetric study of scientific bloggers and how they use different social media (i.e. blogs, social bookmarking systems, and Twitter) for scholarly communication, information dissemination, and creation of visibility. We analyzed linking behavior in blog posts and tweets, number of comments assigned to blog posts and share of publications found in social bookmarking systems. Results show that heavy tweeting and blogging do not result in large numbers of followers and comments, tweets and blog posts contain lots of URLs and self-citations, and share of publications found in social bookmarking systems varies between different platforms.
---
paper_title: Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact

paper_content:

Background: Citations in peer-reviewed articles and the impact factor are generally accepted measures of scientific impact. Web 2.0 tools such as Twitter, blogs or social bookmarking tools provide the possibility to construct innovative article-level or journal-level metrics to gauge impact and influence. However, the relationship of the these new metrics to traditional metrics such as citations is not known. Objective: (1) To explore the feasibility of measuring social impact of and public attention to scholarly articles by analyzing buzz in social media, (2) to explore the dynamics, content, and timing of tweets relative to the publication of a scholarly article, and (3) to explore whether these metrics are sensitive and specific enough to predict highly cited articles. Methods: Between July 2008 and November 2011, all tweets containing links to articles in the Journal of Medical Internet Research (JMIR) were mined. For a subset of 1573 tweets about 55 articles published between issues 3/2009 and 2/2010, different metrics of social media impact were calculated and compared against subsequent citation data from Scopus and Google Scholar 17 to 29 months later. A heuristic to predict the top-cited articles in each issue through tweet metrics was validated. Results: A total of 4208 tweets cited 286 distinct JMIR articles. The distribution of tweets over the first 30 days after article publication followed a power law (Zipf, Bradford, or Pareto distribution), with most tweets sent on the day when an article was published (1458/3318, 43.94% of all tweets in a 60-day period) or on the following day (528/3318, 15.9%), followed by a rapid decay. The Pearson correlations between tweetations and citations were moderate and statistically significant, with correlation coefficients ranging from .42 to .72 for the log-transformed Google Scholar citations, but were less clear for Scopus citations and rank correlations. A linear multivariate model with time and tweets as significant predictors (P < .001) could explain 27% of the variation of citations. Highly tweeted articles were 11 times more likely to be highly cited than less-tweeted articles (9/12 or 75% of highly tweeted article were highly cited, while only 3/43 or 7% of less-tweeted articles were highly cited; rate ratio 0.75/0.07 = 10.75, 95% confidence interval, 3.4–33.6). Top-cited articles can be predicted from top-tweeted articles with 93% specificity and 75% sensitivity. Conclusions: Tweets can predict highly cited articles within the first 3 days of article publication. Social media activity either increases citations or reflects the underlying qualities of the article that also predict citations, but the true use of these metrics is to measure the distinct concept of social impact. Social impact measures based on tweets are proposed to complement traditional citation metrics. The proposed twimpact factor may be a useful and timely metric to measure uptake of research findings and to filter research findings resonating with the public in real time. [J Med Internet Res 2011;13(4):e123]
---
paper_title: A multi-metric approach for research evaluation

paper_content:

Background information is provided about the Web 2.0 related term altmetrics. This term is placed in the context of the broader field of informetrics. The term influmetrics is proposed as a better term for altmetrics. The importance of considering research products and not just scientific publications is highlighted. Issues related to peer review and making funding decisions within a multi-metric approach are discussed and brought in relation with the new metrics field.
---
paper_title: Coverage and adoption of altmetrics sources in the bibliometric community

paper_content:

Altmetrics, indices based on social media platforms and tools, have recently emerged as alternative means of measuring scholarly impact. Such indices assume that scholars in fact populate online social environments, and interact with scholarly products in the social web. We tested this assumption by examining the use and coverage of social media environments amongst a sample of bibliometricians examining both their own use of online platforms and the use of their papers on social reference managers. As expected, coverage varied: 82 % of articles published by sampled bibliometricians were included in Mendeley libraries, while only 28 % were included in CiteULike. Mendeley bookmarking was moderately correlated (.45) with Scopus citation counts. We conducted a survey among the participants of the STI2012 participants. Over half of respondents asserted that social media tools were affecting their professional lives, although uptake of online tools varied widely. 68 % of those surveyed had LinkedIn accounts, while Academia.edu, Mendeley, and ResearchGate each claimed a fifth of respondents. Nearly half of those responding had Twitter accounts, which they used both personally and professionally. Surveyed bibliometricians had mixed opinions on altmetrics' potential; 72 % valued download counts, while a third saw potential in tracking articles' influence in blogs, Wikipedia, reference managers, and social media. Altogether, these findings suggest that some online tools are seeing substantial use by bibliometricians, and that they present a potentially valuable source of impact data.
---
paper_title: Altmetrics: New Indicators for Scientific Communication in Web 2.0

paper_content:

In this paper we review the socalled altmetrics or alternative metrics. This concept raises from the development of new indicators based on Web 2.0, for the evaluation of the research and academic activity. The basic assumption is that variables such as mentions in blogs, number of twits or of researchers bookmarking a research paper for instance, may be legitimate indicators for measuring the use and impact of scientific publications. In this sense, these indicators are currently the focus of the bibliometric community and are being discussed and debated. We describe the main platforms and indicators and we analyze as a sample the Spanish research output in Communication Studies. Comparing traditional indicators such as citations with these new indicators. The results show that the most cited papers are also the ones with a highest impact according to the altmetrics. We conclude pointing out the main shortcomings these metrics present and the role they may play when measuring the research impact through 2.0 platforms.
---
paper_title: Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature

paper_content:

Data collected by social media platforms have been introduced as new sources for indicators to help measure the impact of scholarly research in ways that are complementary to traditional citation analysis. Data generated from social media activities can be used to reflect broad types of impact. This article aims to provide systematic evidence about how often Twitter is used to disseminate information about journal articles in the biomedical sciences. The analysis is based on 1.4 million documents covered by both PubMed and Web of Science and published between 2010 and 2012. The number of tweets containing links to these documents was analyzed and compared to citations to evaluate the degree to which certain journals, disciplines, and specialties were represented on Twitter and how far tweets correlate with citation impact. With less than 10% of PubMed articles mentioned on Twitter, its uptake is low in general but differs between journals and specialties. Correlations between tweets and citations are low, implying that impact metrics based on tweets are different from those based on citations. A framework using the coverage of articles and the correlation between Twitter mentions and citations is proposed to facilitate the evaluation of novel social-media-based metrics.
---

Title: Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

Section 1: Introduction
Description 1: Provide background on the shift in science policy towards demonstrating the societal value of research and the need for new metrics like altmetrics to measure this impact.

Section 2: What are altmetrics?
Description 2: Define altmetrics and discuss their significance and how they differ from traditional metrics like bibliometrics.

Section 3: How can altmetrics be classified?
Description 3: Describe the various classifications of altmetrics and the types of social media tools and platforms used for these metrics.

Section 4: What benefits do altmetrics offer?
Description 4: Outline the advantages of altmetrics, including broadness, diversity, speed, and openness, with specific examples and explanations.

Section 5: What are the disadvantages of altmetrics?
Description 5: Discuss the limitations and challenges of using altmetrics, such as issues with data quality, commercialisation, and potential for manipulation.

Section 6: Discussion
Description 6: Summarize the varying viewpoints on the significance of altmetrics for research evaluation and highlight areas for future research and development in the field.