Paper publication rates

The publication rate of an astronomer, a project, or an institution is the simplest statistic commonly used to gauge impact or productivity. It is based upon the number of research articles (N) divided by the number of participants, P, and the span of time over which the research effort was performed, T, and represented by Rp = N/(PT) in units of papers/year/astronomer. To compare Rp between different time periods, normalizing the rate by the number of authors partially compensates for historical trends in the growth of authorship lists, or in the number of professional astronomers employed to conduct research.

According to a study by Abt (1998), the growth in the number of citable papers in the popular journals Astronomy and Astrophysics (A&A), The Astronomical Journal (AJ), ApJ, PASP, and MNRAS from 1960–1996 was directly proportional to the number of professional astronomers who are members of the American Astronomical Society (AAS), with an average of 0.4 papers/year/astronomer.

The ADS publication database was used to identify a total of 9,000 papers published in 2017 in the journals surveyed by Abt (1981, 1998). The total AAS membership included about 7,000 astronomers for an average rate of 1.3 papers/year/astronomer. This estimate is actually a lower limit, because during any given year, only a fraction of AAS members publish research. Nevertheless, this three-fold increase in publication rate between 1961 and 2017, despite only a 50% increase in AAS membership, speaks to a dramatic and non-linear change in the “productivity” of modern-day astronomers no longer proportional to AAS membership levels. This suggests that comparing publication rates across many decades will be difficult, because factors other than population size seem to be at play in deciding how a specific paper will fare using publication-based metrics alone.

For the CS papers in Table 2, the publication rate, which was previously defined as Rp, can be calculated for each project (col. 10) by dividing the number (N) of refereed papers (col. 7) by the median number, P, of authors (col. 8) and then dividing by the duration, T, of the project (col. 4). The resulting average publication rate across all 19 projects with refereed papers is 0.10 papers/year/astronomer. This aggregate average for CS projects is more than ten-fold lower than the nominal rate calculated for 2017 for non-CS research in astronomy. One reason may be that CS papers have a larger-than-average number of co-authors involved in the paper publication. In a study of general trends in astronomical publications, Hennekin (2012) noted that in 1960, 80% of papers involved 1–2 authors, while only 15% did so by 2009, with a current median level near 4 authors/paper. The median number of authors involved in all of the CS projects in Table 2 is 12, which is significantly higher than typical non-CS publications in astronomy, perhaps providing some of the reason for a lower CS publication rate per author. CS publications appear to attract more co-author participation than typical non-CS papers. The aggregate value of 0.10 papers/year/CS author may actually be a factor of three or four times higher when corrected for general co-authorship increases, but is still significantly lower than the previously discussed estimate for the 2017 rate of 1.3 papers/year/astronomer. Taken by itself, this number suggests that CS papers are being produced by projects at a significantly lower rate than the average modern astronomical papers. This result may also be related to the non-publication effect found by Clery (2018), in which scientific data from a research program, once available, are more difficult to analyze than expected and lead to few or no publications. Beyond simple paper-counting methods, there are other ways to explore how well CS papers and research fares in contributing to the general professional research conversation. The next section discusses how frequently CS research is subsequently cited by the scientific community.

Citation statistics

The 4,515 citations produced by the 143 refereed papers in this study can be summarized in a variety of second-order metrics. A simple estimate of citation output from CS projects can be obtained by dividing the total citations by the total papers, yielding 31 citations/paper. This result does not offer a fair assessment of what to expect from the average paper, however, because averages can be dominated by a small number of highly successful papers. As shown in Table 3, dividing the number of citations (col. 6) by the number of papers published (col. 4) yields the average number of citations per paper for each project (col. 7). Several projects generated the dominant share of the citations, while six projects generated no citations at all by the end of 2017. A better aggregate measure than the average citation rate is the median citation rate, which for this ensemble is 6 citations/paper.

A second measure also normalizes the number of publications to the number of authors and project duration. The idea is to compare project output in a manner that compensates for the fact that some projects have more citations and papers simply because they have been in operation for a long time. It is also advantageous in performing citation studies to “level the playing field” so that a paper with a large author list does not yield a higher impact for that reason alone. This can be accomplished by dividing column 4 by the product of column 3 and column 5, resulting in the “Paper Rates” shown in column 8. For example, the SunGrazer project resulted in 11 refereed publications written by a median of two authors during its 22-year project duration. The normalized paper rate, Rp, for this project is then Rp = 11/(22 * 2) = 0.25 paper/year/author, which appears in column 8. This results in a median annual paper rate for the CS projects of Rp = 0.05 papers/year/author. Similarly, the normalized citation rate is defined by dividing the total citations (col. 6) by the product of the number of published papers (col. 4), the median number of authors (col. 5), and the project duration (col. 3). For example, the SunGrazer project generated 274 citations from 11 papers published by an average of 2 authors over the course of 22 years for Rc = 274/(22 × 11 × 2) = 0.57 citations/paper/year/author, shown in column 9. The table also shows the median values for the 19 projects with refereed papers.

The Abt (1981) study identified 326 papers published in 1961 that generated 6,070 citations after 17 years. To compare this with a modern re-calculation, the ADS was used to identify 2,479 papers published in the same journals as studied by Abt (1981), which were followed until the end of 2017 to determine the citation statics using WoS. These modern papers yielded 26,020 citations, for an average rate of 10 citations/paper and 0.6 citations/paper/year. By comparison, the 143 CS papers published over a variety of years since the start of the earliest CS project (SunGrazer) in 1996 (summarized in Table 3) generated 4,515 citations for a median aggregate rate of 10 citations/paper, and 0.09 citations/paper/year. The CS papers thus appear to be cited at a comparable median rate to the Year-2000 papers, but the annual rate at which they are cited is dramatically lower. If the aggregate citations/paper is an indicator of how well CS papers and their results are received (e.g., impact) by the scientific community, does the low annual rate indicate that CS papers do not stimulate intense interest to cite them? Not necessarily.

Making a direct comparison between the Abt (1981), Year-2000, and CS papers is difficult, because the papers in the CS study were not all published in the same calendar year but over a staggered publication timeline beginning in 1996 and ending in 2017. Hypothetically, for a 7-year project that started in 2010, ended in 2017, and published two papers in 2011 and 2016, the citations for the second paper will be reduced by the same project duration factor of 7 years as the citations for the earlier paper. This unfairly penalizes the more recent project papers and reduces the overall project annualized citation rate. To explore this aspect of the citation comparison in more detail, instead of looking at the aggregated citation rate, this study examines the actual history of paper citations over the project timeline, which henceforth will be called the Project Citation History.

Project citation history

In typical citation studies such as the one undertaken by Abt (1981), all papers published during a single year (e.g., 1961) are followed forward in time. A similar study can be performed for the 143 CS papers by shifting the citation histories for each paper to a uniform publication Year Zero for each project. Figure 1 shows these citation histories for the first ten years of operation, with the citations summed across all papers in each project. The resulting cumulative project citation histories are then plotted on a common annual citation scale.

Although most of the project citation profiles reach their peak within two to three years after the publication of the primary paper, one project, SunGrazer (Plot symbol: A), has a significantly different citation history with a much longer tail to its distribution extending beyond the plot to a maximum of 18 years. None of these distributions are similar to the aggregate of the papers studied by Abt (1981), which reached a peak after five years followed by a slow decline extending to twenty years. This suggests that CS papers generate considerable interest much sooner than conventional papers, however, they also seem to decline in citation popularity at a faster rate.

This effect can be seen more clearly in Figure 2, where the CS citation history for all projects is compared to the Abt (1981) and the Year-2000 publication cohorts. The number of citations each year has been normalized by the total number of papers in each group to determine the annual citations per paper. What is immediately obvious is that CS papers in the aggregate have a higher annual citation rate than conventional papers published in 1961 or 2000, implying that the historical context of these papers is less of an issue than, apparently, their origin in traditional versus CS research activities. Citizen science research results, when appearing in refereed publications, do indeed seem to burn more brightly and fade more quickly in interest compared to traditional research results. Also interesting is that after citation Year 10, the modern paper rates are half the 1961 rates, and are also at least double the CS paper citation rates. Evidently, modern papers represented by the Year-2000 cohort experience systematically fewer long-term citations than older papers, but consistently exceed the citation rates of the roughly contemporaneous CS papers. The time at which the CS papers fall below both the older and the modern papers is roughly 8–10 years after publication. This is probably a useful marker for CS project developers, which defines a maximum interest lifespan for CS papers relative to non-CS papers.

The previous discussion described how CS citations follow the normal publication year profiles of research papers. An interesting subsidiary question is how citations follow the project year. This would be similar in spirit to typical paper citation studies that begin in the year of paper publication such as the Abt (1981) study beginning in 1961, or the previously mentioned modern study for all papers published in the year 2000. For example, a project starting in 2005 has a paper published in 2008 that produces 10 citations in 2010. The 10 citations are counted in the tally for Project Year 4, and this calculation is re-performed for all papers in the project to create the Project Citation History. Taken as an ensemble, the annual citation counts can be aggregated for all 143 papers by representing the citations in terms of the Project Year. This effectively treats a CS project as a single meta-paper published in the project’s inception year and followed to 2017. The individual Project Citation Histories are shown in Figure 3. The citations for Galaxy Zoo have been plotted separately. The sharp drop off in citations for Galaxy Zoo in Years 10 and 11 corresponds to citations recorded in the calendar years 2017 and 2018, for which the later year is incomplete. The smaller drop between Year 9 and 10 is, however, significant because the citation tallies for the corresponding years 2016 and 2017 are believed to be complete, suggesting that this project may have reached its peak citation year around 2016, Project Year 9.

Apparently for the non-Galaxy Zoo papers, citations can be expected to increase to a maximum near Project Year 5, and then decline almost monotonically thereafter. This profile is similar to the aggregate of the papers studied by Abt (1981), which also reached a peak after five years followed by a slow decline. This indicates that the aggregate citation output for individual CS projects as a function of their project year strongly resembles the citation profile for average non-CS research papers.

Alternative metrics

In 1994, the Institute of Physics (IOP) became the first publisher to publish a journal on the World Wide Web. The collection of IOP-accessible journals now spans 75 titles. Because the number of “views” and “downloads” to articles in the online IOP journals could be tallied easily each month, indices for “impact” were soon developed as an adjunct to the more traditional paper citation studies. The expansion of these Internet and social media-based indicators of article popularity and impact, collectively called “altmetrics,” has led to the creation of a number of real-time services such as Altmetric.com (2011), which offers an on-the-spot index of an article’s popularity via an embedded app. Altmetric (Altmetric.com2018) gives the WoS index, but also aggregates all online and non-technical references including “tweets” and downloads of articles, mentions in blogs, and mentions in the news media in its Altmetric Attention Score. In this score, mentions in the news media and blogs received most of the weighting compared to mentions in Twitter, Facebook, or YouTube (Huang, Wang, and Wu 2018).

The previous discussions examined, in detail, how the citation data for WoS used in the current CS study compares with the other citation index services: iopscience.com, the ADS Database, ScienceDirect.com, Institute of Physics (IOPscience), and Altmetric.com. The results for an example CS project, Planet Hunters, are shown in Table 4. For uniformity, only citations appearing through the end of 2017 are included. Not included were citations to arXiv, BAAS, and RNAAS. A dashed line indicates that the journal was not covered by the citation service. Column 1 is the paper publication year. Columns 2–5 are the total number of citations for each paper from the year of publication to 2017.

The citation statistics for the WoS, ADS, and iopScience follow a nearly linear correlation as expected, because these services generally refer to the same body of research articles and employ the same bibliographic citation measures. However, the Altmetric scores are not obviously related to the bibliographic citation statistics. In one case a paper, 2013a, generated between 73 to 96 citations and an AAS of 257, while 2016a generated between 33 to 37 citations and a dramatically higher AAS of 2,139. Similar disparities can be found among the other papers considered, suggesting that it is not a simple matter (linear transformation) to relate the Altmetric.comAAS score to a paper’s placement in the research journals, which in the past (e.g., Abt 1981) has been considered a good gauge of scientific impact.

Along with the growing popularity of altmetrics for assessing article popularity has come intense scrutiny of exactly how to interpret these new metrics as compared to older forms of measure such as paper citations. According to Bastow, Dunleavy, and Tinkler (2014), canonical bibliographic measures such as citations are best interpreted as a measure of the value of a published research article to scientists, while altmetrics are primarily a measure of a research article’s impact and benefit to society and social discourse. Bibliometricians such as Bornmann (2014) are increasingly seeing altmetrics as a supplement to citation counting. The relationship between the newer and older citation indices remains murky, largely because of the very different databases being sampled to form the index. Citations are generally targeted at professional research activity, while altmetrics include non-scientists, students, teachers, and many other groups. Also, while “gaming” citation counts is difficult except through self-citation, counts for altmetrics can be bought or even generated by robots. Also, the number of journals that can provide citations remains relatively fixed over time, while the number of websites, social media platforms, and data providers changes almost annually for altmetrics tabulations. A sudden change in index value may be interpreted as a change in social interest; alternatively, it may show that a provider entered or left the data pipeline. Correlations between altmetrics scores and citations have been found by Evans and Krauthammer (2011) for Wikipedia and by Thelwall et al. (2013) for Twitter. However, the degree of correlation between WoS citation rates and AAS varies significantly from journal to journal and, as discussed by Huang, Wang, and Wu (2018), in most cases no clear relationship exists between them, which was also found for the CS papers.

Citation biases

Although citation studies are becoming widely used to evaluate author productivity in both research and academic settings, using them appears to present a variety of biases. For instance, Letchford et al. (2015) found that brief titles for research papers garner more citations than longer titles. King et al. (2017) found that men cite their own research more often than women researchers. According to Fowler and Aksnes (2007), self-citation accounts for about 10% of a paper’s citation history. Even the number of references in a paper’s bibliography has some impact on citations, according to Vieira and Gomes (2017).

Among the other biases examined in this study was whether the number of papers produced by a project, or the number of ensuing citations, was in any way related to the number of co-authors in the typical paper. Such correlations have been found by, for example, Vieira and Gomes (2009). The CS papers offer an enormous range, by a factor of nearly 200, in the number of co-authors listed on each paper. For example, one project (Einstein@Home) produced several papers with more than 500 co-authors (e.g., the LIGO Consortium), while many of the smaller CS projects involve fewer than four co-authors. The median number of authors for each project is presented in Table 2 column 7. Comparing the number of authors in column 7 with the number of refereed papers in column 6 shows no clear correlation, suggesting that only a small number of authors may be responsible for the bulk of the publications from the groups affiliated with each CS project. In terms of citations, one might presuppose that projects with large teams of authors have greater opportunities for the CS research to be cited in subsequent team publications. However, this expected trend is not borne out by the citation data. For example, one project had a median of twenty authors per paper and generated 324 citations, while a second project had a median of only one author per paper yet generated a comparable 239 citations. This large variation in authors-per-paper suggests that the analysis in the section on paper publication rates, above, normalized by the median number of authors leading to a dramatically lower median estimate for CS citations/year/paper/astronomer (e.g., 0.09 c/y/p/a), is not a reliable indicator of a project’s productivity compared to other non-CS surveys.

Self-Citations

An important issue to consider when dealing with papers in which a small number of citations are involved is “self-citation.” Researchers commonly cite their own relevant research when writing new papers, and in the novel field of citizen science, self-citation of previous CS publications would be expected. Moreover, as shown by Aksnes (2003), the larger the author list on a given paper, the more often self-citation can be expected to occur. According to Shema (2012), some journals even encourage authors to cite other articles in the same journal, presumably to increase that journal’s impact score. According to Fowler and Aksnes (2007), within the general population of non-CS research papers, the self-citation rate is about 11%, and the incentive for self-citing at rates as high as 20% is free from penalty. For example, if the author is being evaluated for tenure, self-citations are not counted against the author as superfluous. Also, not all self-citations are made merely to “pad” a bibliography. If an author is opening up a new field or scientific line of inquiry, self-citations are necessary in a positive way. Among the 4,515 citations in this study of 143 CS papers, about 840 were self-citations, just over an aggregate average of 18%. These included citations to papers written by other members of the project team, or to previously published findings from the project. The distribution of self-cited papers with respect to the total citations for all 143 papers is shown in Figure 4.

The self-citation rates can be binned into three groups based on the total paper citations shown in Figure 4. Group A, with between 1 and 10 citations, displayed a wide range of self-citation rates between 0% to 100%. Group B, with 11 to 100 citations, had an average of 20% ± 10%. Group C with >100 citations had an average self-citation rate of 10% ± 10%. This suggests that self-citation among the refereed journal articles for CS is a large issue, especially among papers that yield only small numbers of citations during their publication lifetimes. In these cases, the papers are being cited only by other CS papers, often within their own project, rather than by a larger base of non-CS investigators. However, because the median number of self-citations per paper is 4, papers in Groups B and C that garner more than 40 citations achieve a balance that is similar to the 11% self-citation rates seen in the general population of papers as identified by Fowler and Aksnes (2007). This suggests a citation threshold at approximately Group B, in which CS papers are being more widely cited by other research communities than by the CS project members.

Heavily-cited papers

What can be said about the proportion of projects that generate the largest numbers of citations in the literature? Based on the tallies in Table 2 of 19 projects with refereed publications, the median citations per published CS research paper is 10 citations/year. The top four projects with papers that cumulatively exceeded 200 citations (upper 21% of projects) account for 3,933 citations, while projects in the lower 37% account for 37 citations. All other things being equal, the odds that any given CS project will exceed 200 citations of its research in its roughly 6-year average lifetime are about one in five.

Among the 58 million papers indexed by the WoS, only 14,500 papers (one-in-3,800) achieved more than 1,000 citations by 2014 (van Noorden, Maher, and Nuzzo 2014). Space science and astronomy topics are a small subset that is drowned out by the far more numerous papers in physics, chemistry, and biology. For space science papers in particular, Pearce (2004) examined citation impact statistics for the 1,000 most-cited papers out of the 439,000 papers published previous to November 2003 that had at least 257 citations by that time. After five years, one paper in 1,000 had reached 253 cumulative citations. For the current CS study, 77 CS papers were found that had been published for five years or longer to match the Pearce (2004) study baseline. These CS papers generated 3,884 citations (87% of the total), with one paper out of 26 having more than 253 citations after 5 years. Papers presenting CS research are significantly more likely than traditional research papers to reach the ranks of the Top-1000 papers in space science and astronomy.