Identifying responses

Because data extraction resulted in an enormous quantity of posts, to identify potentially relevant discussions we used a concordance program (Imao 2016) to search for posts where specific terms could be found in close adjacency to each other (within a range of five words to the left or right). Based on our research question, the following search term pairs were performed: “rule*” AND “game*”; “science” AND “game*”; “score*” AND “rank*”; “game*” AND “rank*”; “play*” AND “science”; “game*” AND “not”; “classif*” AND “competition.” The choice of search terms was driven primarily by our research question, and only secondarily by the frequency of such words in the forum posts. A word frequency analysis using a stop word list to filter out the most common English language function words run to elicit the vocabulary and phrasing used by the Foldit forum participants generated a skewed distribution of content word frequencies. Most frequent terms in the two Foldit forums included “Foldit” (0.99%), “puzzle” (0.55%), “fold” (0.54%), “game” (0.48%) and “score” (0.45%), while the proportion of the terms “science” and “rank*” was 0.15% and 0.07% respectively. Applying the same process to the Galaxy Zoo corpus produced a list with most frequent terms including “galaxy” (1.51%), “zoo” (0.96%), “like” (0.69%), “interesting” (0.66%), and “star” (0.51%). Amongst the search terms, the only terms to appear in the GZ corpus with a frequency of more the 0.01% were “classif*” at 0.37%, “science” at 0.12%, and “play” at 0.01%. As can be expected, the most frequent terms on each forum represent the particular activities of each citizen science project with little overlap, although interestingly, the term “science” appears on the forums with roughly the same frequency.

After a rapid reading of the threads containing instances of adjacency of search term pairs, we chose to focus on the threads retrieved by using the search terms “science” AND “game*”; and “play*” AND “science.” This resulted in the identification of 65 Foldit threads and 11 GZ threads. For GZ, despite a far larger quantity of forum posts in the corpus than Foldit, far fewer posts could be found with the chosen adjacency pairs. This can most likely be attributed to the nature of Foldit, which has been explicitly created as a game, versus GZ, which has not. Examining the identified threads, we then conducted close readings using a strategy aimed at identifying a broad range of opinions and experiences of players about the relationship between science and games through posts that describe experiences and stories reported by participants over time. This search for cases continued until informational redundancy or theoretical saturation was reached. In the case of Foldit, this selection process resulted in 15 threads for further analysis (Table 1).

In the case of GZ, five threads of particular relevance were selected based on the adjacency pair analysis (Table 2).

In narrowing down the number of threads, we did not attempt to obtain a representative sample of participants in the two projects, as we did not collect either direct or indirect demographic information about participants.

Data analysis

We developed a coding frame ourselves using a conventional approach, which is considered appropriate when existing theory or research literature on a phenomenon is still limited (Hsieh and Shannon 2005). We used the actual data to derive codes and categories and to create string links between themes and the data they emerge from. The benefit of this approach is to generate findings grounded in the actual data without imposing preconceived categories or theoretical perspectives. We coded the selected Foldit and GZ forum threads using thematic analysis, according to Braun and Clark’s (2006) guidelines with the assistance of a qualitative data analysis software package, NVivo for Mac. Coding involved two individual researchers who coded the threads identified for each project forum independently. During the coding phase, the two coders exchanged and discussed ideas on their respective datasets to achieve a mutual understanding of the coding frame. To identify initial themes, each individual coder first performed an open coding of their dataset to identify key points, positions, and opinions expressed by participants in relation to the use of games and playful, game-like activity.

With respect to coding and sampling the content used in the analysis, we selected the posts that indicated a clearer topicalization of the relationship between gaming and science.

Thematic analysis of the selected themes resulted in codes that were organized into eight organizing themes and two overarching themes (Table 3).

The different codes used in the analysis of Foldit and GZ forum posts, which were grouped into the eight organizing themes, are presented in the Appendix (Tables 5 and 7 respectively). The percentages of the organizing themes in the forum posts in both projects are presented in the Appendix in Tables 6 and 8. The categories represented by these themes are not mutually exclusive, for example, it is possible for one to acknowledge the importance of sharing to ensure fairness while at the same time being concerned with losing competitive positions. Therefore, rather than a polarization of positions, the themes include a range of different positions.

Community and lack of openness

These two closely related organizing themes include opinions and experiences about two main aspects. Several posts in GZ expressed the belief that knowledge and tools useful to improving the ways that volunteers perform tasks should be shared with all to support a healthy community. For example, one volunteer expressed a common concern that gaming features related to competition such as rankings would hurt the community stating:

“The whole merit idea worries me from both a management and implication perspective. We are here out of common interest, to enjoy and discuss and learn from what we see. And enjoy the company of a great community in the process. So why complicate matters and try to make it something that it is not?”

By contrast, in the case of Foldit, this position was often motivated by a perceived need to avoid the system becoming the exclusive playground of an ever-decreasing elite group of players with access to knowledge of the best practices and resources available. Posts referred mainly to sharing recipes, small scripts of computer code that automate some protein folding processes, with all players. Many recipes are shared publicly, but certain groups have recipes that can be used only by members, and individuals can have recipes that they keep to themselves. Players can use recipes to carry out their strategies more easily when solving puzzles, increase their score and rank up. Some posts blamed group-shared recipes for hampering the progress of the community and turning cooperative groups into competitive entities. Recipes closely guarded by groups were reported to create unfairness, which, according to one player who had chosen to stop playing altogether, was “perhaps the worst aspect of the game today.” As one player stated eloquently, “… playing a non-profit game helping science for the benefit of ALL people but using recipes available for only some people playing looks ironic to me.”

This concern seems to be connected to the perceived closed nature of the game, as pointed out by a player who lamented the apparent secretiveness of the Foldit project itself, where “wins” are not openly available and a lack of after-game analysis prevents examination of a win and what was done to make it happen. “Neither the winning results or the method of working is available, players don’t get to see and interact with the top solutions or get a synopsis of process.”

Another aspect of concern in Foldit was the possibility of reusing existing puzzle solutions. This possibility elicited discordant views from players. In contrast to the argument that reusing shared solutions at the beginning of a puzzle could help work out new better solutions more efficiently, if someone else’s work is an entirely better direction than one’s own, concerns were expressed about players loading shared solutions and using them without substantial improvement, or incorporating them as if they were their own without properly acknowledging the author(s) of the solution. “… I think that attribution should be given to the source of the shared protein because not giving such attribution is plagiarism.”

Reliability of scoring/ranking systems and exploring solutions

Posters on the forums of both projects held discordant views on whether scoring or ranking systems were or could be designed to encourage increased quality in the work of volunteers. In relation to GZ where rankings systems had been briefly tried early in the project lifespan, suggestions from volunteers for adding a leaderboard or similar feature were quickly dismissed by both team members and other volunteers. These responses often included the arguments that ranking systems have a negative effect on the scientific validity of volunteer work and that features like leaderboards are antithetical to the nature of a scientific community. This sentiment on the GZ forums, that rankings reduce scientific validity and are antithetical to the spirit of science, stands in stark contrast to the concerns expressed around scoring on Foldit.

Discourse related to scoring on the Foldit forums took place within a clearly articulated gaming frame, where the notion that ranking performance could be antithetical to the spirit of science is not part of the established repertoires. Instead, without questioning the presence of scoring itself, the details of how scoring was implemented was the focus of much discussion. For instance, some posters thought that only top scoring solutions were exciting to work on and relevant from a scientific point of view, while working on what they called “suboptimal, boring, and scientifically irrelevant solutions” could get players to reach positions in the rankings but “no science, no big thrill either.” This comment suggests that the value of low-scoring solutions was not always clear to players, a thought also pointed out by another poster who expressed the desire to hear more from the Foldit scientists about this aspect and whether people should bother to work hard on a solution if it cannot be top scoring. The potential value of low-scoring solutions was recalled by a poster who mentioned one of such solutions leading to Foldit’s greatest success, the crystal structure of the AIDS-causing Mason-Pfizer monkey virus, a scientific problem that the Foldit players solved in 10 days after having being unsolved for 15 years. This player thus underscored that the scoring system was far from perfect, and the global top scores were not the best gauge of value to science. This claim questioned the reliability of the scoring system (which is a complex feature in Foldit) to explore a large number of design solutions potentially leading to innovative and scientifically sound solutions. The “intentionality” of the scoring system can be unclear to the players as this comment expresses in strong terms, “WHO’S DESIGNS ARE BETTER FOR THE SCIENTIFIC RESEARCH? and what if we stumbled upon an incredible breakthrough but didn’t have it recognized for the effort put into it?” Understanding the relationship between high score and good solutions is crucial but also problematic for the Foldit players. Such a relationship is not always direct. For example, a player reported that Foldit has been plagued by false positives (a false positive is when a solution scores well, but actually does not match the native protein well).

Enjoyment/fulfillment

This category emerged from a range of posts on both the GZ and Foldit forums. In the case of GZ, posts that address issues of enjoyment/fulfillment in relation to gaming were generally of a hypothetical nature, while in the case of Foldit, they tended to be of a concrete nature tied to specific gaming features of the system. GZ volunteers rarely expressed sentiments indicating that they felt that adding gaming elements would increase their enjoyment/fulfillment. Perhaps somewhat counterintuitively, a commonly expressed notion was that GZ was intrinsically a fun activity, and that gaming features would ruin that fun. Typical volunteer statements on the issue followed a pattern where gaming features were depicted as something ugly, and moderators and scientific team members often made statements that adding gaming features would ruin the fun. When presented with suggestions for the addition of contests or gaming features, one moderator noted:

“Thanks for the thought, but we always like people just to come for the fun and do as much or as little as they can contribute. And we have about 150,000 people clicking away at the moment.”

Here again, “fun” is set in opposition to competition and by extension gaming, revealing an established repertoire within the GZ forum community where gaming features were commonly referred to as reducing enjoyment/fulfilment instead of increasing it. This community norm contrasts directly with the norms of the Foldit community, where gaming features are commonly considered to be a key aspect of Foldit as an activity for achieving enjoyment/fulfilment.

The relationship between gaming features and enjoyment/fulfilment or “fun” for the Foldit forum community is most clearly articulated in two threads that discussed the roll-out of a new game client in 2013. This release led to frustration with the software and less enjoyment with the gameplay even among some dedicated and committed players who posted about their dissatisfaction in the forum. Playing for science and having fun were not thought contradictory by participants who volunteer to play the game and do not have their career at stake in the game. As a poster noted, if Foldit no longer had enjoyable elements, then they ceased to be “players” in one respect or the other.

Competition and sharing

Concerns were expressed by both Foldit and GZ volunteers about the effects of competition. In the case of GZ, a few volunteers posted calls for the introduction of features that would support competition. During some periods, volunteers even engineered their own ranking systems formed from self-reported metrics based on repurposing of participation data displayed to individual volunteers while logged-in to their accounts. During a discussion related to such a volunteer-created ranking system, one participant proposed the creation of an elaborate system tied to the quality of galaxy classifications made by volunteers. This suggestion was quickly met with a number of negative replies, even from volunteers engaged in creating their own rankings who noted that they were only posting numbers of classifications for fun, and that all participant contributions were equally important:

“In this project we’re all equal even if there are participant that go from astronomers to nothing (like me). It’s not a game. I believe that statistic is important but we are using them principally for joking here.

In this way, without the explicit framing of a game, statements from GZ volunteers most often contained negative perceptions of competition-related features such as points and rankings.

Similarly, concerns were expressed by Foldit players with regard to issues of competition and sharing. It seems that having to provide high-scoring solutions and rank up as the focus of playing the game constrained the willingness to share ideas and resources. While some posters expressed the belief that sharing is valuable because it levels the playing field and is integral to science, on the other side, they were also afraid of helping “competitors,” losing advantages, and going down in both the individual and group ranks. Players cooperate and share within their respective groups but can have closely guarded recipes and techniques shared only within groups. Therefore, even when sharing was considered valuable, players decided to share their less critical resources. What to share can become a dilemma as this comment indicates eloquently:

“On the one hand, the whole purpose of Foldit being for the good of the scientific community (…) On the other hand - I find myself rather jealously guarded and protective of my very best techniques and discoveries, while sharing mainly only what I might call my “second-tier” insights, techniques and scripts…. On further reflection, however, I know that the people I respect and admire the most in this game are not always those at the tops of the scoring heaps… rather, they are those few who excel not only in scoring, but in willingness to teach, and to share, and to assist the less accomplished members of their team, and to foster newbies’ growth.”

Thus far we have noted that claims about the necessity of sharing are concerned with creating a two-tiered system where a few players have access to resources unavailable to others. Along the same lines goes the concern for creating barriers to entry making it difficult for new players to learn the tricks of the trade, while “protecting the old guard” and restricting competition in the game. Reluctance of some experienced players to teach newbies more than just the basics of the game was linked to the control exerted by teammates, who would “watch” what their group members share with non-members.

Purpose of the project and the relationship between gaming and science

As the findings related to the previous categories show, there was a significant difference in the ways that Foldit and GZ volunteers expressed their sentiments toward gaming in their respective citizen science projects. In both cases, however, the core of the established community norms can be understood as the relationship between gaming features and volunteer understandings of the nature of citizen science as an activity. For the GZ forum community, a clearly articulated norm was that gaming is not compatible with scientific values as expressed by members. Statements by GZ volunteers, especially in relation to the relationship of game-like features and the health of the community, often included arguments for aspiring to a scientific ideal expressed as an activity intrinsically based on openness and sharing where all contributions are valued. For example, one volunteer responded to a post about the relative performance of different community members writing:

“I believe that the zoo is a special place. It’s not important if you’re an expert, a scientist, an astronomer, nothing or whatsoever. Or if you classified 10 or 100000 galaxies. We are a team. Everyone is important and no one is predominant. It’s a sort of one for all and all for one. We’re part of a great scientific project not a game.”

Participants commonly described each other as being of equal value to the project with scientific contribution and personal learning as their primary goals. The notion that gaming was incompatible with the nature of GZ as a citizen science project was also regularly expressed by moderators and scientific team members with statements during discussions of rankings or other game related features such as:

“this is a voluntary science project, not a game or competition. People contribute and learn as much as they want or can. So it really wouldn’t help us meet the objectives of the project.”

This sentiment of gaming as antithetical to citizen science and more generally scientific practice was widely expressed by volunteers, moderators, and scientific team members. It highlights an established community norm that was relatively infrequently challenged. By contrast, Foldit forum community members expressed a wide variety of different opinions on the game and framed its purpose accordingly.

While posts analyzed from the Foldit forum generally did not display uncertainty about the link between science and the game, uncertainty was sometimes expressed in the form of a question starting a discussion about the quality of the scientific contribution of the game, such as in the thread on whether Foldit was about “real” science or just another video game. The lack of after-game analysis with the Foldit team to examine solutions and their contribution to science led a player to wonder whether the game had something to do with solving scientific problems. Another player echoed this view, claiming that, without proper context and feedback, what they were doing was simply playing a video game for the highest scores, regardless of whether what they were producing made sense. As responses to these comments indicating frustration of not making a real contribution to science, some other players claimed that their role was not that of “amateur scientists hell bent on finding a big solution to some mysterious problem,” but rather that of assisting the research of a “real” scientist to produce large amounts of results.

Alongside the framing of players as contributing without being on par with scientists, the characterization of Foldit as primarily a game was similarly clear in some posts. For example, one player emphasized the recreational character of the project as a fun game, although many participants found enticing the chance of achieving a real solution, or getting lucky and making a breakthrough.

Overview of the Results

It is possible to draw a demarcation line in terms of values between Galaxy Zoo and Foldit, as indicated in Table 4. On the one hand, GZ participants report an imperative of sharing, equality, and openness, and prioritize contributing to science over other goals. In this community, participants are concerned with corrupting science if tasks are turned into competitive games, as this is antithetical to the spirit of openness that ought to be part of science. By contrast, Foldit participants report that competition involves a limited sharing of data and methods, and solutions are often developed in closed rooms to reach a competitive advantage over other players. Foldit participants often express their motivation being based on reaching a high score and that fulfilment lies in the competitive structure of the game.

Implications of the results

Based on our results we draw the following higher-order implications, which might be considered when designing online citizen science projects:

• Game design decisions (e.g., how a game score is calculated), such as competitive scoring mechanisms, can influence the performance of players in terms of selection of strategies.
• Volunteer contributors develop different sets of values with regard to science, and citizen science project managers need to be sensitive to such value changes within the community of volunteers.
• While games often give instant attribution to the highest score, it is also important to make attribution and credit for volunteer contributions in non-gaming contexts clear and fair.
• Competition may both attract and dishearten volunteer participants to contribute.

Limitations and further work

We consider the major limitations of this study to be in the context of the generalizability of our findings as they are mainly informed by the discussion forum posts from the participants in the two projects. As a consequence, our findings are limited by the scope of the discussion forums and the characteristics and number of citizen scientists who participate in them, a particularly low number in the case of Foldit. In this regard, our results may not provide a fair representation of the total populations of GZ and Foldit participants. However, while several published studies on citizen science focus on the analysis of single systems (Tinati et al. 2017), this work has looked across two platforms to compare and contrast the tensions between gaming and science in two projects.

The practical circumstances under which we conducted this study allowed for single coding of each project forum sample. While we realize that interjudging reliability is often perceived as the standard measure of research quality to validate a coding scheme, some scholars argue that a single researcher conducting all the coding can be both sufficient and preferred in studies where being embedded in ongoing relationships with research participants is critical for the quality of the data collected (e.g., Morse and Richards 2002). In this study, the two single coders are particularly familiar with their respective project, as they have spent time getting to know the technical environment and establishing relationships with participants. Finally, to ensure relevance of our interpretations of the data, we discussed our analysis and the results in several research group settings, as well as the 2017 Citizen Science Association conference.

With respect to the use of a conventional approach to the development of the coding scheme, while it allows generating knowledge based on participants’ unique perspectives and is grounded in the actual data, it also fails to develop a comprehensive understanding of the context, thus failing to identify key categories (Hsieh and Shannon 2005).

There are a number of gaps in our knowledge about the ways in which volunteers perceive gamified citizen science, which are highlighted by our findings and would benefit from further research. Following from this study, we need to better understand the role of normative ideals of science in the cultures of contribution developed by citizen scientists. How do citizen scientists develop scientific values? Arguably, scientists play an important role in this development, as they are often those who design online citizen science projects, define their purpose, and decide how citizen contributions are valued and acknowledged. The competitiveness of a gaming approach seems to be a contested issue among citizen scientists, therefore game-based projects should be carefully considered not only for the opportunities they offer to attract and retain contributors, but also to support responsible research and to strengthen collaboration between science and society.