Coordinating Advanced Crowd Work: Extending Citizen Science

Crowdsourcing work with high levels of coupling between tasks poses challenges for coordination. This paper presents a study of an online citizen science project that involved volunteers in such tasks: not just analyzing bulk data but also interpreting data and writing a paper for publication. However, extending the reach of citizen science adds tasks with more dependencies, which calls for more elaborate coordination mechanisms but the relationship between the project and volunteers limits how work can be coordinated. Contrariwise, a mismatch between dependencies and available coordination mechanisms can be expected to lead to performance problems. The results of the study offer recommendations for design of crowdsourcing of more complex tasks.


Introduction
The past decade has seen a rapid growth in the number of online crowdsourcing projects [5,13].Many of these projects involve the crowd in rather simple tasks (i.e., microtasking).A limited number of projects ask participants to solve complex tasks, but these often rely on small groups of participants to submit complete answers (e.g., innovation competitions or the Climate CoLab) or require design of a workflow that decomposes the task into microtasks [21].For the later, researchers have proposed ways to automatically generate workflows [e.g., 2] or to use the crowd to create them [e.g., 17].Others have examined ways to chain modular tasks [e.g., 29].
Underlying these efforts is the notion that complex tasks can be divided into pieces to be done by individuals, but that doing so is complicated by the possibility of dependencies among the pieces that need to be managed.To develop a deeper understanding of how to better design and support advanced work in crowdsourcing, the present paper asks: What coordination challenges do online groups face when undertaking work with a high level of coupling between tasks?
To answer the research question, we studied crowdsourced citizen science work.Citizen science is a broad term describing scientific projects that rely on contributions to scientific research from members of the public (i.e., citizens in the broadest sense of the word).There are numerous kinds of citizen science projects: some have volunteers collect data, while others have volunteers analyze already-collected data.The interactions between volunteers and the project organizers are often via the Web, e.g. on a site that accepts contributed data (e.g., eBird) or that presents data and collects volunteers' annotations (e.g., Zooniverse).As a result, citizen science is sometimes described as crowdsourced science [e.g., 26].
Studies of citizen science volunteers suggest that many are motivated by the opportunity to contribute to real science [27,31] and by recognition for such contributions [32].Accordingly, some sponsors of citizen science projects seek to involve volunteers more deeply in the science of the project: not just collecting or processing data, but also taking part in further data analysis and even paper writing [23].
Efforts to further involve volunteers in more advanced tasks are viewed as important in part to demonstrate that citizen science is not just crowdsourcing without pay, an exploitation of the citizen scientist volunteers by project scientists.To be fair to the volunteers, project scientists need to give back [30,24], and expanding access to science is one way to do so.Allowing participants to see and talk about the data is only the first step in expanding access [42].However, to successfully include volunteers more deeply in scientific research requires careful consideration of the kind of project management needed, i.e., how to coordinate these contributions.
This paper presents a case study of the Galaxy Zoo Quench project, a project sponsored by the Zooniverse in which volunteers were invited to write an academic paper in collaboration with the project scientists.The Galaxy Zoo project had already had great success involving volunteers to work on classification of galaxies.The capability of the volunteers to do original work had seemingly been proven by discoveries such as Hanny's Voorwerp, a novel astronomical object identified by a citizen scientist [18].Furthermore, citizen science volunteers had been observed to engage in their own analyses of project data, posting questions and results to the discussion boards [36,3] and some had been involved individual in further research.The next logical step appeared to be involving a group of volunteers in scientific collaboration through the entire process of scientific research, from data analysis to publication, which seemed like a credible goal.The project can be seen as moving the Zooniverse from what Bos, et al. [4] term an Open Community Contribution System to a Distributed Research Center.
Theoretically, we draw on coordination theory to explore the challenges associated engaging members of a crowd in advanced science tasks.The Quench case is rich and can be viewed from numerous perspectives, but we chose coordination theory for our analysis because it seemed to provide insight into the challenges faced by a distributed group trying to work together.To support this analysis, we first analyze the work of citizen science projects and the process of writing an article, to explore the nature of dependencies and coordination that would apply in the individual phases of the Quench project.

Theory: Coordination theory
We first introduce the topic of coordination and present the fundamentals of coordination theory, the theoretical foundation for this paper.Coordination, defined as "managing dependencies between activities" [20], is a central feature of collective action.This definition of coordination is consistent with the large body of literature developed in the field of organization theory [e.g., 12,35] that emphasizes the importance of interdependence in group work.
Coordination theory [20] synthesizes contributions from different disciplines to develop a systematic approach to the study of coordination.Malone and Crowston [20] analyzed group action in terms of actors performing interdependent tasks to achieve some goal; i.e., in an organizational process [6,8].These tasks might require or create various resources.For example, in the case of writing a scientific paper, actors include the authors and various members of the research team.Tasks include collecting data, performing analyses and writing a revising a manuscript.Resources include data, analysis reports and the analysts' and authors' time and effort.
In this view, actors in collective action face coordination problems arising from dependencies that constrain how tasks can be performed.Studying coordination thus means analyzing the dependencies that emerge among the tasks in a system and identifying how those dependencies are managed.
In contrast to other theories that consider dependencies among actors, coordination theory classifies dependencies as occurring between a task and a resource, among multiple tasks and a resource, and among a task and multiple resources.The dependencies between a task and a resource are shown in Figure 1.Dependencies between a task and a resource arise because a task uses or creates a resource.For example, a data analysis task uses data that has been collected and preprocessed and creates analysis reports that might be used to write a paper.Resources may also be directly interdependent due to physical connections (the right side of Figure 1), e.g., a section of a paper that refers to results established in a prior section or data sets that need to be analyzed as an ensemble.
Shared use of resources can in turn lead to dependencies between the tasks that use or create the resource.These dependencies come in three kinds, as shown in Figure 2. First, producer-consumer or flow dependencies match Thompson's sequential dependency [35]: one task creates a resource that a second uses.For example, in a data analysis pipeline, the flow of data from one analysis to another creates a dependency between those tasks.Flow dependencies further imply the need to manage the usability of the resource and  the timing and location of its availability (that is, a flow dependency has three aspects), e.g., data from one stage of an analysis pipeline must be suitable for the next stage and made available on time.Second, a shared-output or fit dependence occurs when two activities collaborate in the creation of an output (in the case where the output is identical, there is potential synergy, since the duplicate work can be avoided).For example, data analyses to support a paper need to be tailored to work together.
Finally, a shared-input dependency emerges among activities that use of a common resource (like Thompson's pooled dependency).For example, data collection might require a specific scientific instrument, constraining how data collection tasks are done, e.g., a schedule of observation times.Note that information as a resource is shareable, which can ease management of shared input dependencies, but simultaneously creates a different dependency of ensuring that different tasks are working with the same version of the data.
The key point in coordination theory is that dependencies create problems (or possible synergies) that may require additional work to manage.Malone and Crowston [20] called this additional work coordination mechanisms.For example, if expertise is necessary to perform a given task (i.e., there is a task-actor dependency), then an actor with that expertise must be identified and the task assigned to him or her.The work of identifying an expert and maintaining a task assignment system constitutes the coordination mechanism.
There are often several mechanisms that can be used to manage a given dependency.For example, mechanisms to manage the dependency between an activity and an actor include (among others): (1) having a manager pick someone to perform the task; (2) assigning the task to the first available actor; (3) a labour market in which actors bid on jobs; and (4) selfassignment of work, as in many volunteer projects.
To manage a usability dependency (part of a flow dependency), the resource created might be tailored to the needs of the consumer (meaning that consumers must provide information about their needs to the producer) or a producer might follow a standard so the consumer knows what to expect.Usability dependencies are particularly salient in scientific research.Data that are collected must be appropriate for the research question and be credible according to the standards of the field.Analysis reports must meet the expectations of the field and provide answers to questions of interest.Papers must be written in the genre of a scientific paper, with the details of the genre differing from field to field.An important part of the training of a scientist is to learn the specific expectations for data, analysis reports and papers in the scientist's research field.That is, the expertise needed to do a task includes knowing how to do it in the way expected by users of the output.
Coordination mechanisms may be useful in a wide variety of organizational settings.For example, managerial assignment of tasks to subordinates is commonly employed.Conversely, organizations with similar goals achieved with more-or-less the same set of activities will have to manage the same dependencies, but may choose different coordination mechanisms, thus resulting in different processes.
Finally, the mechanisms are themselves tasks that must be performed by some actors, and so adding coordination mechanisms to a process may create additional dependences that must themselves be managed.For example, for a manager to be able to assign a task may require information about the abilities and current workload of subordinates, requiring additional work to gather this information.
It should be noted that in developing the coordination theory framework, Malone and Crowston [20] describe coordination mechanisms as relying on other necessary group functions, such as decision making, communications and development of shared understandings and collective sense making [7].To develop a complete model of a process would involve modeling all these aspects: coordination, decision making, communication and sense-making.In this paper though, we will focus on the coordination aspects, mostly bracketing the other phenomenon.
In summary, coordination theory provides a lens with which to analyze group processes in terms of tasks, resources, resulting dependencies and selected coordination mechanisms.Furthermore, the fit or lack of fit between the dependencies and available coordination mechanisms may explain problems faced by the group in achieving its goals.

Coordination in citizen science projects and in paper writing
In this section, we present a theoretical analysis of citizen science projects from a coordination-theory perspective as a basis for analyzing the work of Galaxy Zoo Quench.We start by presenting an analysis of the work of Galaxy Zoo, which was the basis for the Quench project.These analyses are based on our own experiences with the sites and published studies of these citizen science projects [e.g., 25,40,36,33].The quality of the data created by the citizen scientists for scientific research emerges as a key issue from these analyses [30,41] and provide a comparison point for understanding the more ambitious work of Galaxy Zoo Quench.We then develop an analysis of the coordination needed for the task of writing a paper, as writing a paper was the goal of the Quench project.

GalaxyZoo.
Galaxy Zoo (http://galaxyzoo.org/) is a citizen science project that has volunteers support scientific inquiry by online analysis of the millions of astronomical photographs collected by the Hubble Space Telescope, the Sloan Digital Sky Survey, and others.Specifically, the Galaxy Zoo system asks individuals to answer a series of questions about the shape of a galaxy captured in an image (e.g., the number of spiral arms or how round or elliptical they are).The resulting data supports astronomical research on galaxy morphology.The workflow for the data analysis task in the project, from galaxy classification to astronomical research, is shown in Figure 3.
Figure 3 also shows the flow of data from occasional serendipitous discoveries.Every image is inspected by human analysts, who may identify oddities in the images.Citizen scientists working on classification have found novel astronomical objects, such as Hanny's Voorwerp [18].As the figure shows, such discoveries are handled outside the regular flow in the project and support research other than the planned project research [36].
One coordination problem in the Galaxy Zoo project is task assignment, matching an image to be classified to a volunteer.In the Galaxy Zoo project, this dependency is handled by the system simply giving the next image to be classified to the next available volunteer who has not already seen it [28].This approach has the advantage of being simple and requiring no information about the image or volunteer.
A second problem is ensuring data quality, that is, the usability of the data classifications for the research project.In Galaxy Zoo (and similar projects), this usability dependency is handled by having multiple volunteers repeat the classification and taking the consensus.
In summary, the tasks of Galaxy Zoo have minimal dependencies that can easily be handled by the system.As a result, the level of coordination needed in the Galaxy Zoo and similar projects is minimal.

Paper writing.
In contrast to citizen science classification, the dependencies in writing a scientific pa-per are more complicated.Figure 4 shows the structure of dependencies involved, based on published work on coordination in writing [11], Wikipedia in particular [e.g., 15] and a detailed coordination-theory analysis of a comparable process, writing software [10,9].
A first difference between Figures 3 and 4 is the presence of dependencies between the parts of the paper, the outputs of the paper writing tasks.Only a few tasks in writing, such as proofreading, are like galaxy classification in that they can be done without affecting other tasks [15], i.e., by crowdsourcing [1].For the most part, different parts of a paper cannot be written independently.For example, the research problem presented in the introduction to a paper must be supported in the literature review, answered in the data analysis, and so on [39].Furthermore, the voice and writing style of the different sections needs to match.These dependencies among parts of a paper impose constraints on how the paper parts are written [15].To manage these dependencies requires additional work as authors must either plan the writing process in advance [38,11], e.g., by developing a shared vision for the paper [39] (collectively or led by one person [14]), or writing and revising their parts to fit with other parts.
[34] report on a system to create microtasks to support paper writing, but despite the design intent, observed "considerable interaction among group members" using the system.
A second dependency is a shared-output dependency, created when two authors work on tasks that have the same output, i.e., two authors working on writing the same part of the paper.Galaxy Zoo also has multiple volunteers work on the same galaxy image, but because there are a small number of possible results, a simple consensus rule is usually sufficient to merge the classifications.However, many more differences can arise in writing a paper.At a basic level, problems of simultaneous changes to text can be managed by a shared document editor such as Google Docs [19].However, there can be problems at a conceptual level that are more difficult to identify and resolve [11].To manage this dependency requires some tech-  nique to mitigate these possible conflicts in output, e.g., picking one version and rejecting the others or manually merging the changes.A third dependency is the task-actor dependency.Unlike the system assignment in Galaxy Zoo, volunteers working on a paper will likely chose for themselves which tasks to work on, as in Wikipedia.Reliance on self-assignment of tasks fits the voluntary nature of the project, but raises two potential problems.
First, people choosing to work on some part of the paper might not be good at it, i.e., their contributions might not be usable.In a conventional team, members would be assigned to tasks based on skills, but in a voluntary setting, skills are not guaranteed.A paper writing process will have to include mechanisms to assess if a writing contribution is acceptable [16].For example, in Wikipedia, editors police edits and modify or revert problematic ones.Conversely, efforts could be made to provide the volunteer with the necessary skills, e.g., by providing training.
Second, a volunteer might not be reliable, meaning that a promised contribution might not appear [30].The writing process will thus also need mechanisms to handle missing contributions.This problem interacts with the second dependency, shared output, as one way to minimize problems from the former issue is to have only one person at a time work on a task (i.e., assign authors for each document section), but such a process is problematic if there is a chance that the task (i.e., the document section) will not be completed.
A final dependency is between the creation of the paper and the use of the paper by its intended audience.In the basic work of citizen science projects, the usability of the resulting data set is managed by having the science teams design the process of creating the data, with carefully imposed quality checks [33].For scientific writing, this dependency is handled in part by pro-cesses such as peer review that check for article quality.However, much of the process is handled by the authors themselves acting as proxies for the readers.Knowing the scientific literature, scientific authors pick topics and write in ways that they know will be useful for that community (e.g., in the genre of a scientific article).A volunteer-driven writing process will need ways to provide information about the needs and desires of the readers to the volunteer authors, who again cannot be assumed to have specific knowledge.
In summary, the task of writing a paper displays a more complicated structure of dependencies than a prototypical citizen science project.As a result, in the Quench project, we expect to see either additional work done to manage these dependencies, or problems arising from these dependencies going unmanaged.Identifying the kinds of coordination mechanisms created or needed will be informative for managers of citizen science projects interested in involving volunteers in these additional kinds of scientific work and by extension, to other crowd researchers.

Methods
Methodologically, the present study of Zooniverse Quench combines collaborative basic research [37] and coordination analysis [8].We introduce each in turn.
The present study engaged in collaborative basic research as defined by van de Ven [39] to understand the design and outcomes of a specific kind of crowdsourcing, online citizen science.We did so through a close collaboration with developers, designers and educators at Zooniverse.Data gathering included questionnaires, interviews and focus groups addressing volunteer motivation and learning as well as trace data analysis on a variety of topics.We conducted extensive analysis of the discussion board associat- BGS000000f/discussions/DGS000023b j Quench project: a proposal aimed at reviving and completing it BGS000000e/discussions/DGS000022f 1 Unless a complete URL is given, URLs start https://quenchtalk.galaxyzoo.org/#/boards/ed with Galaxy Zoo Quench to map the history of the project and important events and decisions made over the course of the project.A list of talk posts referenced in the paper is given in Table 1.As well, the study draws on several years of prior engagement with the broader citizen science community beyond Zooniverse.
Coordination analysis [8] led us to pay attention to dependencies in the work processes in Galaxy Zoo and Galaxy Zoo Quench.The analysis has six steps: defining process boundaries, collecting data, determining actors and resources, determining activities, determining dependencies and model verification.We analyzed our data using this technique, which highlighted dependencies in the system led to our documentation of the coordination process associated with each project.Equally important, this technique allowed us to specify areas where the management of dependencies broke down, causing coordination problems.

Results: Coordination problems in Zooniverse Quench
We turn next to an examination of dependencies, coordination mechanism and observed coordination problems in the Galaxy Zoo Quench project.We start by presenting a synopsis of the history of the project before turning to a coordination analysis.

Case background
The Galaxy Zoo Quench project aimed to research, write and publish an academic paper in collaboration with citizen scientists.The topic of the Quench project was "quenched" galaxies, that is, galaxies that have ceased star formation.Galaxies can quench for different reasons and understanding why different kinds of galaxies quench can shed light on the processes of galaxy evolution.The plan was to code a collection of quenched galaxies for various properties and then compare those galaxies to a matched sample of unquenched galaxies to identify their distinctive properties.Volunteers would classify the galaxies, as in other citizen science projects, conduct data analysis and co-author a professional journal article (Source a).
The plan was to complete Phase 1, the classification process, by 1 August 2013 and then proceed to the second phase, data analysis and discussion.The goal for the end of phase 3 was to submit an article to Monthly Notices of the Royal Astronomical Society (MNRAS) Letters, the online portion of the MNRAS Journal, totaling 4-5 pages.Figure 5 presents the flow of data throughout the project, indicating in grey boxes the major outcomes of each of the Phases.
We next describe each of the phases in more detail with attention to the coordination difficulties encountered.Phase 1 consisted of coding galaxies with the characteristics of post-quenched galaxies.The classification included characteristics believed to be related to quenching, specifically galaxies merging, tidal debris, both or neither.Galaxy classification is a mature process that has been used on several citizen scientist projects, mostly notably Galaxy Zoo.Many participants classify each galaxy such that the answers of any one individual has little effect on the outcome.The classification was somewhat delayed, but successfully completed (in that the selected galaxies had classifications) by the end of August 2013 (Source a).The project as executed included an additional, initially undescribed, phase between the initial coding and data analysis that we have labelled Phase 1b in Figure 5.This phase represents the first collective task in the process, building consensus on the data created in Phase 1 to generate a dataset for analysis in Phase 2.
The assumption was that the galaxies were coded, the results could be used for analysis but in fact it turned out to be a significant undertaking for the group to reach a consensus on the dataset.
First, as the few volunteers who were continuing to Phase 2 started to use the data, they raised concerns about how the final classification was assigned (Sources b, c & d).The initial algorithm used to determine a classification was to take the option selected by the most volunteers, as in other Galaxy Zoo projects (Source e).For example, if "merging" was selected by 3, "tidal debris" by 6, "both" by 2, and "neither" by 7, the galaxy would be classified as "neither", even though together the other choices that indicate an interesting finding had been chosen more often.This discrepancy was fixed by revising the algorithm to add the count of the three interesting findings together.
Second, the process of revising the dataset led to concerns about the usability of the data.Volunteers were uncertain about the data reliability given the significant changes made between versions.In some cases, errors crept into the files as they were processed by different people.For example, identifiers for the galaxies in the data file are 18 digit numbers.If the file is opened in Excel (a common tool for citizen scientists since it is widely available), these long numbers could be converted to floating point numbers and truncated, changing the ID, a problem that beset some versions of the data file.There was also inconsistency in variable labeling between datasets, which raised questions about the data provenance (Source f).
A third set of questions arose about the control group of galaxies.To provide a comparison to the quenched galaxies, the scientists involved in the project selected a control group of 3002 galaxies, but did so independently from the citizen scientists.The citizen scientists requested clarification on the selection of the control group, which was explained, but doubts remained (Source g).Throughout the project, a recurrent discussion involves the suitability of the sample of galaxies for the study.Sampling had to be done carefully to avoid introducing bias into the results.Participants developed different subsamples based on different selection rules, but did not seem to reach consensus about which sample should be used.
The next phase of the project, Phase 2 in Figure 2, was data analysis.The lead scientist working with the volunteers had encouraged them to "play" with the data and to "have fun and ferret out interesting trends in the data" (Source h).The intent was that the volunteers would explore on their own and then share interesting results with the group, thus experiencing the process of scientific discovery.As noted above, volunteers had already been observed engaging in analyses of other data sets, and the specific volunteers involved seemed capable of such work (i.e., they had the necessary skills for the task of analysis).Furthermore, different analyses could be done in parallel, i.e., there was no dependency between them.
Unexpectedly though, the group encountered difficulties in this phase.Volunteers perceived the task as too open-ended and so did not know how best to proceed.Part of the volunteer feedback on the project was that the project needed more scaffolding of the research process.
Further, during this phase, the lead scientist became unavailable for some time and none of the other scientists on the project could take on a leadership role (Source i).Problems caused by the absence of a single key individual would not be surprising in a conventional team, but it was unexpected in the context of a citizen science project in which members were ableand expected-to make independent contributions.
The volunteers attempted to continue the project, with extensive discussion and various analyses developed.However, the volunteers did not reach a final decision about what should be done, so Phase 2 did not progress to having the desired final set of analyses and a scientific story.As a final analysis was not done, Phase 3, writing, never started.
In 2014, a citizen scientist attempted to revive the project, receiving responses from the other citizen scientists, as well as from 3 scientists (Source j).However, the discussion ended without the project restarting and there were no further posts on the Galaxy Zoo Quench Talk board.

Discussion
In this section, we interpret the case using coordination theory to identify what kinds of dependencies existed, how those dependencies were managed or not managed and the impact of these dependencies on project performance.
Phase 1 focused on the tasks of classifying galaxies.Participants could work independently and concurrently to classify the post-quenched galaxies, with minimal dependencies creating constraints on their work.Classification is a mature process, with a sound technological platform and significant history of being completed in Galaxy Zoo, as well as other citizen science projects.The task of looking at an image and clicking on classifications is well-defined.Citizen scientists were both producers and consumers of the data, at least for those continuing to participate beyond Phase 1, so they had significant motivation to complete the task in a timely fashion.As a result, Phase 1 was completed successfully.
Phase 1b was the first collective task in the process.In this phase, the volunteers undertook several tasks to refine the data set for analysis.There is a dependency among these various data refinements tasks because they are towards a common output.For example, an important part of the analysis was determining which galaxies to include or to exclude in a way that did not introduce biases in the sample that would affect the results.In this phase, the volunteers started to experience difficulties ensuring that the decisions were made consistently.Another interpretation is that the analysis task has a usability dependency with the creation of the data set and the various steps undertaken to refine the data are ways to manage this dependency.However, it was difficult for the volunteers to know what criteria were appropriate, given their lack of expertise in this analysis.
Phase 2 seemed to suffer from more significant coordination problems.First, seemingly to encourage exploration and serendipitous discoveries, the project scientists seem to have provided only general guidance about what analyses should be done, planning to react to the findings of the volunteers.The problem experienced by the volunteers was parallel to the difficulties in developing a suitable data set: even when they have the skills to do an analysis, the volunteers do not have the expertise to know which analyses will be suitable for publication, so they cannot ensure the usability of their output for the next phase, paper writing.
Our initial expectation was that the project would face challenges particularly in Phase 3, due to the complexity of academic writing and level of coordination required to generate a coherent paper.However, as Phase 3 did not start, this case does not provide data to illuminate this question.
In summary, our analysis of the dependencies in the project suggest a key problem throughout was ensuring the usability of the outputs of each phase of the project for the next phase.In the first phase of the project, the usability of the galaxy classifications was ensured by the design of the coding system and of the Zooniverse system.Even here, issues arose because the coding system was more complicated, requiring a different aggregation technique.Next, creating a data set that was suitable for analysis (Phase 1b) required not only coding galaxies but also selecting a suitable sample, which requires expertise to do in an acceptable way.Finally, in Phase 2 the project ran into unexpected difficulties in finalizing a set of analysis results that would support a paper.Because the volunteers were not expert in astrophysics, it did not seem possible for them to say what analyses would be suitable.
It is interesting to speculate what would have been the result in the Quest case if the volunteers had been given more specific direction on which analyses to run.However, this approach would have been contrary to the goal of the project, which was to allow volunteers to engage in discovery on their own.Another way to express the problem experienced is that there was a need to decompose the overall task of developing an analysis into more specific subtasks that different volunteers can work on.However, the volunteers lacked the astrophysical knowledge needed to do this decomposition and the concomitant recombination.As a result, when the lead project scientist was unable to continue giving guidance, the analysis process ground to a halt.The continued interest of the volunteers suggests that the project did not suffer from a lack of motivation on their part.However, the task of managing the usability dependency between analysis and paper writing (and to some extent, between data collection and analysis) turned out not to be one that could be entirely delegated to a volunteer, no matter how motivated.

Conclusions
From our initial analysis, we expected that citizen scientists would encounter problems coordinating the work of writing a paper due to the increased coordination demands of this task as compared to the low level of dependencies in typical citizen science work.Unexpectedly, the Quench project encountered significant difficulties at the prior phase of developing a dataset and conducting analyses, even though volunteers had an interest, motivation and prior demonstrated ability to conduct analyses and in principle the tasks to be done had low interdependencies as different analyses could be carried out separately.
In the reported case, a key issue throughout is the apparent difficulty for volunteers to assess the usability of their work as a scientific product, a task that requires scientific domain knowledge to be able to perform.In Zooniverse, volunteers thrived when given clear tasks.A few could take on more advanced tasks.However, they were ultimately not able to make decisions about what constituted an interesting dataset or result.Without that input, the project could not progress.
Our analysis leads to several recommendations for how to support advanced work with citizen scientists.Given the reliance of citizen science on volunteers selfselecting tasks, the first recommendation is that it is necessary to carefully analyze the tasks to ensure that they are feasible for volunteers.
First, it is important to have a complete accounting of what that tasks are.The analysis done in designing the Quench project seems to have overlooked the work that precedes and surrounds specific analyses.Specifically, the scientists did not seem to account for the work that must be done to ensure that a dataset is usable for analysis or to select which analyses will be interesting to perform.It may that for experienced researchers, this type of work "goes without saying", but in a crowd setting, it needed to be spelled out.
A second issue the case highlights is the difference between knowing how to do a task and knowing what users of the output will find useful.It seems that much of the work of ensuring the usability of outputs required tacit knowledge, in this case about what data should look like or what analyses are interesting for publication.The problem of volunteers evaluating their results has been noted in other crowdsourcing settings [22].For a task to be suitable for crowd work, these evaluation criteria need to be made explicit.
It is tempting to say that the problems experienced in the Quench project are due to the volunteers' lack of knowledge and that project managers should have selected more suitable participants.However, a rigorous volunteer selection process is in some ways counter to the spirit of voluntary projects like citizen science.Nor is there any guarantee that volunteers with the right skills exist or are interested in the project.
Given a need for specific skills, citizen science projects sometimes provide training, which can be quite intensive (e.g., in the details of a data collection protocol).However, it does not seem feasible to train volunteers to develop the kind of insight needed to know what kinds of data or analyses will be interesting for publication.Indeed, even advanced graduate students in a topic can struggle with these questions.
Instead, we recommend that projects faced with these sorts of usability dependencies implement feedback mechanisms to quickly evaluate proposals from the volunteers and to provide guidance on improving them (likely the original plan for the Quest project).However, it is hard to know whether feedback alone would be enough to guide volunteers to a publishable analysis result.
And as noted, our analysis of the coordination needed for collaborative writing suggests that the volunteers would have faced significant challenges had they gotten to Phase 3. Exploring the kinds of challenges involved in this sort of work remains a topic for further research.

Figure 1 .
Figure 1.Tasks and resources and dependencies between tasks that create/use resources and among interdependent resources.Figure 2. Dependencies between tasks based on shared use of resources.

Figure 2 .
Figure 1.Tasks and resources and dependencies between tasks that create/use resources and among interdependent resources.Figure 2. Dependencies between tasks based on shared use of resources.

Figure 3 .
Figure 3. Flow of data in the Galaxy Zoo project.

Figure 4 .
Figure 4. Expected structure of dependencies in writing a paper.

Figure 5 .
Figure 5. Expected structure of dependencies in Galaxy Zoo Quench (data flows from the bottom up).

Table 1 .
Galaxy Zoo Quench talk posts referenced in the paper.