Concepts of coordination theory

Malone and Crowston (1994) analyzed group action in terms of actors performing interdependent tasks to achieve some goal; i.e., in an organizational process (Crowston 1997; Crowston and Osborn 2003). These tasks might require or create various resources. For example, in the case of writing a scientific paper, actors include the authors and various members of the research team. Tasks include collecting data, performing analyses, and writing and revising a manuscript. Resources include data, analysis reports, and the analysts’ and authors’ time and effort. In this view, actors in collective action face coordination problems arising from dependencies that constrain how tasks can be performed. Studying coordination thus means analyzing the dependencies that emerge among the tasks and identifying how those dependencies are managed. Note that coordination theory focuses on the dependences between tasks, but does not analyze the details of the tasks themselves, e.g., their difficulty or diversity.

In contrast to other theories that consider dependencies among actors, coordination theory posits dependencies as occurring: 1) between a task and a resource and 2) among multiple tasks and a resource. The dependencies between a task and a resource (type 1) are shown in Figure 1. These dependencies exist when a task uses or creates a resource. For example, a data analysis task uses data that has been collected and preprocessed and creates analysis reports. An important kind of resource is the time and effort of an actor (human or otherwise) who can work on the task.

Shared use of resources by different tasks (type 2) can in turn lead to dependencies between the tasks that use or create the resource. These dependencies come in three kinds, as shown in Figure 2. First, producer-consumer or flow dependencies match Thompson’s sequential dependency (Thompson 1967): One task creates a resource that a second uses. For example, in a data analysis pipeline, the flow of data from one analysis to another creates a dependency between those tasks. Flow dependencies further imply the need to manage the usability of the resource and the timing and location of its availability (that is, a flow dependency has three aspects), e.g., data from one stage of an analysis pipeline must be suitable for the next stage and made available on time.

Second, a shared-output or fit dependence occurs when two activities collaborate in the creation of an output (in the case where the output is identical, there is potential synergy, because the duplicate work can be avoided). For example, data analyses to support a paper need to be tailored to work together.

Finally, a shared-input dependency emerges among activities that use a common resource (like Thompson’s pooled dependency). For example, data collection might require a specific scientific instrument or the time of a human observer, constraining how data collection tasks are done and requiring some coordination, e.g., a schedule of observation times.

Resources may also be directly interdependent due to physical connections (the right side of Figure 1), e.g., a section of a paper that refers to results established in a prior section or data sets that need to be analyzed as an ensemble. In this case, tasks that use one resource may have dependencies with another task that uses an interdependent resource.

The key point in coordination theory is that dependencies (of all kinds) create coordination problems that may require additional work to manage. Malone and Crowston (1994) called this additional work coordination mechanisms. For example, if expertise is necessary to perform a given task (i.e., there is a task-actor dependency, a special case of task-resource), then an actor with that expertise must be identified and the task assigned to him or her. The work of identifying an expert and maintaining a task assignment system constitutes the coordination mechanism. To avoid coordination problems (e.g., a task not getting done), this additional work must be performed by someone (e.g., a manager or even the expert him or herself).

Several mechanisms often can be used to manage a given dependency. For example, to manage a usability dependency (part of a flow dependency), the resource created might be tailored to the needs of the consumer (meaning that consumers must provide information about their needs to the producer) or a producer might follow a standard so the consumer knows what to expect. Usability dependencies are particularly salient in scientific research. Data (a particularly important resource) that are collected must be appropriate for the research question and be credible according to the standards of the field to be useful for analysis. Analysis reports (another resource) must meet the expectations of the field and provide answers to questions of interest to be useful for paper writing. Papers must be written in the genre of a scientific paper to be publishable, with the details of the genre differing from field to field. An important part of the training of a scientist is to learn the specific expectations for data, analysis reports, and papers in the scientist’s research field. That is, the expertise required to accomplish a task includes knowing how to do it in the way expected by users of the output.

Note that in developing the coordination theory framework, Malone and Crowston (1994) describe coordination mechanisms as relying on other necessary group functions, such as decision making, communications, and development of shared understandings and collective sense-making (Crowston and Kammerer 1998). To develop a complete model of a process would involve modeling all these aspects. In this paper, though, we will focus on the coordination aspects, mostly bracketing the other phenomena.

In summary, coordination theory provides a lens (i.e., a set of concepts) with which to analyze group processes: In terms of tasks, resources, dependencies, and coordination mechanisms. The theory posits that the fit or lack of fit between the dependencies and available coordination mechanisms may explain problems faced by the group in achieving its goals. Finally, the framework provides a way to structure the analysis by listing possible dependencies to look for.

Virtual citizen science

A specific example of a virtual citizen science project is Galaxy Zoo (http://galaxyzoo.org/), for which volunteers support scientific inquiry by online analysis of the millions of astronomical photographs collected by the Sloan Digital Sky Survey and others. Specifically, the Galaxy Zoo system asks individuals to answer a series of questions about the shape of a galaxy captured in an image (e.g., the number of spiral arms or how round or elliptical they are) to support astronomical research on galaxy morphology. The workflow for the data-analysis task is shown in Figure 3. Our analysis is based on our own experience with Galaxy Zoo and other virtual citizen science projects as well as published studies of these projects (e.g., Prestopnik and Crowston 2012; Wiggins and Crowston 2015; Tinati et al. 2015; Simpson et al. 2014).

One coordination problem in these projects results from a task-actor dependency that requires a task assignment coordination mechanism to match a task (an image to be classified) to an actor (a volunteer). This dependency is handled by the system simply giving the next image to be classified to the next available volunteer who has not already seen it (Reeves et al. 2017). This approach has the advantage of being simple and requiring almost no information about the image or volunteer.

A second problem is ensuring data quality, that is, the usability of the classifications as they flow to the research project. The quality of citizen science data is a key issue for many projects (Riesch and Potter 2014; Wiggins et al. 2011). In these projects, this usability dependency (an aspect of a flow dependency) is handled by having multiple volunteers repeat the classification and using the consensus to eliminate occasional classification errors.

Figure 3 also shows a flow of data from occasional serendipitous discoveries (the yellow dot in the Figure). Every image is inspected by human analysts who may identify oddities in the images, such as the Voorwerp discovered in Galaxy Zoo. As the Figure shows, such discoveries are handled outside the regular project flow, and they support research other than the planned project research (Tinati et al. 2015).

In summary, the main tasks of virtual citizen science projects such as Galaxy Zoo have minimal dependencies, and the coordination mechanisms needed are easily provided by the Zooniverse system.

Paper writing

In contrast to classifying data for citizen science, writing a scientific paper includes more varied dependences that potentially pose challenges to the writers. Figure 4 shows the structure of dependencies involved, based on published work on coordination in writing (Erkens et al. 2005), Wikipedia in particular (e.g., Kittur et al. 2009), and a detailed coordination-theory analysis of a comparable process, writing software (Crowston and Scozzi 2002; Crowston and Scozzi 2008). There are also differences in the diversity and difficulty of the tasks included, but these are not considered in analyzing the task dependencies.

A first difference between Figures 3 and 4 is the presence of dependencies among the parts of the paper, the outputs of the paper-writing tasks. Only a few tasks in writing, such as proofreading, are like galaxy classification in that they can be done without affecting other tasks (Kittur et al. 2009), e.g., by crowdsourcing (Bernstein et al. 2015). For the most part, different parts of a paper cannot be written independently. For example, the research problem presented in the introduction must be supported in the literature review, answered in the data analysis, and so on (Wichmann and Rummel 2013). Furthermore, the voice and writing style of the different sections need to match. These dependencies among paper parts impose constraints on how those parts are written (Kittur et al. 2009), posing coordination challenges to the people working on them.

To manage these dependencies requires additional work: Authors must either plan the writing process in advance (Viégas et al. 2007; Erkens et al. 2005), e.g., by developing a shared vision for the paper (Wichmann and Rummel 2013) collectively or led by one person (Kim et al. 2014), or they must write and revise their parts to fit with other parts (i.e., mutual adjustment). Teevan et al. (2016) report on a system to create microtasks for paper writing, but despite the design intent of minimizing dependencies, they observed “considerable interaction among group members” using the system.

A second dependency is a shared-output dependency, created when two authors work on tasks that have the same output, i.e., two authors working on writing the same part of the paper. Galaxy Zoo also has multiple volunteers work on the same galaxy image, but because the possible results are few in number, a simple consensus rule is usually sufficient to merge the classifications. However, many more differences can arise in writing a paper. At a basic level, problems of simultaneous changes to text can be managed by a shared document editor (Lowry and Nunamaker 2003). However, problems can arise at a conceptual level that are more difficult to identify and resolve (Erkens et al. 2005). To manage this dependency requires a technique to mitigate these possible conflicts in output, e.g., picking one version and rejecting the others or manually merging the changes.

A third dependency is the task-actor dependency. Unlike the system assignment in Galaxy Zoo, volunteers working on a paper will likely choose for themselves which tasks to work on, as they do in Wikipedia. Reliance on self-assignment of tasks fits the voluntary nature of the project but raises two potential problems.

First, people choosing to work on one part of the paper may not be good at writing, i.e., their contributions might not be usable. In a conventional team, members would be assigned to tasks based on skills, but in a voluntary setting, skills are not guaranteed. A paper-writing process must include mechanisms to assess if a writing contribution is acceptable (Kittur et al. 2007). For example, in Wikipedia, editors police edits and modify or revert problematic ones. As an alternative approach, rather than rejecting substandard results, efforts could be made to provide the volunteer with the skills necessary to achieve a good outcome, e.g., by providing training.

Second, a volunteer might not be reliable, meaning that a promised contribution might not appear on time (Riesch and Potter 2014). The writing process will thus need mechanisms to handle missing contributions. This problem interacts with the second dependency, shared output, because one way to minimize problems from the former issue is to have only one person at a time work on a task (i.e., assign authors for each document section). Such a process is problematic, however, if there is a chance that the task (i.e., the document section) will not be completed.

A final dependency is a usability dependency (part of the flow dependency) between the creation of the paper and the use of the paper by its intended audience. In the basic work of citizen science projects, the usability of the resulting data set is managed by having the science teams design the process of creating the data, with carefully imposed quality checks (Simpson et al. 2014). For scientific writing, this dependency is handled in part by processes such as peer review that check for article quality. However, much of the process is handled by the authors themselves acting as proxies for the readers. Knowing the scientific literature, scientific authors pick topics and write in ways that they know will be useful for that community (e.g., in the genre of a scientific article). A volunteer-driven writing process will need ways to provide information about the needs and desires of the readers to the volunteer authors, who again cannot be assumed to have specific knowledge needed to make these decisions themselves.

In summary, the task of writing a paper displays more and more varied dependencies than a prototypical citizen science project, thus posing additional coordination challenges to the volunteers. As a result, in the Quench project, we expected to see either additional work done to manage these dependencies, or problems arising from these dependencies going unmanaged. Identifying the kinds of coordination mechanisms created or needed will be informative for managers of citizen science projects interested in involving volunteers in these additional kinds of scientific work and by extension, to other crowd researchers.

Coordination analysis

Crowston and Osborn (2003) led us to pay attention to dependencies in the tasks in Galaxy Zoo Quench. The analysis has six steps: Defining process boundaries, collecting data, determining actors and resources, determining activities, determining dependencies, and model verification. We analyzed our data using this technique, which highlighted dependencies in the system and led to our documentation of the coordination process associated with each project. Equally important, this technique reveals areas where the management of dependencies broke down, causing coordination problems.

Phase 1

Phase 1 of the Quench project focused on the tasks of classifying galaxies. In this phase 1, the volunteers coded galaxies to find those with the characteristics of post-quenched galaxies. The classification included characteristics believed to be related to quenching, specifically galaxies merging, tidal debris, both, or neither. Galaxy classification is a mature process that has been used on many citizen science projects, mostly notably Galaxy Zoo. The classification was somewhat delayed, but successfully completed in that the selected galaxies had classifications by the end of August 2013 (Source a). Of the respondents to the post-survey, 129 (70%) reported working on Phase 1.

Phase 1b

The project as executed included an additional phase between the initial coding and data analysis, labelled Phase 1b in Figure 5. This phase, which was not explicitly described in the announced plans for the Galaxy Zoo Quench project, represents the first collective task in the process, building consensus on the data created by the individual classifications made by the volunteers in Phase 1 to generate a dataset for analysis in Phase 2. The assumption was that once the galaxies were coded, the results could be used for analysis. In fact, reaching consensus on the dataset turned out to be a significant undertaking for the group.

First, as the volunteers started to use the data, they raised concerns about how the final classification was assigned (Sources b, c & d). The initial algorithm used to determine a classification was to take the option selected by the most volunteers, as in other Galaxy Zoo projects (Source e). For example, if “merging” was selected by 3, “tidal debris” by 6, “both” by 2, and “neither” by 7, the galaxy would be classified as “neither,” even though together the other choices, which indicate an interesting finding, had been chosen more often. This discrepancy was fixed by revising the algorithm to add the count of the three interesting findings together.

Second, the process of revising the dataset led to concerns about the usability of the data. Two of the themes in the post-survey were data and tool problems. One respondent commented that “data sets were slow to become available, they contained errors and the analysis tools were immature or buggy.” For instance, volunteers were uncertain about the data reliability given the significant changes made between versions. In some cases, errors crept into the files as they were processed by different people. For example, identifiers for the galaxies in the data file are 18-digit numbers. If the file is opened in Excel (a common tool for citizen scientists and scientists more generally because it is widely available), these long numbers could be converted to floating point numbers and truncated, changing the ID, a problem that beset some versions of the data file. There was also inconsistency in variable labeling between datasets, which raised questions about the data provenance (Source f).

A third set of questions arose about the control group of galaxies. To provide a comparison to the quenched galaxies, the scientists involved in the project selected a control group of 3,002 galaxies but did so independently from the volunteers. The citizen scientists requested clarification on the selection of the control group, which was explained, but doubts remained (Source g). Throughout the project, a recurrent discussion involves the suitability of the sample of galaxies for the study. Sampling had to be done carefully to avoid introducing bias into the results. Participants developed different subsamples based on different selection rules, but did not seem to reach consensus about which sample should be used.

Phase 2

The next phase of the project, Phase 2 in Figure 2, was data analysis. The lead scientist working with the volunteers had encouraged them to “play” with the data and to “have fun and ferret out interesting trends in the data” (Source h). The intent was that the volunteers would explore on their own and then share interesting results with the group, thus experiencing the process of scientific discovery.

As noted above, volunteers had already been observed engaging in analyses of other data sets in other projects. While about half of the respondents to the post-survey commented on a perceived lack of confidence in their own abilities (or a lack of time that would be needed to gain the skills), others seemed capable of such work. Figure 6 shows the responses to questions on the pre-survey about data analysis skills, indicating that the majority of respondents felt that they had relevant skills (with the exception of writing a paper). The subset of volunteers who worked on Phase 2 seemed even more knowledgeable. Indeed, a number of the responses to the post-survey note the high level of the discussion as a deterrent to participation, e.g., “The knowledge and ability of the remaining participants so far exceeded my own knowledge.” Furthermore, different analyses could be done in parallel, i.e., there was no dependency between the analyses that would require the volunteers to coordinate their work on the analyses.

Unexpectedly, though, the group encountered difficulties in this phase. Volunteers perceived the task as too open-ended and so did not know how best to proceed. A theme in the post-survey was unclear guidance. For example, one respondent commented that he “was unsure how to report my results, and on a more basic level was unsure about which descriptive and inferential stats were the most relevant to the project.” Another stated more bluntly, “I never could figure out what they wanted, what the point was, so, I was there for maybe 20 minutes, clicked out, VERY confused, and never went back.” Part of the volunteer feedback on the project was that the project needed more scaffolding of the research process.

Unfortunately, during this phase, the lead scientist became unavailable for some time and none of the other scientists on the project could take on a leadership role (Source i; also noted on the post-survey). Problems caused by the absence of a single key individual would not be surprising in a conventional team, but they were unexpected in the context of a citizen science project in which members were able—and expected—to make independent contributions. The volunteers attempted to continue the project, with extensive discussion and various analyses developed. However, the volunteers did not reach a final decision about what should be done, so Phase 2 did not progress to having the desired final set of analyses and a scientific story. As a final analysis was not completed, Phase 3, writing, never started.

In 2014 (and again more recently), a citizen scientist attempted to revive the project, receiving responses from the other citizen scientists, as well as from three scientists (Source j). However, the discussion ended without the project restarting, and no further posts appeared on the Galaxy Zoo Quench Talk board.