Study Context

We gathered information from 13 candidate CS project leaders who involved volunteers in observation-based data collection and expressed interest in learning more about their volunteers’ skill proficiency. Candidates were identified through our work on a prior project (Stylinski et al. 2020), and through a snowball sampling method. From these, we selected five based on four predetermined commonalities that were verified during the interview process: (1) databases containing repeated submission of data by individual (unique, known) volunteers, (2) existing procedures for validating volunteer data, (3) scientific investigation as a primary goal (see Wiggins and Crowston 2011), and (4) primarily adult volunteers. Beyond their data validation practices, one of the five had conducted prior studies of volunteer skills prior to this project (Parrish et al. 2019) and another used machine learning based on skill levels, but had not studied skills directly (Zevin et al. 2017).

Our five partners reflect the range of CS projects and included field monitoring, species identification, and image classification projects that utilized a range of data collection strategies: hands-on; out-of-doors repeated sampling; and online, crowdsourcing projects. The number of volunteers ranged from a few hundred to several thousand, and available data ranged from hundreds to over one million data points (See Table 1). Each team included a CS project director (who received a stipend) and an analyst (who received payment for their work).

Instruments and Procedures

Four semi-structured interviews were conducted with each team via Zoom using a standard protocol designed to capture information on secondary analysis processes. Baseline interviews were conducted in spring 2018 to plan the secondary analysis. Questions focused on the tasks and science inquiry skills volunteers needed to fulfill the citizen science project work, the common mistakes that volunteers might make, and how the project’s data validation process worked to catch those mistakes. Using the results from these interviews, our team continued to support planning efforts through the summer of 2018 via facilitated exercises both prior to and during a two-day in-person meeting.

Additional facilitated exercises were used to document and reflect on the process each partner team used to conduct their secondary analysis. The first of these was a Skill Ranking Worksheet that includes nine questions that helped prioritize the list of variables identified in the first interview based on data availability and the kinds of results that were likely to have the greatest influence on the project. We also shared a Skill Hierarchy Worksheet that was used to identify the higher-level goal or outcome of interest for the analysis (e.g., accurate data collection), as well as the underlying tasks, skills, and sub skills that must be accomplished to achieve the overall goal. Copies of both worksheets can be found at the project’s website (https://sites.google.com/umces.edu/embeddedassessment/welcome-to-eas).

A midpoint interview was conducted between fall 2018 and winter 2019; for most teams, these occurred after data had been prepared for analysis. Questions in the second interview centered on each team’s progress toward conducting their secondary analysis with a focus on the steps that were both easier and more challenging than originally expected. The third and fourth interviews were conducted after the CS team had completed their final analyses. These began in late 2019 and continued through early 2021; the third interview was conducted soon after each team completed their analysis and final interviews were conducted several months later. Interview questions for the third interview documented the key findings of the analysis and the value of the process to the project. The final interview focused on ways teams had used the results of their analysis to inform their work and the CS field.

This research was approved by the University of Maryland Institutional Review Board (IRB #1072528). A sample set of interview questions are included in the supplemental documents.

Coding

To analyze the interview data, three researchers used a six-step collaborative qualitative analysis process to develop two coding schemes (Richards and Hemphill 2018). A deductive coding scheme focused on the steps CS teams took to prepare for and conduct their secondary analysis. An inductive scheme documented the reasons behind CS teams’ decision-making throughout their secondary analysis. A coding scheme from the evaluation literature was also used (Alkin and King 2016; Bundi, Frey, and Witmer 2021). The four-part scheme with CS-based examples is described fully in Davis et al. (2022). For the purposes of this analysis, data related to two codes were examined in greater detail for five projects and in the context of secondary analysis. For a full list of codes and definitions, see the supplemental documents.

Interview data were coded and analyzed using NVivo 12 with consensus coding. Two researchers coded each document independently and then argued to consensus on disagreements. The themes from the interviews are presented below.

What kinds of information about volunteer skills are “hiding in plain sight” in data records?

To begin this work, each team reviewed their data records to identify the specific data points that could serve as indicators of a volunteer’s skill. These were considered the skills that teams could investigate as part of a secondary analysis; teams identified between 5 and 20 skills as part of this process. Table 2 presents the skills, using the language shared by each team. Most of the skills named were related to specific data collection tasks such as navigating to the data collection site or taking measurements. A smaller number were broad measures related to data collection, such as submitting complete data or reporting zero observations. Only one project included data interpretation as a skill that was included in their data records.

Some skills were named by multiple teams, including accuracy of identification tasks, submitting complete data, location accuracy, accurately recording habitat features, the frequency of reporting outliers, the rate at which volunteers confused commonly mistaken objects, data interpretation skills, and species accumulation.

The range in the types of variables listed also hints at the value of this analytic method in that the variables span the entire inquiry process. Some variables relate to verifying the necessary conditions for high-quality data before the collection process begins (e.g., accurate location, dual calibration), while others are related to data collection itself (e.g., identifying land cover, using the key accurately). Examples of skills related to later stages of the inquiry process include submitting complete data and data interpretation.

What kinds of information do CS projects use to make decisions about conducting a secondary analysis of data records?

Once they had identified a full list of skills that could be investigated through secondary analysis, teams prioritized their list using the Skill Ranking Worksheet and through meetings with our research team. Each interview also included details related to the decisions that contributed to each team’s analysis. Two themes were common. The first focused on ways the team hoped to use the results of their analysis. The most common reason for wanting to do (or not do) a secondary analysis was intellectual interest. As exemplified in the quotes provided in Table 3, leaders were interested in learning more about the experiences they were providing for volunteers and the ways volunteers engaged with the support provided by the project. Four teams each made decisions based on the potential to adaptively manage their program and to guide future evaluation efforts. In these cases, teams expected the results to provide data that could be used to improve the training provided to volunteers and ways to study volunteers and their contributions to the project. Three teams made decisions based on the potential to affect the CS field. In these cases, teams chose an analysis that they believed would have broad applicability across their sector of CS (e.g., water quality monitoring projects).

The second focal point for decision-making was whether data were in a format that was easy to access and use. Decisions were more often based on the data that were available, though all five teams narrowed the focus of their analysis by discussing both the data they could and could not access in meaningful ways in their existing data records.

In the examples below, data were available but the effort required to make those data usable for secondary analysis was considered beyond the scope of the current project.

It would be a combination of [using the online feature], data mining, as well as linking proficient users who talk about images in a specific way, and link it up with their particular evolution of how well they classify over time along with…the way they talk about the images evolve over time…Theoretically, that data quote-unquote exists, but it doesn’t exist in a way that we’d have the time to study or dig into it.

We have all of the datasheets archived, and all the images archived…We’ve talked about…going backwards in time and putting accuracy in [older records where it is missing]. But that literally requires somebody going back to the datasheets and then looking to see what’s in the database, and then putting in the accuracy scores…Yeah, it would be hundreds and hundreds of hours.

An important factor in these decisions was the time required to accomplish secondary analysis tasks. The importance of time in determining evaluation effort is consistent with the prior literature on CS evaluation (Stylinski et al. 2020). All partners had to balance their interests with the time and budget available to prepare for and conduct the analysis.

How can conducting secondary analyses of data records answer questions about volunteer skills to inform CS projects and the field?

Table 4 presents the questions investigated by CS teams and a summary of their results. Despite the range of skills listed, all teams chose to focus their secondary analysis on a topic related to accurate data collection. Even with this shared focus, the range of skills that teams selected remained broad: (1) accurate site navigation, (2) submitting complete data as a measure of accuracy, (3) relationship between measurement context and accuracy, (4) accumulated knowledge of volunteers based on the number of identified species, and (5) online user behavior in response to an inaccurate image classification.

Table 4 summarizes the evaluation questions that were answered by each team’s secondary analysis, as well as the overall results for each CS project. As shown in the italicized text, all CS teams included at least one independent variable in their secondary analysis to explore whether and how variability in program or data collection context related to volunteers’ skill. For example, McNeill and Vastine (2019) shared a correlation analysis that explored the relation between ALLARM staff support and volunteers submitting complete data. Jackson et al. (2020) used a combination of nonparametric tests and mixed-effects logistic regression to understand volunteer engagement with different Gravity Spy resources. Simonis and Pattengill-Semmens (unpublished manuscript) used generalized additive models to explore the growth in volunteers’ ability to identify new species over time and in relation to individual versus group-based data collection contexts. Existing data records were used to support a wide range of analyses that included correlation analysis, analysis of variance (ANOVA), general linear models, generalized additive models, and sequence analyses. The detailed methods and results of this research is beyond the scope of the current study, and can be found instead in the work cited above.

As noted, we expanded on evaluation use findings in Davis et al. (2022) to examine in more detail how CS teams used the results from their secondary analysis. At the time of their final interview, four of five teams had used their results to make changes to their CS project, and all five teams had shared their results in at least one way to inform the CS field (see Table 5).

Four of five projects made programmatic changes to volunteer training. In multiple cases, for example, the results revealed that volunteers needed additional content or skill training on a specific topic; three projects responded by adding content to their initial training materials, two updated their website or online support systems, and two projects integrated additional training content into their follow-up communication and training for volunteers.

Three teams each used the results from their secondary analysis to make changes to their data collection procedures. In all cases, these teams applied what they had learned from their results to design the data collection procedures for similar, new CS projects that were being initiated. Three teams also used the results from their analysis to change their data validation processes (e.g., to ensure that volunteers were collecting field data at the intended location). Regarding volunteer management, two teams realized that their project would benefit from collecting additional information about their volunteers; both teams added new registration processes to collect this information so that it could inform their work moving forward.

All teams used the results to attempt to persuade others (e.g., to gain board member support for project expansion, to retain current and obtain new funding, and to recruit new collaborators). Three teams shared their results through conference presentations or publications to inform the field (Jackson et al. 2020; McNeill and Vastine 2019; Simonis and Pattengill-Semmens, unpublished manuscript). Two teams shared the results of their secondary analysis with volunteers to provide feedback and be transparent about their secondary analysis process.

What were the challenges in conducting secondary analyses of data records to understand more about CS projects and volunteers?

Throughout the process of planning for and conducting secondary analyses, we and our CS partner teams reflected on the expected and unexpected challenges of this work that might be used by others to decide whether to use this approach within the context of their own project. Identifying skills was one such challenge. A number of skills were eliminated from consideration during the baseline conversations. See Table 6 for a full list, by project. A total of 84 potential skills were named in the baseline interview discussions, and 34 were eliminated. Twelve were eliminated as independent variables related to the project, rather than dependent skill variables. Some of these focused on project engagement (e.g., when and how volunteers were trained, the frequency of project participation), whereas others focused on the data collection context (e.g., whether volunteers used available project supports or characteristics of the physical environment where data were collected).

An additional 22 skills, half of which were considered vital to successful data collection, were eliminated because they were not captured in existing data records. These included contextual factors that affected data collection (e.g., whether and how visual feedback was used during an observation, whether group members helped with an identification), and a volunteer’s ability to detect the organisms or habitats of interest to the project (e.g., level of effort given to search for organisms to record). Data entry skills and recording zero observations were additional skills that were of interest but that could not be verified using data records.

The baseline conversations also illuminated a limitation of using secondary analysis of data records to investigate skill gains. Some variables were eliminated because data were not stored to track mistakes as evidence of learning. For example, online data entry forms use several strategies to help prevent common errors and typically record the final response submitted without recording the foregoing occurrence or types of mistakes. Similarly, some projects overwrite volunteer data as part of the validation process to correct for errors. It is logical that the “correct” answer would be the only one recorded, as not all projects require an audit trail for validation. However, recording only the final answer eliminates the potential traces of learning in action. These competing priorities are described best by our project partners:

They’re too smart [the data collection filters]. They let people know when they’ve made mistakes [but] we don’t get to see our volunteers self-correct. We don’t get to see them, over time, refine the technique.

In terms of database design and the way that we track information, there’s kind of two goals that don’t really easily converge and that’s tracking and preserving what participants do so that we can evaluate it and having the very best possible data set. Without having a complete duplicate—where you maintain the original and then you also have the version that we’ve finessed over time because we know where errors happened and we’re making improvements—we don’t have a perfect record.

The results above demonstrate that identifying skills is not a straightforward process. Preparing participant engagement data was also a challenge for four of the five teams. In three cases, these data were available to teams but not in a format that was easy to access and use. Volunteer management data were not stored in a database for two teams and thus had to be entered and merged with the scientific data records prior to analysis. In another example, online user behavior data were captured, but not in a format that was immediately useful to the analyst; this team spent time querying the database to isolate the data needed. When reflecting on preparing volunteer management data, these teams shared the following:

So the easy side was compiling the volunteer participation, I would say, so training days, do they follow-up in a follow-up meeting, or conference calls, and [with] the quality control data. That was really easy to pull together. And then the thing that is really time-consuming…is taking [our data records] and then trying to bring that into a format where that all lines up.

We had quite a bit of work to do, kind of going back and reconstructing some of the stuff that we needed on the volunteer end of things. That might be one thing that was kind of harder than we anticipated. We’ve done a really nice job of keeping track of the stuff that we’re doing for data quality control. But these other measures of how many trainings a volunteer attended, those records were really messy…So we’ve gone back through and reconstructed all that now and so now we have a good spreadsheet with all those things.

Using the Python API to the Google Analytics, I roughly can reproduce the clickstream to some degree for that individual at that time…The biggest issue for me is that the Google API docs is super haphazard, especially for Python. So it’s just kind of a learning curve type of thing.

Four teams experienced challenges related to sample size. Two had too much data and spent time determining the appropriate data subset to use based on two constraints: the time needed to clean the data, and the time it takes to run complex higher-level statistical models. Other teams had the opposite challenge and did not have enough data to conduct multivariate analyses to understand relationships between participant engagement and skill levels. Challenges related to sample size were described as follows:

Probably the hardest [thing] is going to be finding our way around this very giant database and getting the export stuff streamlined so that [the analyst is] getting what they need and nothing that they don’t. And I guess another thing would be for me, making sure that the information about [participation] is in a way that makes sense and is easy to incorporate into the analysis.

I think that sample size sort of prevented us from looking at skill development over time too. We were just looking at whether or not they were good at getting to the sites, not so much did their skills at doing that improve over time, and that was just an issue with not having enough returning volunteers to be able to do that.