Introduction

Citizen science efforts seek to advance science research while promoting science learning among volunteer participants (National Academies of Sciences, Engineering, and Medicine 2018). Both goals depend on well-designed training and resources that prepare volunteers to follow scientific protocols (i.e., standardized methods) for collecting or interpreting data (National Academies of Sciences, Engineering, and Medicine 2018). If this preparatory support is not well aligned with a project’s protocol or volunteers’ strengths and motivation, then volunteers may lack sufficient skill proficiency to complete the work (Brown and Williams 2019) and data may be devalued or discarded (Burgess et al. 2017). If a protocol exceeds the interests or abilities of volunteers, these participants may not complete required tasks or may even drop out of the project (West and Pateman 2016).

These examples point to the need for robust data on volunteer outcomes, which can include volunteer proficiency on targeted science inquiry skills, their science knowledge, and their self-efficacy to make scientific observations (Phillips et al. 2018). Evaluation provides a pathway to acquiring this evidence in ways that are rigorous and can meet the goals and needs of citizen science (Phillips et al. 2018; Stylinski et al. 2020). Results from evaluation can be used to inform design and improve project programming, including recruitment, training, and protocols; to aid in understanding impacts on volunteer outcomes; to validate project successes; and to advance best-practices in the field. For instance, a citizen science leader could use a performance-based evaluation to have volunteers demonstrate their proficiency on science inquiry skills targeted by the project (e.g., identify a species, navigate to a collection site, or estimate counts), which in turn could ensure acquisition of high-quality science data and alignment with science literacy outcomes (Becker-Klein, Peterman, and Stylinski 2016). Despite these benefits, evaluation in citizen science remains limited, especially for volunteer skill proficiency (Bowser et al. 2020; Burgess et al. 2017; Phillips et al. 2018; Stylinski et al. 2020). Cited challenges include lack of time, of expertise, and of supporting resources (Phillips et al. 2018; Stylinski et al. 2020).

Here we present a baseline qualitative study that introduces and illustrates the diverse and important ways that evaluation can be used to support and advance citizen science efforts. We approach this work by directly linking the citizen science field to the extensive literature on evaluation use (described in the next section) and by viewing these linkages through the lens of practitioners (here, practitioners are citizen science project leaders). Specifically, we ask: how do project leaders use findings from the evaluation of their citizen science effort? Our study focuses on leaders from 15 citizen science projects who were deeply involved in developing and implementing evaluation in their individual projects and in using the resulting findings. A long-held assumption in the field of evaluation is that stakeholders who are more involved in the process will be more likely to use evaluation (Daigneault 2014). Additional benefits of stakeholder participation in evaluation include feelings of satisfaction, ownership, and trust and fairness, as well as perception of higher validity and credibility (Froncek and Rohmann 2019). Thus, we describe the context of practitioner-led efforts across citizen science projects, and consider the role that collaboration played in evaluation use in our discussion.

Evaluation Use

There is extensive literature on evaluation use, and active discussion on the definition of this complex concept that includes associated theories, influencing factors, and more (e.g., Alkin and King 2016, Alkin and King 2017, King and Alkin 2019). Patton (2020) recently provided a straightforward and broad definition of evaluation use: “whatever understandings, learnings, actions, changes, attitudes and/or knowledge [that] follow from evaluation findings and/or process” (p. 588). He notes that it varies by context and purposes, and includes intended and unintended uses and users. Drawing from their historical review, Alkin and King (2016) outline four broad types of evaluation use (conceptual, instrumental, symbolic and process). Bundi, Frey, and Widmer (2021) provide a brief summary of this four-part typology. We offer this typology in Table 1 and have coupled each type with an example relevant to citizen science. The first three types of evaluation use are based on the findings of an implemented evaluation. Conceptual use occurs when these findings deepen a project leader’s understanding and shift their perspectives about volunteers and the design of their citizen science efforts. Programmatic use occurs when this new understanding leads to project revisions, such as changes in training topics, online resources, and scientific protocols. Dissemination use occurs when the evaluation findings are shared with stakeholders, including volunteers, funders, staff, and other practitioners. The final type—process use—does not come from the evaluation findings but instead centers on gains associated with participating in the evaluation process itself (Patton 1997). It is characterized by shifts in citizen science project leaders’ knowledge, attitudes, or behaviors about evaluation as a result of participating in the evaluation process.

Although evaluation use is one of the most researched areas in the evaluation literature, only in the last decade have researchers examined the extent of evaluation use in different contexts (e.g., Daigneault 2014; D’Ostie-Racine et al. 2016; Peterman and Gathings 2019; Shaw and Campbell 2014). Our baseline research contributes to this small number, and as far as we know, is the first multi-project exploration of evaluation use in citizen science.

Context

As part of a larger study funded by the National Science Foundation (DRL #1713424) to promote evaluation use in citizen science, our team collaborated with leaders to develop performance-based embedded assessments that could be implemented and used in their respective projects. Embedded assessment can be a form of evaluation. It is well matched to informal learning experiences—like citizen science—because it is seamlessly integrated into the experience and allows participants to demonstrate their knowledge and skills (Peterman et al. 2017). Embedded assessment can include analyzing science data submitted by volunteers or assessing volunteers as they perform a skill required by a citizen science protocol. For this study, one embedded assessment strategy (Secondary Analysis) focused on analyses of existing scientific data collected by volunteers to search for evidence of skill proficiency (Peterman et al. in review), while a second strategy (Shared Measures) produced embedded-assessment measures around common observation skills, such as “notice relevant features,” that could be used by more than one project (Becker-Klein et al. in review).

We had project leaders in both strategy groups actively participate in the evaluation process and associated decision-making. This included developing analysis procedures for existing data or creating evaluation measures; implementing the procedures or measures; analyzing the resulting findings; and using these evaluation findings in their projects. The leaders collaborated with each other and our team as they tackled this work over two to three years. Leaders from five Secondary Analysis projects initially gathered to discuss and reflect on the volunteer science inquiry skills that would be the focus of their separate embedded assessment efforts. They then worked independently to organize and analyze their own existing datasets (i.e., science data submitted by volunteers), meeting multiple times with our team and the other leaders to share their process, hurdles, successes, and final results (Peterman et al. in review). In a similar fashion, we guided the leaders from ten Shared Measures projects in selecting science inquiry skills for the shared embedded-assessment measures (i.e., measures that would be relevant across many projects). These project leaders then collaborated closely with our team and with each other to co-develop several embedded assessments of these skills and to share their plans, successes, hurdles, and final results from these assessments (Becker-Klein et al. in review).

To ease challenges associated with both embedded assessment strategies, we focused on a single broad science inquiry skill—scientific observation. Thus, we selected leaders for both strategy groups who directed citizen science projects that had adult volunteers collect observation-based environmental data. All leaders also expressed an interest in expanding their evaluation efforts. Beyond this, their projects represented some of the diversity in citizen science efforts focused on scientific observations as they differed in the mode of participation (computer- and field-based), geographic scale of participation (local, regional, and nationwide), longevity (founded in 1993 to 2016), number of volunteers (80 to 20,000), and types of data collected (biotic, abiotic, and astronomical data).

Instrument and Procedures

For each project, we conducted and transcribed semi-structured recorded video interviews (30 to 90 minutes), which provided some standardization of the questions while allowing for context-specific variability associated with the exploratory nature of our work (Adams 2015). All interviews were conducted with the project leaders and sometimes with other staff members. For simplicity, we refer to this group as project leaders hereafter. Interviews were conducted midway through the evaluation work (between fall 2018 and winter 2020) and after the project leaders had completed their evaluation work (between fall 2019 and spring 2021). The midpoint interviews asked project leaders about their evaluation work thus far, while the endpoint interviews had them directly reflect on the four types of use in the context of their evaluation findings and their participation in the evaluation. In the endpoint interviews, we concluded by asking if project leaders had anything to add. If they did not mention collaboration with other project leaders in their response, we specifically followed up by asking their thoughts on the value of the collaborative aspect of this work.

Analysis

We analyzed the data using thematic analysis and a deductive coding scheme (Miles, Huberman, and Saldaña 2018) to identify cross-cutting themes aligned with known types of evaluation use. Specifically, two researchers on our team created an initial codebook based on Table 1, and piloted it using a portion of the dataset to more fully describe each code and to provide examples. After a review by the rest of the team, we made refinements and produced the final version (see Supplemental File 1: Evaluation Use Codebook). Given our small sample size, they used consensus coding to analyze the data (Miles et al. 2018), with the two researchers coding each document independently with the NVivo 12 software, and discussing any disagreements until consensus was reached. We also reviewed meeting notes and any artifacts (reports and graphics) provided by the project leaders to help us understand findings associated with their evaluation use. We included any instances in which project leaders described concrete intentions to directly use the evaluation findings or process in the near future. We did not code any project leaders’ comments related to changes already underway before this study began.

Results

In this section, we describe how the project leaders used findings from their embedded assessment, and we have organized these descriptions by the four evaluation use types. We also share their reflections on the importance of collaboration in the evaluation process.

Discussion

In this study, we sought to demonstrate the value of evaluation within the citizen science field. Little has been published on how evaluation processes and findings can support and advance our field, which in turn may limit how practitioners view it. Here, we characterized evaluation use by applying a typology from the evaluation literature to the citizen science field (see Table 1). To ensure this research is relevant to the broader field, we examined evaluation use through a practitioner lens by documenting the work of leaders from 15 citizen science projects who were deeply involved in evaluation development and implementation.

In our study, we found that project leaders used evaluation of their citizen science efforts in different and important ways. They gained new and deeper understanding of their volunteers and programming (conceptual use); made critical changes to their projects (programmatic use); shared their evaluation findings to persuade stakeholders, inform the field, and motivate participants (dissemination use); and expanded their attitudes and actions with regard to evaluation (process use). Many of these evaluation uses centered on guiding development or improvements of a project (also called formative evaluation). Specifically, knowledge gains from evaluation prompted project leaders to change their training, revise their protocols, add resources, and even terminate an unproductive project. By contrast, other evaluation uses sought to determine the efficacy or impact of citizen science projects (also called summative evaluation). Through reports, presentations, and publications, the project leaders shared their findings related to skill proficiency with their volunteers, other staff members, practitioners in other programs, funders, researchers, and evaluators.

We believe evaluation use by these project leaders may increase with time. That is, they may need time to consider additional ways that their new understanding and attitudes about volunteers and programming (conceptual use) and evaluation (process use) can be employed to support, improve, and promote their projects. This extended work could include reflecting on new questions revealed by the evaluation, which could provide further insights for individual efforts and potentially the field overall. For example, in the future, one of the project leaders might come up with a way to build on volunteers’ everyday skills associated with pattern recognition to help them improve their scientific observation skills. Increased evaluation use over time is supported by Shaw and Campbell (2014), who found that process use rose in the months after their project ended. Our findings related to process use are particularly encouraging, as a deeper understanding of evaluation purpose, methods, and results may help address its limited application across the citizen science field.

We believe it is unlikely that we would have seen this level of evaluation use without the facilitation and collaboration that was central to this study. We purposely provided this facilitation in the context of use. We foregrounded evaluation use by stating from the beginning that we wanted the evaluation results to be useful for the project leaders, and we had them regularly reflect on use at our group meetings and during our interviews. For example, in a midway meeting with the Secondary Analysis group, we had the project leaders and their data analysts share what they learned about their participants and their skills after re-analyzing existing data collected by their volunteers, along with how they are using the results thus far. Overall, this framing ensured that application of evaluation remained central as the leaders made decisions about what volunteers’ science inquiry skills to assess and how to assess them.

Additionally, the leaders in our study actively participated in the evaluation of their projects. The project leaders in the Shared Measures group worked very closely to co-develop and reflect on two different embedded assessments that could be used by multiple citizen science projects. The project leaders in the Secondary Analysis group worked more independently on re-analysis of their existing datasets, but they did meet multiple times to ask questions, compare progress and results, and reflect on impacts and applications. Across both groups, most of our project leaders highlighted how these interactions helped them as they articulated the science inquiry skills that were the focus of their embedded assessments, implemented the embedded assessments of these skills, analyzed and interpreted their findings, and considered ways to use these findings.

We speculate that this deep engagement with the process and resulting data likely had a positive impact on their subsequent use of the findings. Collaborating with stakeholders—those who have a stake in the evaluation or its results such as project leaders, implementing staff, funders, and beneficiaries—is frequently discussed in the evaluation literature (Brandon and Fukunaga 2014) and is becoming more prevalent in evaluation practice (Cousins 2020). There is evidence that it promotes different types of evaluation use and particularly process use (Cousins and Chouinard 2012). We are unaware of other work on this topic within the field of citizen science, but this claim has been substantiated within the context of other informal learning collaboration. For example, Peterman and Gathings (2019) found that the EvalFest collaborative evaluation model was effective at promoting the use of evaluation within science festivals. Similarly, a study of the Nanoscale Informal Science Education Network (called NISE Net) found that their collaborative team-based inquiry approach resulted in museum staff who valued and used evaluation more regularly (Bequette et al. 2019). Grack Nelson et al. (2019) described the value of building capacity within and across NISE Net, including a diversity in evaluation approaches, a shared appreciation for evaluation, and the use of evaluation to make data-informed decisions.

The individuals who make up the evaluation team—including type of stakeholders (e.g., directors, staff, funders, beneficences)—can influence the use of the resulting findings (Brandon and Fukunaga 2014). The project leaders in our study were well positioned to use their evaluation because all were intimately involved in their citizen science endeavors, and they could influence associated decisions. That is not to say that they did not face challenges associated with evaluation use, such as limited ability to reprogram online data collection systems and insufficient funds to make needed training improvements. The relationship between stakeholders and evaluators is also critical for promoting evaluation use (Cousins and Chouinard 2012), and depends on the evaluator’s technical skills and ability to interact effectively with project staff and other stakeholders (King and Stevahn 2012).

Conclusion

In discussing evaluation use, Patton (2020) encouraged “help[ing] people in the situation pay attention to their use options, if that seems appropriate and useful” (p. 588). Helping citizen science leaders more fully understand the important and diverse ways that evaluation can support individual projects and the larger field could increase the prevalence of evaluation in citizen science (Schaefer et al. 2021). We offer four recommendations to advance this work. First, we recommend project leaders use the typology described in this research to understand the different ways that evaluation findings can meet their project goals. Second, we encourage project leaders to be actively involved in evaluation processes and to work collaboratively with evaluators. This aligns with Stevahn and King (2016) who offered steps that informal learning evaluators can take to foster evaluation use, including building strong personal relationships with stakeholders that center on using evaluation results, providing structured interactions to help stakeholders interpret and apply results, and allowing the time needed to do this kind of deep thinking and collaboration. Third, we recommend project leaders also collaborate with each other on evaluation efforts to capitalize on shared resources and expertise around evaluation capacity and use. As shown by Grack Nelson et al. (2019) for the NISE NET community, such efforts might be particularly well suited for citizen science projects that are already part of existing networks (e.g., those associated with water quality monitoring or using shared resource such as CitSci.org). Finally, project leaders’ emphasis on the importance of collaboration was an unanticipated outcome of our study, and it needs to be explored more systematically. Thus, we recommend future studies that more deeply explore collaborative evaluation within citizen science, such as the extent of stakeholder involvement and strategies to support collaborative evaluation within the opportunities and bounds of citizen science.

Data Accessibility Statement

The codebook with examples from the dataset are provided in the Appendix. To maintain anonymity and confidentiality of study participants, data from this study cannot be made publicly accessible.

Supplementary File

The supplementary file for this article can be found as follows: