Broadly applicable shared measures

Shared measures are defined as “rigorous measures that can be shared, or applied, across programs that are addressing the same construct or outcome” (Grack Nelson et al. 2019, p. 60). The literature on the importance of shared measures provides a strong rationale for the need for assessments to be broadly applicable, especially in informal learning environments (Grack Nelson et al. 2019; Hussar et al. 2008). Shared measures can take many forms, such as surveys, tests or quizzes, observation protocols, etc. Recently, research and evaluation teams have pioneered the development of survey-type instruments specifically for use in assessing volunteer outcomes within cross-project evaluations of citizen science (e.g., the Developing, Validating and Implementing Situated Evaluation Instruments [DEVISE] scales; Phillips et al. 2014); science interest and science classroom practices (Noam et al. 2017); “activation” of science learning that can bridge formal and informal contexts (Learning Activation Lab 2018); and outcomes for scientists who participate in public engagement activities (Peterman et al. 2017). Grounded in theory and developed using a process to ensure that the tools measure what they are supposed to measure across contexts, these shared measures have tremendous potential to propel evaluation and research about informal learning outcomes among participants in citizen science projects.

Authentic

For an EA to be authentic, it must directly examine participant performance on related tasks, rather than relying on indirect or proxy activities (Wiggins 1990). The learning environment must be considered and mirrored as closely as possible (Ashford-Rowe et al. 2014). For instance, if a project is about observation of bees, then the assessment should be about observing bees and not other species or other types of data collection. Other project considerations should also be taken into consideration, such as the type of observation conducted. For example, a project that asks volunteers to examine photographs should have an assessment involving sample photographs.

Performance-based

A performance-based EA for volunteer science inquiry skills asks volunteers to use or apply a skill rather than reporting on their own abilities or on their self-efficacy with said skills (Fu et al. 2019). Both formal and informal science education have seen recent calls encouraging researchers and evaluators to begin using performance as a key metric of skill (National Research Council 2009; National Research Council 2010; National Research Council 2015). Participants need to demonstrate that they can perform a skill by engaging in a task or set of tasks that requires them to apply the identified skill (Ashford-Rowe et al. 2014).

Integrated

Finally, integrated tasks are incorporated into a project’s specific activities. This could be incorporated into a project’s training or could be part of regular data collection or submission of data. Integrating assessments into the curriculum or instruction process is a key component of embedding assessments (Wilson and Sloane 2000), and it is widely recognized to be challenging to accomplish (Sloane et al. 1996). Yet, there has been a call in the field for “more direct and less obtrusive measures,” (Fu, Kannan, and Shavelson 2019); this is one of the primary advantages of EAs—they can be integrated into the programmatic experience without adding on to the assessment burden for volunteers.

Here, we describe the processes used, and opportunities and challenges encountered, in creating shared embedded assessments of CS volunteer skills (within ten different CS projects) that aim to meet the criteria of broadly applicable, authentic, performance-based, and integrated. This work was part of a larger study funded by the National Science Foundation (DRL #1713424).

To further understand the success of this development process for creating new methods to assess volunteer observation skills, we developed the following primary question to guide the inquiry:

To what extent do these embedded assessment tools meet our criteria for shared embedded assessments of skills: (1) broadly applicable, (2) performance-based, (3) authentic, and (4) integrated?

Shared embedded assessment development process applied to 10 citizen science projects

Staff from ten CS projects (see Table 1) were partners in developing, implementing, and revising the shared EAs developed over the course of three years. Two of these project leaders were identified through our initial work on a prior grant (Stylinski et al. 2020) in which we created a set of customized EAs. We continued to recruit project leaders through additional inquiries with known relevant CS investigations, as well as through a snowball sampling method. We intentionally limited our search to focus on a single discipline within CS (environmental science) for which nature observations are an important skill. We believe that this focus on environmental science CS projects represents a substantial portion of the CS field—a portion broad enough to capture important diversity, yet narrow enough to allow for the development and testing of new EA tools customized to a subset of science observation skills. We interviewed several project leaders in the fall of 2018, asking questions about what a typical volunteer does in their project, what science inquiry skills volunteers learn and practice, and whether the project provides training for those skills. On the basis of the interview results, project leader interest, and selection for heterogeneity in several areas (i.e., different species/phenomena of interest, online and in person, and different research topics), we ended up with 10 projects. The ten CS projects chosen intentionally included a diverse array of research topics and identification foci, as illustrated in Table 1.

These 10 CS projects worked with our team over the course of three years to participate in and provide feedback to create, test, and implement two shared EAs. The current study is based off of our previous work on a National Science Foundation grant (DRL #1422099) to test and articulate an EA development process that was customized to specific projects (described in detail in Peterman et al. 2017). In contrast, this current study focuses on a Shared EA Development Process in which the leadership team (consisting of the authors on this paper) collaborated with project leaders to guide them through the process of creating EAs to be used across multiple projects (Figure 1). The work was carried out in three distinct phases.

Stage 1 focused on working with the projects to determine what skill to measure, and consisted of (1) collaboratively identifying common science inquiry skills, and (2) articulating each project’s goals and activities that align with those skills. An in-person meeting initiated the collaborative process with project leaders, followed by a series of group conversations identifying both the skills to focus on and existing activities that featured those skills and potentially demonstrated skill proficiency or development.

The leadership team took the information from this meeting and follow-up conversations to draft two potential EAs: one for the skill of notice relevant features and one for the skill of record standard observations, both of which are a part of the larger task of scientific observation and thus, science inquiry. Our work builds on that of Eberbach and Crowley (2009), who defined characteristics that distinguish everyday observations from scientific observations. In the case of the EA for notice relevant features, an observer must be able to match what they see (e.g., floral structure) with their content knowledge (e.g., plant families). For this EA, volunteers are asked to identify organisms in different photos (relevant to a specific project), and are prompted to “show their work” with the question, “what features of the animal did you use in your identification?” The EA for record standard observations is related to what was described in Eberbach and Crowley’s (2009, p. 56) framework as “Record observations using established disciplinary procedures and representations.” In general, we refer to “standard observations” as those that provide a consistent or uniform set of measurements to describe a phenomenon or event. For this EA, project leaders created a three- to five-minute video to simulate the perspective of the data collector, and then volunteers were asked to watch the video and fill in a modified data sheet as if recording the data observed in the video.

Each project leader chose the EA for the skill most relevant to their project’s activities, resulting in five projects choosing each EA. Project leaders iteratively provided input and feedback on the draft EA, getting it ready for Stage 2 of the process.

Stage 2 focused on gathering evidence and making revisions to ensure the draft EAs were collecting trustworthy data about the skills they hoped to be measuring. There are various methods to check that measures are gathering reliable and valid data, but this project focused on two methods that would be most useful for the EAs developed: an expert review process and think-aloud interviews (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education 2014).

Expert reviewers examined whether the tasks on each EA were sufficiently addressing the skill area of interest. The two draft EA measures were reviewed by four advisors with expertise in measurement, evaluation, scientific inquiry, and CS, using an adapted version of Newman, Lim, and Pineda’s (2013) expert review process. For each EA task, experts were asked to provide feedback on how well that task aligned with the project’s definition of that skill area, and if needed, provide suggestions for how the task might be revised to better align with the skill. The expert review results were then used to revise the EA tasks so they more thoroughly and accurately addressed the skill areas.

Think-aloud interviews were then conducted with volunteers from two different citizen science projects (one project for each draft EA). To recruit participants, project leaders emailed volunteers inviting them to participate in a think-aloud interview. For the notice relevant features EA, sixteen volunteers signed up, and from these, 10 were selected to ensure variability across gender identity, age, and self-reported skill level of identifying animals. Eight volunteers expressed interest in testing the record standard observations EA, all of whom were included in the think-aloud sample. The purpose of the think-aloud interviews was to help uncover any confusing or misinterpreted tasks. During the think-aloud interviews, CS volunteers described their thinking processes out loud as they responded to each task in the assessment (Beatty and Willis 2007). This process helped to uncover if EA tasks were interpreted, and thus completed, as intended; if not, further revisions were made in preparation for pilot testing.

In Stage 3 of the process, the projects were asked to pilot the revised EA tools with a few participants first, and then later implement a broader field test within their projects. The leadership team worked with CS project leaders to customize and finalize the EA for their project and to determine how to integrate it into project activities (for both skills, EAs ended up integrated into the project training for most projects). At the end of this stage, the leadership team facilitated another meeting to examine findings for each project and to reflect on the process.

In the first round of applying the shared EA development process, five different CS projects worked together in small groups to each create an EA for a particular skill. These two small groups, consisting of five projects each with different content areas and formats, were each able to develop an EA for notice relevant features and for record standard observations. All 10 CS projects piloted some version of at least one of the EAs.

Once the first round of EAs were developed, the leadership team facilitated a second round of EAs, and three projects that had not done so before chose to try out the EA developed for notice relevant features. In the scope of this study, the EA for record standard observations was not tried in the second round by any of the CS projects.

Example of notice relevant features embedded assessment

One of the partner projects that co-developed the EA for notice relevant features asks volunteers to identify animals seen in photos. The identified features include picking out the animal from its background as well as recognizing particular features of a species, such as the color of the ears (i.e., for a red fox) or a stripe on the back (i.e., for a grey fox). The EA includes different photos taken from a camera trap, and participants are asked to determine if there are indeed animals in the photo, and then identify the animal(s) as specifically as possible. For instance, one photo is a red fox (see Figure 2), which volunteers would be able to identify if they knew the distinguishing characteristics of black-tipped ears, white-tipped tail, and black “boots.” Via an online questionnaire, volunteers answered the questions, “Please identify this animal as specifically as possible,” and “what features of the animal did you use for your identification?”

Example of record standard observations embedded assessment

One of the projects that co-developed the EA for record standard observations produced a video simulating the data collection process for identification of butterflies. The video, which is just under six minutes, offers a first-person perspective, and starts with a sweeping view of a verdant meadow full of flowers, cut through by a path. The person taking the video walks through the meadow, panning from left to right and back again as they “look around.” When the species of interest is spotted, the camera focuses on it. The intent of the video is that the project volunteer watching should feel as if they themselves are walking through the meadow; the volunteer records any species of interest that they see on a datasheet provided, which is then scored by project leaders to determine whether the volunteer correctly identified the family or species shown in the video. In one part of the video, a distractor (an example of the protocol not being followed correctly) is introduced as the camera pans behind the person walking (volunteers are only supposed to record data they see in front and to the sides of them).

Data collection

To understand the efficacy of the EAs during testing and implementation, we interviewed the project leaders at two different points. The first interview was conducted with each project in December of 2020 (immediately after Stage 2 of the Shared EA Development process) to gather perspectives about the implementation of their EAs. Specifically, we asked project leaders to consider which parts of the EA development process they considered to be most valuable, and which were most challenging, and whether they had any specific difficulties or “aha moments.” We also asked the leaders to consider our ideal EA and which components (i.e., broadly applicable, authentic, performance-based, and integrated) were met adequately by the two EAs developed. A second interview was conducted in spring 2021 (at the conclusion of Stage 3 of the Shared EA Development process) to see how project leaders used the findings from the EAs, but data from this interview were not relevant for this study. All interviews lasted between 30 and 60 minutes and were recorded and later transcribed verbatim.

Notes were kept from meetings that occurred in Stages 1, 2, and 3 for any relevant project feedback. For instance, in the final meeting (after Stage 3), project leaders were asked to rate how well they thought the EAs had met each of the criteria (i.e., authentic, performance-based, and integrated), and discussion ensued.

This research was approved by the University of Maryland Institutional Review Board (IRB #1072528). Informed consent was provided by all volunteers who participated in interviews and meetings.

Data analysis

Coding schemes to analyze the project leader interview questions were developed by a team of three researchers (see Supplemental File 1: Sample Interview Questions and Codebook), using six steps of collaborative qualitative analysis (Richards and Hemphill 2018). First, individual researchers reviewed different interview transcripts to conduct open coding and to create memos to guide the development of codes. The team met regularly throughout this process to discuss individual findings, and to work together to develop a preliminary codebook. The two codes relevant for the project leader interviews were about the process of developing EAs (i.e., mention of the value of meeting with other project leaders) and the EA components (i.e., discussion of the four components of an ideal EA). The team then tried to apply the preliminary codes to separate transcripts and met again to review and refine the codes. Two coders then pilot tested the codebook, independently coding the same transcripts, and final adjustments were made to the codebook on the basis of those experiences. Once the codebook was finalized, consensus coding was conducted on all data records. Two researchers coded each interview independently, and then compared codes; all disagreements were discussed, and the final code was agreed upon. Interview data were coded and analyzed using NVivo 12.

To what extent are these tools considered broadly applicable?

One way to determine whether an EA is broadly applicable is to consider the application of the tool to a project for which it was not originally developed. In this case, each of the three projects that used the EA notice relevant features in the second round of EA development found it relatively straightforward to take the instructions developed initially and create their own version of the EA customized for their particular project. One project was able to copy the survey developed in the first round from a different CS project, and change the photos and species question slightly to use the EA for their own project. So, instead of asking to identify “which animal” a volunteer saw, they were asked “which butterfly” they saw, but the rest of the survey remained the same.

When asked how they felt about creating EAs that could apply across projects, most project leaders (80%) thought that the two EAs developed, and especially the format or process of these assessments, were indeed applicable to projects beyond their own, as exemplified by the following two quotes.

I think the video was broadly applicable, the technique…obviously [the particular species] are very specific, but I think the video technique can be really workable for many projects.

I think the notice relevant features assessment is broadly applicable. I feel like [projects could] easily change out photos, change out the answers for the features and then it would apply to a bunch of different programs.

Some project leaders noted challenges in creating an assessment in that in order to apply across projects, there had to be customization of EAs to individual projects. One said,

…it seemed like things could be developed that worked in really interesting and helpful ways for individual projects, I’m less convinced that this is something that can be done across projects…there could be some styles or approaches that are generalizable, but I think they’re going to have to be pretty heavily customized for each project.

To what extent are these tools considered authentic?

Challenges in the authenticity of the EAs were raised in the validation process, and project leaders agreed that it was quite a challenge to use videos as a format to mimic real-life data collection, questioning the authenticity of the EA. Ensuring that videos showed enough detail and paused on the identified species long enough for someone to be able to adequately see the details of the species or phenomenon was an obstacle. These considerations led project leaders to retake their videos to pay attention to how a viewer might see the images.

Some project leaders noted other ways that the video did not mimic data collection in the real world, such as noting that the view seen through the lens of a camera is not the same view as the eye can see, with its peripheral vision capabilities. One said,

I loved the video, but it was not like looking directly at the [species] because it is a much more narrow view.

Project leaders also noted that when participants collect data, they can turn their heads and bodies in response to cues, but when watching a video, viewers are constrained by the scope and direction of the videographer’s lens.There were even a few complaints of jarring or dizzying scenery due to video camera instability in one of the videos that involves walking while collecting data.

Even with the challenges identified above, almost all (80%) of project leaders stated that they thought the EAs developed were indeed authentic to their projects. Some project leaders mentioned,

I think they are authentic since we used actual photos that people had submitted…I don’t know if you could get more authentic than asking them while they are submitting [their data].

I know these assessments are very authentic to my project, because they really fit in seamlessly with what I wanted people to do, so I would consider that very successful.

It certainly felt authentic. [The assessment] was dealing with a real problem and using real people and real data, and so the authenticity was there.

The projects that felt the assessments were less authentic to their project talked about how either the level of analysis was not quite right or the assessment may have not been quite authentic to project activities. For instance, two people said,

At points, it felt like we were getting down to a level of detail that almost seemed unrelated [to the project’s training]…I think [the assessment] gets to participant learning rather than just from the content side [of what our trainers do with participants].

I thought we did a pretty good job on all of the [EA components], but there was some question as to the authenticity of the mode.

To what extent were these tools considered performance based?

Project leaders were in consensus (100%) that the embedded assessments were performance based. This finding was intriguing to the research team leaders, because there was some question about whether the notice relevant features EA, which was an online questionnaire, could be classified as performance based. However, project leaders felt that since participants were being asked to do a task, rather than self-report on their ability to do that task, it did indeed qualify as a performance-based assessment. For instance, some people said,

[Participants] were being asked to identify features and demonstrate their skills as far as being able to figure out what are relevant features, so I think [the assessments] seem clearly performance based to me.

I think they’re performance based, yeah, [participants are being asked to] follow instructions and identify stuff.

In addition, some project leaders appreciated that these assessments could be used in their training as a way both to gauge skill level and to give participants a chance to practice the skill. One person asserted,

…I think what’s nice is that this is something that I feel like we could refresh and continue to utilize both as a training reinforcement as well as an opportunity for us to assess [their skills] on the back end.

To what extent were tools considered integrated?

The component that was least endorsed as being successfully achieved by project leaders (50%) was integration. Several reasons for lower ratings on the integrated component were cited, such as: the assessments were not placed comprehensively into project activities and platforms (they were primarily used as a training exercise), the assessments were not used for all project participants, and the assessments generally missed the mark. Some of the interviewees asserted,

Sending people a video doesn’t seem like bad advice, it’s just like testing them, which is fine, but I always struggled with this notion of it being integrated, that was always the tricky part for me.

I hesitate to say we missed the mark, I just don’t know that there’s another good way to do that…but I think that unless we’re going to do virtual reality or something like that, a video [is not that integrated].

Part of [our project] is this fairly complex, integrated platform of software, and I didn’t have a way to integrate the assessments into that platform.

Yet, other project leaders felt that including the assessment into the training was a way of integrating it seamlessly into project activities, as exemplified in the following quote:

You didn’t feel like you were asking them to do something extra, but you could make it make sense as part of the training…there was a natural place, it didn’t feel forced.