Student learning outcomes in citizen science and authentic science investigations

As noted above, the citizen science research community has explored a variety of learning and attitudinal outcomes, while GLOBE also considers the formal learning perspective. As a preliminary step to measuring learning outcomes, in this study we seek to explore and identify a potential range of them.

Bonney et al.’s (2009) model identified potential outcomes and indicators in several categories, including awareness/knowledge/understanding, engagement or interest, skills, attitudes, and behaviors. Phillips et al. (2018) extended this argument into a framework that translates outcomes developed from the informal science learning literature into a range of citizen science–specific learning outcomes. These include i) interest in science and the environment; ii) self-efficacy for science and the environment; iii) motivation for science and the environment; iv) knowledge of the nature of science; v) skills of science inquiry; and vi) behavior and stewardship. GLOBE learning outcomes may also depend on students’ interest and motivation, which influence the topic they choose to explore. Topic selection is also influenced by established GLOBE protocols and activities. As in Chase and Levine’s (2016) framework, student motivation and topic selection may be driven by biophysical or environmental characteristics, student awareness of opportunities to pursue the topic within GLOBE, current or former monitoring, and sociocultural aspects such as the opportunity to participate in a broader GLOBE campaign with other youth.

Much of the literature has focused on adult and informal learning within citizen science (e.g., Aristeidou and Herodotou 2020; Bonney et. al. 2009; Chase and Levine 2016; National Academies 2018; Phillips et al. 2018), or student learning from participation in science fairs or inquiry instruction rather than student learning from a citizen science investigational context (e.g., DeLisi et al. 2020; Houseal, Abd-El-Khalick, and Destafano 2014; Craven and Hogan 2008). Aristeidou and Herodotou (2020) conducted a systematic review of the effects of citizen science on learning and literacy outcomes, and noted a lack of studies of citizen science in formal settings. This study begins to bridge that gap and to consider what features of citizen science investigations may be aligned with learning outcomes.

A National Academies report on Learning Through Citizen Science (2018) proposes a model in which citizen science provides experiential opportunities aligned to what we already know about human learning: engagement with other people, constructing knowledge through activities, developing appropriate content knowledge, interacting with data, and using both the tools and the practices of science and inquiry. They further incorporate motivation and interest (Geoghegan et al. 2016; Frensley et al. 2017) as key contextual factors that mediate possibilities for learning. This report also mentions interest in the potential for citizen science to develop disciplinary knowledge but a lack of evidence for such outcomes. A synthesis paper by Phillips et al. (2019) similarly observes a gap in the field surrounding the goal of many citizen science projects to develop understanding of the nature of science, with only weak evidence for achieving this.

Inquiry instruction and student-centered authentic science activities within formal settings provide the best analog to the GLOBE citizen science program. DeLisi et al. (2020) find that student learning as a result of science fair participation is most associated with opportunities to participate in evaluating and critiquing findings of projects. These authors argue that science fairs, like GLOBE or other citizen science opportunities accessible to youth, are an unusual opportunity for students to take ownership over a complete scientific investigation and to think critically about their own investigative work. Houseal, Abd-El-Khalick, and Destefano (2014) established student learning gains in a formal teacher-student-scientist partnership initiative and found that giving students agency in their research investigations and providing true authenticity can impact learning.

This echoes the broader perspective of Minner, Levy, and Century (2010), who conducted a research synthesis of K–12 inquiry instruction across studies from 1984 to 2002, and found that emphasizing active thinking, assigning students responsibility for learning, or drawing conclusions from data can positively influence student learning. Craven and Hogan (2008) summarize experience with science fairs similarly to the citizen science community’s findings on citizen science outcomes: The nature of investigations, and the extent to which learners are engaged in real-life issues at the nexus of science content and personal/societal motivations, differentiate high-quality experiences from rote ones.

Characteristics of the data

113 entries into the IVSS and 110 entries into the SRS events were considered for inclusion. Ultimately, after excluding projects lacking a report or poster and those projects that were illegible (such as blurry poster photographs), a total of 207 projects were carefully examined. Of these, 97 (47%) were drawn from IVSS, 96 (46%) from SRS, and an additional 14 (7%) had been submitted to both opportunities.

Age/grade band information was available for only 203 of the projects (Table 1), using GLOBE’s international definitions aligned to approximate age groups. Because only one undergraduate project was submitted, that group has been removed from most analyses. In part because SRS events took place within the United States, the majority of projects were completed by US students.

Ethics and consent

The study and all protocols were reviewed by the Oak Ridge Site–wide Institutional Review Board and were determined not to constitute human subjects research because of the exclusive use of archival data. The data and metadata used for analyzing the IVSS reports were publicly available on the GLOBE website. There was no interaction with the students. For the SRS reports, photographs of student research posters were taken by members of the GLOBE team at in-person SRS events under a media release signed by parents/guardians.

In both cases, the project materials do contain personally identifying information. Materials were stored on an encrypted system at ORAU. Beyond IRB requirements, the researchers were committed to ensuring that no students would recognize their own projects in the research, including this manuscript.

Literature review to identify relevant frameworks for coding student investigations

The study began with a thorough review of relevant literature in citizen science and in public participation in scientific research. From an initial list of prominent papers, the authors followed citations to approximately 20 reports and journal articles. The authors created an annotated bibliography and an initial list of codes, representing GLOBE-relevant characteristics of participant investigations, motivations, experiences, and outcomes in citizen science (Table 2). The codes on this initial list were drawn from the various frameworks and as a result in many cases overlapped. The authors identified the most useful constructs for characterizing and organizing citizen science projects within the GLOBE context.

Closely related or overlapping codes were combined, and the entire set was organized loosely into related dimensions and themes. Throughout the code refinement process, labels were used to indicate the initial source of the code, which remained traceable throughout the study; these source labels remain attached to each code listed in Supplemental File 1: Frequency of Individual Code Occurrences by Project, by Grade Band, and by Country.

Development and refinement of codes

The main coder (author Martin) started with the initial list of codes pulled from the literature, already organized into top-level dimensions and secondary themes, and reviewed them, which resulted in a few changes. The coder tested this initial list on a randomly-selected subset of 10 IVSS and 10 SRS projects, revised the list, and sent the codes for final review by authors Malmberg, Chambers, and Czajkowski. This final list of codes was then applied to a new random sample of 10 IVSS and 10 SRS projects for final testing before moving forward. The final list of codes, organized by dimension and theme, are included (along with findings) in Supplemental File 1.

Two types of codes require special discussion. The first category are codes and characteristics that are specific to the GLOBE context. Because some students chose to submit a report or poster that was not completed within the GLOBE program, codes were created to identify whether each project was a GLOBE project. Projects were also assigned to GLOBE “spheres” or topic areas (hydrosphere, biosphere, atmosphere, and soil), and tagged with the type of technology used to collect the data, because this is of special interest to GLOBE. Finally, we added codes related to GLOBE’s virtual badges, indicating project elements like interaction with a scientist or GLOBE students at another school.

Another category of codes, which were of particular interest to the authors, are drawn from Kastens (2014a, b). These brief concept papers outline a framework describing youths’ progression as they develop the skills and habits of mind of scientists who work with data. We adapted Kastens’ framework to differentiate between student projects that display non-mediated knowledge (data as collected by students in raw format, or represented in simple tables), simply mediated knowledge (data translated to a representation or model), and more sophisticated demonstrations of mediation of knowledge (data translated to a representation or model with benchmarks or other interpretive elements to guide understanding).

Coding the full dataset and interrater reliability

Once the codes were fully validated and finalized, the coder then applied them to the entire set of projects, including re-coding the randomly-selected projects in the testing phase. Each project was carefully examined and coded by a single evaluator who assessed whether each code was or was not present. The evaluator sought specific evidence for the characteristic in the artifacts. Thus, the question answered by coding was never whether the students completed a given task or learned a particular skill, but whether their research artifact provided a demonstration or evidence of a given facet. The absence of evidence in these artifacts is not evidence of absence in the students’ experiences or their learning. While this is a potential weakness of the study, it is also reflective of the SRS and IVSS, in which artifacts are incomplete representations of the citizen science experience.

Notably, some codes are independent of others and could be selected regardless of other characteristics, such as the codes related to the topic. As many such codes as were applicable were selected. In other cases, though, the coder would likely only select one from a set of codes. For example, a given investigation would be coded as either a planned investigation or an unplanned/open-ended investigation, but not as both. In other areas, the coder strived to select only the most salient option from a set of codes, such as the motivation for the selection of a topic.

To investigate the reliability of coding, a subset of 47 projects were double coded by an independent coder (author Miller-Bains). Unlike the main coder, the independent coder had no prior exposure to GLOBE, and required some training to recognize key elements. Following this interrater reliability sub-study, several adjustments were made to the code list and relevant projects were re-coded.

Based on the results of two phases of reliability coding, estimates of rater agreement were calculated using R Core Team (2021). Details of the interrater reliability study are reported in Supplemental File 2: Tabulation of Interrater Reliability Scores and Prevalence Indices. In the case of some codes, the agreement values did not reach the lower threshold for acceptable reliability. Overall, agreement varied widely across codes, but some of the variation can be attributed to both the dichotomous nature of the coding scheme and the infrequency of certain codes (Eugenio and Glass 2004). Based on this interrater reliability study, the authors advise caution in the interpretation of the codes, particularly for those who are unfamiliar with the GLOBE program.

Quantitative and network analysis

The completed dataset was quantitatively analyzed to study the frequency of each characteristic. These findings were disaggregated by student age/grade band and by country (United States versus Non–United States). The dataset was also analyzed using the tools of social network analysis to identify and cluster characteristics that frequently appeared together (e.g., semantic network analysis; Doerfel 1998) using the open-source Gephi social network analysis software (Bastian, Heymann, and Jacomy 2009).

A visual inspection of the semantic network did not suggest any clear organization (Prell 2012). The Blondel et al. (2008) method for identifying clusters or subcommunities using the modularity metric was calculated in Gephi to identify groups of co-occurring codes. After reviewing the results of the initial clustering algorithm, some codes were excluded that did not aid in the creation of a framework of student investigation characteristics. For example, the codes related to the topic area (biosphere, soil/pedosphere, hydrosphere, and atmosphere) were independent of the clustering this study sought to examine. Once these codes had been eliminated, the modularity calculation was repeated to identify modularity clusters—sets of codes that often coexisted together in the network, and that were less likely to coexist with codes in any of the other modularity clusters. These clusters, therefore, are indicators of cohesive sets of characteristics of student GLOBE investigations. The evaluator examined each cluster and the codes present within it, revisited the literature sources in Table 2, then conducted so-called second cycle qualitative coding (Saldaña 2015). This phase of theoretical coding identified commonalities among the codes appearing in each cluster, for which the evaluator developed descriptive titles. Theoretical coding further identified similarities and distinctions between each cluster, which suggested ways of grouping and organizing the clusters. The final descriptive framework, presented in the section entitled “Findings,” presents the sub-clusters identified by the algorithm and the larger theory-driven clusters into which these sets were organized by the evaluator.

Caveats and limitations

The most significant limitation to the study is that coding of projects was largely completed by a single rater, albeit one with substantial experience in GLOBE. We attempted to ameliorate this limitation through discussion among the authors, substantial code reviews, and the interrater reliability inquiry. We do suggest caution in drawing conclusions from this limited data. Throughout this process, we noted that knowledge of the GLOBE program was helpful in conducting the study and interpreting the findings. Although our intent is and has been to situate these findings in a context that is of use to the broader citizen science community, GLOBE’s specific prominence as a case study is clear.

Second, the Blondel et al. (2008) modularity algorithm used in the Gephi software for the semantic network analysis is nondeterministic. Each iteration results in somewhat different assignments of codes among clusters, and the evaluator has the ability to make some choice in the algorithm’s settings, allowing for further variation. The sub-clusters identified and presented in the section entitled “Findings” are based on converging solutions that appeared over multiple runs, but the nondeterministic nature of this technique is a limitation of the study. Ultimately, the main-level clusters are not based solely on the modularity metrics, but were assigned by the researchers based on careful examination of the quantitative findings, and theory-driven qualitative analyses of the codes within each modularity cluster.

Finally, the student artifacts used were not the ideal data source for some of the characteristics of interest. For instance, we were not able to assess whether students increased their knowledge or skills, but only whether they demonstrated such an increase by discussing or mentioning it in their materials. Thus, this manuscript is most fairly treated as a study of the artifacts resulting from the investigations rather than a full study of the citizen science experience in GLOBE.

Quantitative prominence of characteristics

The full list of 89 codes is presented in Supplemental File 1, along with the frequency of each code across the projects and disaggregated by student grade band and country. While Supplemental File 1 provides a full accounting of all codes used in this project, Table 3 provides a summarized level of insight into the codes by providing the theme areas and, where applicable, sub-theme areas around which the codes were organized.

In most cases, codes illustrative of more complex scientific thinking and scientific processes became more common as the grade band of the student citizen scientists increased. For instance, while about 4% of projects overall employed unstructured, exploratory observations as the main learning and investigative strategy, this was more common at the lower and upper elementary levels (20% and 11%, respectively) than among secondary school–aged citizen scientists (3%). In other cases, the opposite was true: While 27% of projects overall discussed actionable information of concern in the local community or environment, this occurred in 60% of lower elementary citizen science investigations. However, because more middle and secondary school students participated overall, these disaggregated findings are only descriptive and suggestive. A study including a larger pool of student investigations would be needed to substantiate differences by grade.

Clusters and descriptive framework

The co-occurrences network is displayed in Figure 1, which includes only the codes that were retained in the analysis to identify clusters. The left-hand panel indicates the structure of the network before the modularity clustering analysis was applied. This perspective shows each individual code as a node in the network, with radius and color depth indicating the frequency of each code. The number of linkages emanating from each node indicates how commonly that node co-occurs with others, and the thickness of each link indicates how frequently the two nodes connected by that edge co-occurred. This fundamental semantic network data comprised the input used for the next step, the cluster analysis, the result of which is displayed in the right-hand panel. The right-hand panel displays this same network with each node in the same location, after the application of the modularity algorithm and the final qualitative coding by the evaluator in the final step of analysis. The color-coding in the right-hand panel indicates the main cluster to which each code was ultimately assigned. These clusters along with subclusters are further described in Figure 2.

Figure 2 illustrates the framework derived from IVSS and SRS projects. It represents each of the clusters and sub-clusters; the area of each bubble is based on the prominence of that cluster or sub-cluster among the student projects. Note that each project could belong to multiple clusters, and the framework provides a way to describe projects holistically based on multiple elements, rather than a way to assign student citizen science investigations to a specific cluster.

On the basis of a qualitative examination of the types of codes and characteristics in each identified cluster, and on the literature sources in Table 2, we identified three tiers of student citizen science investigation, from limited to first to second tier, each increasing in the sophistication of practices and methods captured by the codes within that cluster. The second-tier projects included indicators of more sophisticated, complex, or evaluative thought processes among student investigators. The framework also includes two additional major clusters, which stand apart from the scientific or methodological maturity of the investigation. These additional clusters demonstrate student citizen scientists’ thoughtfulness, connection to context, or motivation and self-efficacy to act based on findings. The figure also demonstrates how the clusters related to project content and student focus on impact or action span across those three tiers of projects. The final cluster, displayed at the bottom of Figure 2, includes submitted project materials that were unrelated to GLOBE.

The relative prominence of each cluster across the projects is displayed below in Table 4. Note once again that each project can fall into multiple clusters. Rather than being used as a tool to divide projects into categories, this framework provides a way to describe multiple intersecting elements of projects. For instance, a project with serious flaws in the analysis and interpretation of field data can also demonstrate student engagement around environmental action. The cluster frequencies broken down by grade band and by country are provided in Supplemental File 3: Frequency of Project Alignment with Identified Clusters and Sub-Clusters by Grade Band, and by Country.

Notable individual codes

This section presents a selection of interesting codes and the frequency of their occurrence among the projects, indicating the prominence of selected characteristics among the 2018 investigations (see Supplemental File 1). Since codes are not mutually exclusive, groups of codes in Supplemental File 1 may display totals of less or more than 100%.

Only approximately 23% of GLOBE citizen science investigations had a fully formed and complex hypothesis (predictive, falsifiable, measurable relationship between more than one variable that could drive a complete investigation), whereas 53% of the student investigation had an oversimplified hypothesis or a hypothesis that was not fully formed; for example, such a hypothesis might predict the quantity/measure of a variable or simply propose measurements to explore a variable’s range of values in the local environment. Finally, about 22% of project hypotheses contained a fundamental weakness or error, such as hypotheses that misunderstood the meaning of a key variable, that based a prediction on a misunderstanding of the science content area, or that misunderstood the scope/scale of the hypothesis relative to the scope/scale of the investigation. This suggests that some additional attention may be needed to achieve citizen science goals related to understanding of the scientific process.

Most projects did not include a statistical analysis or included a poor/improper analysis (68%); in these projects, students might read values from a graph or table and visually interpret findings, while other students improperly fit lines or analyzed a different set of variables than those under investigation. It was quite common for students to plot the independent variable against time in one visualization or table, and to plot the dependent variable across time in a second visualization or table, rather than comparing them directly against each other, in cases where the research question was not about change over time. An additional 17% of projects included basic statistical analysis such as descriptive statistics, and 9% used more complex techniques such as inferential statistics and line fits. Similarly, in approximately 49% of projects, students created a basic data representation such as a bar chart. Only 20% of projects demonstrated deeper understanding through data representations, such as projects that interpreted or described a pattern visible in the representation, or compared data values to a benchmark value.

Although only 10% of GLOBE citizen science investigations were thoughtful about placing their smaller-scale project into the context of a major challenge or problem (such as climate change), most students did use their projects to respond to an impact on themselves, their school, or their community (58%). About one-quarter of projects proposed some actionable use of the knowledge or information they gained, while 14% did take action in the field, such as a community presentation or field site cleanup.

Taken together, these findings suggest that the GLOBE program has successfully encouraged citizen science projects to consider a broader perspective and has helped youth citizen scientists consider themselves as part of the environment that they study and the community to which they belong. Further action may be needed to translate this deeper understanding to more robust investigation, analysis, and interpretation. The descriptive framework of clusters, discussed below, provides insight into areas where further work is needed.

Discussion of the clusters and descriptive framework

Phillips et al. (2018) proposed a framework for measuring learning outcomes that does not exactly align to the purpose or the findings of this study, but does reflect the various ways that this study considered engagement and investigation characteristics. Rather than assessing individual outcomes, this GLOBE study considered how citizen scientists’ developmental path along Phillips et al.’s outcomes like self-efficacy and skills of science inquiry would be reflected in the investigations they developed, conducted, and presented. Figure 3 demonstrates some of the alignment between the two frameworks, and the ways in which multiple outcome types could be reflected in various elements of the GLOBE framework.

Similarly, it was interesting to consider co-occurrence of clusters within individual GLOBE student investigations. For instance, in 20 of the 36 projects (56%) that demonstrated manipulation of variables or engineering solutions, student citizen scientists also considered impact and ecological action, suggesting linkages between self-efficacy and action-oriented approaches to citizen science. Of the 15 investigations that indicated a sense of student self-efficacy and relevance of the project to their life, 10 (67%) conducted very simple, basic projects, suggesting that youth’s skills in science inquiry may differ from their understanding of the value and relevance of the nature of science. Further investigation into the skills, identity, interests, motivations, and self-efficacy of youth participants in citizen science projects like GLOBE could further build connections between the outcomes of individual participants and the characteristics of the contributions they can make to citizen science initiatives.

We also observed patterns in how frequently student projects fell into each sub-cluster based on student age or grade band. The most simple/basic project sub-cluster was most common among lower primary students (40%). Similarly, the most complex/robust projects with more sophisticated indicators of science process skills were observed only in students at the middle school (10%) and secondary (18%) levels. By contrast, lower primary students were much more likely to consider impact and ecological action (60%, compared with 31% of projects overall) or to demonstrate change in their own self-efficacy (29%, compared with 7% overall). Lower primary students were also more likely to conduct investigations focused on engineering solutions or investigations with control variables (40% versus 17% overall).