Citizen science (CS), which involves members of the public in authentic scientific research, has the potential to generate important ecological datasets and, in some cases, influence the values and behaviors of participants (Ellwood, Crimmins, and Miller-Rushing 2017; McKinley et al. 2017; Stepenuck and Green 2015). At present, however, CS datasets are generated abundantly in certain ecosystems and are limited or non-existent in others. Many successful CS projects focus on urban or backyard habitats and are capable of generating scientific data at an unprecedented scale (Bonney et al. 2009a; McCaffrey 2005). However, only a handful of projects are systematically generating data in remote, mountainous areas (Erb, McShea, and Guralnick 2012; Jackson, Gergel, and Martin 2015; McDonough MacKenzie et al. 2017), despite concerns that the impacts of climate change may be intensified at high elevations (Mavris et al. 2015; Pauli, Gottfried, and Grabherr 2014). Because traditional research efforts in these settings may be resource limited (Jackson, Gergel, and Martin 2015), CS programs that involve hiking clubs and other outdoor-oriented organizations could play a crucial role in monitoring some of these rugged locations (McDonough MacKenzie et al. 2017).
In addition to landscape-derived data gaps, certain participant groups remain underrepresented in CS. Many existing CS projects acknowledge that the majority of their volunteers are middle-aged or older (Crall et al. 2013; Jordan et al. 2011; Wright et al. 2015). Accordingly, CS programs that specifically target youth are on the rise, in part due to the perceived potential for these endeavors to generate future environmental advocates (Ballard et al. 2016). One promising strategy to increase youth participation in CS may be to design projects specifically tailored to youth organizations (Bonney et al. 2009a; Dickinson et al. 2012; Purcell et al. 2012).
Outdoor adventure education (OAE) organizations teach leadership, interpersonal, and wilderness skills in outdoor settings (Hanna 1995; Priest 1986). In contrast, environmental education organizations are singularly designed to teach students about ecology or environmental issues (Priest 1986). Though OAE organizations often incorporate aspects of environmental education, their primary focus is on personal and social growth. While OAE may not emphasize science or ecological knowledge to the same extent as environmental education, OAE organizations consistently deliver gains in student self-perceptions of ownership and empowerment (Hanna 1995; Hattie et al. 1997; Paisley et al. 2008; Sibthorp, Paisley, and Gookin 2007), which are recognized as critical “variables” for inspiring responsible environmental behavior (Hungerford and Volk 1990). Similarly, OAE programs are known for their experiential nature; experiential education is defined by a wide umbrella of learning environments in which students take a direct, active role (Kolb 1984). In contrast, environmental education programs have sometimes been criticized for overemphasizing content knowledge over student ownership and empowerment (Fry 2017; Hungerford and Volk 1990). At present, a minimum of several thousand youth participate in OAE programming annually (NOLS 2016), but Internet searches reveal that few are formally involved in CS projects. Because OAE organizations typically operate in remote areas, CS projects in this context have the potential to generate data in previously inaccessible locations. In addition, partnerships with OAE organizations would create an opportunity to engage younger audiences in CS. Finally, these audiences may derive more impact from CS when their experience is paired with OAE’s potential to instill agency in its participants. An important first step in exploring CS partnerships with OAE organizations is to investigate the attitudes towards CS among potential volunteers in this setting.
Similar to partnerships with classroom or museum educators, CS projects in an OAE setting must be championed by group leaders to reach a “captive audience” of students (Wells et al. 2012). By “training the trainer,” i.e., providing guidance to instructors in these scenarios, a project may ultimately harness more participants (Bonney et al. 2009b). However, in the context of a classroom-oriented CS project, Trumbull et al. (2005) cautioned that CS project managers must “acknowledge two rather than a single group of learners: Teachers as well as students.” In the case of OAE organizations, administrators and instructors represent the gateways to accessing student audiences and remote data points. As a result, it is crucial to understand the attitudes toward CS of gateway participants such as group leaders, teachers, or administrators.
Though research is available on the CS experiences of group leaders, an understanding of the motivations of individual CS participants is viewed as critical to the implementation of successful CS projects (Bonney et al. 2009b; Raddick et al. 2013; Rotman et al. 2012). Previous research suggests that CS participants are often altruistically self-motivated (Dickinson and Bonney 2012). For instance, participants of Galaxy Zoo, an online astronomy initiative, were motivated primarily by a desire to contribute to science (Raddick et al. 2013). Similarly, in a CS project that involved smartphone-based environmental monitoring, a desire to contribute to the environment prevailed as a primary motivator of participants (Land-Zandstra et al. 2016). In more involved environmentally focused CS projects that require days or months of commitment, participants have described their desire to learn as the primary motivation (Domroese and Johnson 2016; Merenlender et al. 2016).
Volunteer retention is critical to generating CS data sets with high scientific or conservation impact. In the case of high-elevation ecosystems, long-term monitoring may be necessary to detect and mitigate the impacts of climate change on natural systems (Pauli, Gottfried, and Grabherr 2014). To fulfill longitudinal research demands in remote locations, a potential CS project must recruit and retain a sufficient number of participants. To do so, CS project developers should identify and meet the needs and interests of volunteers (Beaubien and Hamann 2011). Volunteer satisfaction may also be linked to data quality, yet another reason for CS project managers to investigate the benefits and challenges experienced by their volunteers (Wright et al. 2015).
The Clark’s Nutcracker Project was developed by the author in the summer of 2016 in partnership with Dr. Taza Schaming. We engaged NOLS (formerly the National Outdoor Leadership School) in a contributory CS project (Shirk et al. 2012) to investigate the impacts of habitat decline on a high-elevation bird species. Specifically, the project focused on a critical avian seed disperser, the Clark’s Nutcracker, in response to the widespread decline of whitebark pine, the bird’s obligate partner in a keystone mutualistic relationship. The goals of the project were to 1) generate data on Clark’s Nutcracker habitat use in wilderness ecosystems that have traditionally been very difficult to access; 2) educate participants about the Clark’s Nutcracker and whitebark pine ecosystems; and 3) increase participant awareness of CS as an avenue for individuals to contribute to conservation. In addition to launching and managing the project, the author investigated the perceptions of NOLS instructors and administrators toward the Clark’s Nutcracker Project throughout one project season. Engaging in exploratory action research, I used online questionnaires and semi-structured interviews to identify the motivations, benefits, and barriers experienced by gateway participants during the project’s first season. Ultimately, I provide recommendations for future projects that seek to integrate OAE and CS.
Founded in 1965, NOLS expeditions educate some 5,000 students each year in various locations representing nine countries and five continents: USA, Canada, Mexico, Chile, Norway, India, Tanzania, Australia, and New Zealand (S. Rochelle, personal communication). The organization’s mission is “to be the leading source and teacher of wilderness skills and leadership that serve people and the environment” (NOLS 2017). On NOLS expeditions, students learn communication, good judgment, and resilience by developing outdoor skills, practicing risk management, and leading in a hands-on environment (S. Rochelle, personal communication). As a prime example of how OAE organizations incorporate aspects of environmental education in their programming, one of NOLS’ stated curriculum areas is environmental studies. Specifically, there are five objectives associated with environmental studies at NOLS: 1) explore the natural world through observation and application of ecological concepts, 2) develop a sense of place by experiencing wilderness and exploring relationships with the surroundings, 3) articulate an environmental ethic, 4) understand land management and environmental issues, and 5) apply Leave No Trace (LNT) principles to camping and travel (S. Rochelle, personal communication). The LNT philosophy “teaches people of all ages how to enjoy the outdoors responsibly” and includes seven concepts: Plan ahead and prepare, travel and camp on durable surfaces, dispose of waste properly, leave what you find, minimize campfire impacts, respect wildlife, and be considerate of other visitors (Center for Outdoor Ethics 2012). Within environmental studies and other topics, NOLS instructors are given a large amount of freedom to decide exactly how they will cover the NOLS curriculum areas with their students. Instructors must provide their students with many mandatory hard skillsets, such as backcountry cooking and the basics of wilderness medicine, and also deliver certain educational themes, such as diversity and inclusion. Even so, a team of instructors may change their teaching approach from one expedition to another, and each instructor may have a slightly different interpretation of overarching themes. NOLS instructors adapt the prescribed curriculum areas and enhance their teaching based on individual experiences, values, and the needs of their student group.
NOLS originally launched in western Wyoming amidst the dramatic scenery of the Wind River Mountains. Despite its geographic and operational expansion, the organization remains rooted in the Wind Rivers; the school’s 30-day Wind River Wilderness course remains its most popular field education offering, and on any given day in mid-summer, some 600 students and instructors are participating in the dozens of NOLS expeditions backpacking through the mountain range (S. Rochelle, personal communication). At the southern end of the Greater Yellowstone Ecosystem, the Wind River Mountains encompass 2.25 million acres ranging between 6,000 to 13,800 ft. in elevation and contain most of the highest peaks in Wyoming (Figure 1). These mountains are the source of headwater streams for three major river systems: The Colorado, the Missouri, and the Columbia. The Wind Rivers are primarily administered by USDA Forest Service within the Shoshone National Forest and Bridger-Teton National Forest. The vast majority of the remote mountain range is encompassed by three wilderness areas: The Bridger, Fitzpatrick, and Popo Agie Wildernesses. Another segment of the range falls within the Wind River Indian Reservation.
Clark’s Nutcrackers (Nucifraga columbiana) are the sole dispersal agent of the imperiled whitebark pine (Pinus albicaulis) (Tomback 1982). The whitebark pine is recognized as a keystone species because of its calorie-dense seeds that serve as a food source for many species of birds and mammals, its role as a soil-stabilizing nurse plant, and its influence on watershed dynamics (Callaway 1998; Chang, Hansen, and Piekielek 2014; Ellison et al. 2005; McKinney and Tomback 2007; Tomback and Achuff 2010). The pine depends entirely on the seed-caching Clark’s Nutcracker to disperse its seeds, but this mutualistic relationship may be seriously threatened by widespread habitat decline (McKinney, Fiedler, and Tomback 2009). The increased presence of an invasive fungus (Cronartium ribicola), which causes white pine blister rust, and a climate-facilitated native insect, the mountain pine beetle (Dendroctonus ponderosae), have resulted in range-wide losses of whitebark pine (Logan, Macfarlane, and Willcox 2010). In the Northern Rocky Mountains, whitebark pine mortality rates have been particularly severe, estimated around 80% in the Greater Yellowstone Ecosystem (Macfarlane, Logan, and Kern 2013). In 2013, aerial surveys revealed relatively healthy populations of whitebark pine in the Wind River Mountains, but Macfarlane et al. question whether this high-elevation range will remain a refuge for the species over time (2013). Recent modeling efforts suggest that the Wind Rivers may remain hospitable for whitebark pine in the face of climate change, but these projections do not incorporate variables such as insect outbreaks or disease (Chang, Hansen, and Piekielek 2014).
Taza Schaming studies Clark’s Nutcracker populations in response to whitebark pine decline (Schaming 2015, 2016). Her efforts to monitor Clark’s Nutcracker habitat use are particularly relevant in the Wind River Mountains where relatively healthy whitebark populations may yield important baseline Clark’s Nutcracker data for restoration targets in other areas. In the event that local whitebark pine populations decline precipitously, trends in Clark’s Nutcracker habitat use in the Wind River Mountains also may inform conservation interventions. Over time, citizen-conducted occupancy surveys will augment Dr. Schaming’s ability to investigate Clark’s Nutcracker habitat use at a scale that is ecologically relevant to the far-ranging bird (Hoffman and Potts 1981; Schaming 2015).
Before launching the Clark’s Nutcracker Project, I first introduced the idea to the NOLS Research Manager, who coordinates diverse initiatives to investigate the efficacy and outcomes of the organization’s activities. Through a series of conversations, I adjusted the study design and approach until receiving approval from the Research Manager to formally launch the project. In this time frame, I received access to NOLS curriculum materials, and adapted Clark’s Nutcracker Project materials to reflect aspects of NOLS environmental studies curriculum. The opportunity to participate in the Clark’s Nutcracker Project was introduced to a subset of NOLS instructors at an annual faculty conference shortly before the start of the summer expedition season. Additionally, a brief email announced the opportunity to a larger pool of instructors, and basic information about the project was made available on internal NOLS webpages. Instructors then self-selected to participate in the project. Day-to-day interactions between the author and NOLS staff provided many other opportunities for individual outreach over the course of the field season.
Participating instructors attended single training sessions individually or as a two-to-three person instructor team. The three-part training led by the author lasted approximately one hour. First, instructors learned about the ecological context for the research project. We explored whitebark pine and its interdependent relationship with the Clark’s Nutcracker, as well as the importance of whitebark pine ecosystems and the threats facing this high-elevation habitat. Second, the training focused on data collection and bird and tree identification; instructors had a chance to look over project resources and discuss protocols. Finally, we briefly discussed possible ways to connect the CS project to other aspects of the NOLS curriculum and introduced the opportunity for instructors to participate in the research presented here.
Each participating instructor group received supporting materials. These included a written protocol, data sheets, a laminated tree identification guide, and Clark’s Nutcracker and whitebark pine fact sheets. In addition, they received a research kit containing an iPod shuffle® and a camera or a project SD card. To help participants as they learned how to distinguish Clark’s Nutcrackers by ear, the iPod shuffle® contained audio files of Clark’s Nutcrackers and two other similar-sounding bird species. These materials were enclosed in a lightweight padded case and a protective, waterproof sleeve to facilitate instructor transport.
Subsets of instructors and students from 25 NOLS expeditions conducted opportunistic occupancy surveys for Clark’s Nutcrackers. Around 75% of these expeditions conducted at least one survey (Figure 2), resulting in 54 total surveys completed. An additional eight expeditions did not succeed in their intentions of collecting data (Figure 2).
Each survey included two basic parts: A rapid assessment of habitat conducted by the entire survey team, with group size ranging from 3-15 individuals, and a 10-minute, individually conducted point count to detect the presence or absence of Clark’s Nutcrackers (Table 1). The point-counts were conducted simultaneously by group members to allow for inter-group comparison of observations as a quality control check. I requested that surveys be conducted during daylight hours, in the absence of severe weather, and within 100 meters of conifers, but otherwise provided instructors the freedom to opportunistically determine survey locations and times. This approach maximized flexibility for other expedition objectives and requirements.
|Group Habitat Survey||Individual Nutcracker Survey|
|Required datasheets||1 per survey group||1 per observer|
|Metadata||expedition group, instructor and student names||name and expedition group|
|Data||qualitative weather description||Nutcrackers seen and/or heard:
yes/no by minute
|4 habitat photos in cardinal directions||estimated total number of nutcrackers detected|
|% ground cover estimates
by tree species and/or
vegetation or abiotic type
|Notes||brief written description of the site||optional record of survey including:
non-target species observed
nutcrackers observed outside of survey
nutcracker behavior observed
In addition, instructors were given free rein in their approach to educating students about the project context and scientific methodology. I offered teaching ideas and suggestions but did not mandate a set approach or minimum time needed to cover this content. Though I was unable to witness their field lessons firsthand, instructors informally described employing a variety of teaching approaches including short lectures, experiential tree identification activities, and student-led presentations. In many cases, the instructors who did not survey with their expedition groups informally reported having nevertheless taught lessons or completed activities relating to the ecological context of the project.
At project trainings, I encouraged each instructor to provide feedback upon exiting the field via an online questionnaire, an in-person interview, or both. I encouraged across-the-board participation irrespective of whether an instructor succeeded in data collection. Beyond informal conversations, I did not succeed in soliciting feedback from instructors that had opted not to participate in project trainings.
The 13-item questionnaire was developed independently by the author and administered via GoogleForms (see Supplementary File), and a link to this questionnaire was emailed to every instructor who participated in a project training upon exiting the field. The questionnaire asked instructors to select from pre-determined choices or to write in a response to describe their scientific background, motivations for participating, confidence in data collection, and intentions to participate in the future. The questionnaire also included open-ended questions about potential project improvements, project benefits and drawbacks, project influence on environmental studies curriculum, and citizen science’s potential role in stewardship education. The brief questionnaire represented my best attempt to balance the nuance of my research interests with the time constraints of busy field instructors.
A subset of 12 instructors chose to participate in semi-structured interviews that lasted from 45 minutes to 1 hour and 15 minutes. Interview questions focused on participants’ familiarity with CS, positive and negative experiences with the project, and general outlook on citizen science within the context of NOLS (see Supplementary File for a list of interview questions). Each semi-structured interview included pre-determined questions, but follow-up questions and exact content of interviews differed among participants to capture the wider spectrum of interests and experiences that varied by individual (Wilson 2014).
In addition to instructors, I solicited five interviews from NOLS program supervisors, who are responsible for preparing each instructor team for an expedition by providing trainings, guidance, and logistical support. Program supervisors debrief with both student groups and instructors after an expedition, so they could provide further insight into both the student and instructor experience surrounding the project. I also interviewed six upper-level NOLS administrators (e.g., the Rocky Mountain Assistant Director, the NOLS Research Manager, and the NOLS Stewardship Director.) Interviews with program supervisors and other administrators were similar in content to the instructor interviews; however, non-applicable questions that pertained to project implementation in the field were replaced with questions about organizational attitudes towards the project and the role of CS at NOLS (see Supplementary File). All interviews were audio recorded with permission and later transcribed verbatim using HyperTranscribe Version 1.5.3.
To evaluate instructor scientific background, motivations, confidence in data collection, and likelihood of future participation, I used descriptive analysis—calculating the frequency and percent respondents by selected response—for the corresponding multiple-choice items within the online questionnaire. In a separate process, I evaluated barriers and benefits to participation along with desirable project attributes through combined qualitative analysis of both instructor interviews and instructor responses to the seven open-ended items from the online questionnaire. Though interview questions were worded differently than those in the online questionnaire, the topics addressed were largely similar, e.g., instructors’ positive and negative experiences with the project. In general, I posed fewer, more-concise questions in the online questionnaire, for example, “what, if anything would you change about this project?” Conversely, in interviews, I attempted to solicit more detailed responses to the same topic, for example, by instead posing the following questions: “What were the drawbacks and/or logistical challenges to this project?” and “What aspects of this project did you and/or your students struggle with?” By merging the qualitative analysis of interviews and open-ended online questionnaire responses, I could reflect the experience of a greater number of instructors in my corresponding findings. Across this combined data set, I used grounded theory methods to iteratively generate codes that served as descriptive labels with which to organize large amounts of information (Corbin, Strauss, and Strauss 2008). All codes were applied using HyperResearch Version 3.7.3. First-cycle codes emerged as broad categories (e.g., positive feedback, negative feedback), and were refined through careful comparison within categories and between sources (Saldaña 2013). After second-cycle coding to distill categories into more precise concepts and phrases, I quantified the frequency and percent of respondents by code. To complement this analysis, I applied the same coding scheme to interview transcripts from my conversations with NOLS administrators. I triangulated between these sources and detailed field notes of informal conversations to reach the following insights into the participant experience during the first season of the Clark’s Nutcracker Project (Miles, Huberman, and Saldaña 2014). Due to resource limitations, the author solely conducted the coding and analysis process. Codes were identified and applied as consistently as possible, however, the study lacks inter-coder reliability and validity checks. As one measure of external validity, however, the following findings were reviewed before publication by stakeholders including the NOLS Research Manager and one NOLS instructor.
Out of a minimum of 164 instructors who received word of the project via email, 56 individuals, representing 35 separate backcountry expeditions, opted to participate in the Clark’s Nutcracker Project by attending the required training. Each expedition included up to thirteen students and up to four instructors. Instructors conducted at least one survey for Clark’s Nutcrackers on approximately three-quarters of these participating expeditions, involving a total of 213 students in data collection. However, in the case of eight expeditions, instructors failed to collect data with their students despite having trained for the project. Of the expedition groups that collected data, most completed just one survey. No single expedition group completed more than five surveys.
Of the 56 instructors who trained for the project, 12 instructors self-selected to participate in interviews for a response rate of 21%. Each of these instructors had conducted at least one Clark’s Nutcracker survey on their expedition except for one respondent who had not found sufficient time or weather conditions to do so. Twenty-eight instructors responded to the online questionnaire for a response rate of 50%. Every questionnaire respondent had conducted one or more surveys in the field with the exception of two instructors who did not succeed in collecting any data. Accounting for the five instructors who participated in both the online questionnaire and interview, I heard from a total of 35 individuals for an overall response rate of 63%.
The vast majority of NOLS instructors are college-educated (Shannon Rochelle, personal communication). Based solely on online questionnaire responses (N = 28), nearly 18% of responding instructors (n = 5) described themselves as having no formal background in science. Nearly 43% of instructors (n = 12) described their highest level of science education as having taken some science-related courses in college. Almost 29% of responding instructors (n = 8) reported majoring in a science-related field as an undergraduate, and just under 11% reported that they had completed graduate-level science-related coursework (n = 3). Based solely on interviews (N = 12), eight instructors had heard of CS before this project (67%), with three of those eight having previously participated in some form of CS. However, one-third of respondents (n = 4) had no previous awareness of CS before the Clark’s Nutcracker Project.
The 28 instructors who responded to the online questionnaire described their motivations to participate in the project by selecting any combination of seven provided statements (Figure 3). Ninety-two percent of respondents (n = 26) identified the following statement as an influence: “I thought [the project] would enhance my Environmental Studies curriculum in the field.” Other prominent motivations included hearing about the project in person (n = 23, 82%), a desire to contribute to conservation (n = 22, 79%), an affinity for science (n = 19, 68%), and a desire to learn about the ecosystem (n = 17, 61%). Though given the opportunity, no participant chose to write in an additional response beyond the options provided.
In the combined analysis of both instructor interviews and open-ended questionnaire responses (N = 35), instructors most frequently identified issues with habitat-related data collection as a barrier to participation. Forty-three percent of responding instructors (n = 15) commented unfavorably on the complexity and length of this survey component. More specifically, 31% of instructors (n = 11) relayed their uncertainty and discomfort surrounding percent cover estimates. When asked to identify project improvements, around 20% of instructors (n = 7) requested that the habitat survey be simplified. Seventeen percent of instructors (n = 6) specifically indicated that the habitat survey took longer than anticipated, and as such, contributed to the second most-identified barrier: Time limitations.
Thirty-four percent of responding instructors (n = 12) cited limited time as both a barrier to their data collection efforts in the field and a roadblock for their initial or continued participation in the project. Of the 56 instructors who participated in the project training, 14 instructors representing 8 expeditions took materials for data collection into the field but returned with no data while many others completed fewer surveys than anticipated. As one instructor related, “[t]he time requirements for coordinating the observations as small groups were unfortunately enough for our [instructor] team to punt these opportunities behind other course goals (i.e., travel/logistics for a large group of 17 persons, basic outdoor living skill classes, and risk management) and did not lead to adequate data collecting.”
In addition to describing time as a limiting factor during expeditions, three instructors also mentioned the difficulty of fitting the one-hour instructor training into the two-day briefing period before expeditions. In my interviews with NOLS administrators, two program supervisors made note of the same perceived barrier. During briefings, instructor teams prepare equipment and plan expedition logistics; several mandatory trainings, unrelated to this CS project, also occur in this short window of time.
Another frequently identified barrier to participation was student disappointment upon not encountering Clark’s Nutcrackers during the 10-minute survey window. Because of the low likelihood of encountering nutcrackers in every survey, project trainings specifically highlighted the value of systematic data collection, the importance of absence data in occupancy assessments, and quiet surveys as an opportunity for self-reflection. Despite these attempts to preempt dissatisfaction, nearly 23% of responding instructors (n = 8) described their students as “bummed” or “frustrated” by surveys with very few or no nutcrackers. As one instructor related, “… there was some disappointment in not being able to tell you all the times we saw a Clark’s [nutcracker]. We saw all these Clark’s [nutcrackers] and then when we did the study we only saw one.” Perhaps in light of these experiences, 20% of responding instructors (n = 7) expressed interest in recording nutcracker sightings opportunistically as opposed to participating in presence-absence surveys.
Four instructors and two administrators identified their concerns about student data quality as another barrier to participation. In the 28 responses to the online questionnaire, a majority of instructors described themselves as “confident” or “extremely confident” in their own ability to collect accurate data (n = 21, 75%), but most were only “somewhat confident” in their students’ abilities (n = 18, 64%) (Figure 4). As one instructor expressed, “I was questioning … the data, I mean it’s a lot of work to go through a bunch of chickenscratch … not to say that in a disrespectful way … but it’s hard to know how much of this can be used in the real science data world.” Other topics that surfaced as barriers for three or less instructors included a lack of student motivation, equipment difficulties, and curriculum trade-offs, for example, teaching about nutcrackers instead of something else.
Though instructors described a range of benefits arising from their participation in the project, several of these benefits were linked to the project’s perceived value as a teaching tool, or a vehicle that improved instructional outcomes. Across interviews and open-ended questionnaire responses (N = 35), 80% of instructors (n = 28) favorably described an increased awareness in their students relating to participation in the Clark’s Nutcracker Project. As one instructor related, “I got to hear students say things like, ‘Oh, now I am noticing all the different kinds of birds that are around us all the time,’ … it is often very challenging to get students to key into that kind of stuff—this project did that really well from an education point of view.” Many other instructors also described their students as paying greater attention to their surroundings as a result of the project.
Fifty-seven percent of responding instructors (n = 20) also reported the educational benefit of increased student buy-in to environmental studies curriculum stemming from the Clark’s Nutcracker Project. From the perspective of one of these instructors, the CS model resulted in students feeling “invested” in learning about the Clark’s Nutcracker. In the words of another instructor, “[r]ather than simply learning about something for the sake of learning about it, it gave them a sense of purpose, a reason for learning about something.” As additional support for this finding, one program supervisor related his perception of how students are prompted by the immediate environment to react to the ecological context of the project:
“You read about the issues that are going on and you see pictures, but when you go through there, and suddenly you get high on a summit and you see all the forest and you see all the patches of dead trees all over the place. Once you start noticing those things, it’s like ‘Oh yeah…’ Our students are able to connect what we’re talking about, what we’re teaching … They see that it is real.”
Beyond perceived student impacts, 37% of the 35 responding instructors (n = 13) appreciated the structure and focus that the project provided to their environmental studies curriculum. Under the umbrella of environmental studies, instructors have various prescribed and personally defined learning objectives for their students, and many participants felt that the project provided a focus for these intentions. One instructor described the project as “taking the things I am passionate about and giving [these topics] context and the structure.” Seventeen percent of instructors (n = 6) related that the project constituted “the meat” or the “most legitimate piece” of the environmental studies curriculum they covered on their expedition, while 11% of instructors (n = 4) remarked positively upon the opportunity to tie several environmental studies curriculum areas into a cohesive lesson surrounding nutcrackers and whitebark pine. For example, one instructor appreciated that she was “able to teach much of the [NOLS] ecology curriculum using the nutcracker-whitebark relationship/story as a case study of sorts.”
As a teaching tool, nearly 49% of instructors (n = 17) also found value in the project’s experiential nature. Instructors either explicitly mentioned the project as “experiential” or “hands-on” in their positive feedback or remarked favorably on the project’s emphasis on “doing” as opposed to “talking.” As one instructor related, “… NOLS does a good job of journaling prompts and writing about your wilderness ethic, and ok, now, what are you going to do about your wilderness ethic? And this was a cool piece of doing.”
During interviews with NOLS administrators (N = 12), one half of respondents reflected positively on the Clark’s Nutcracker Project as a learning opportunity for instructors (n = 6). Four program supervisors specifically identified this project as beneficial to newer instructors or instructors who are generally less knowledgeable about environmental studies. One program supervisor commented, “here’s an opportunity for [instructors] to have this pretty much structured and delivered — like all you have to do is go out there and do it. I think that’s pretty cool because I think that is what has felt overwhelming for instructors at times —is just not having that information.” Indeed, in interviews and open-ended online questionnaire responses (N = 35), 34% of instructors (n = 12) identified personal learning as a benefit of participating in the Clark’s Nutcracker Project. Specifically, instructors reported gaining bird and tree identification skills as well as a better understanding of current ecological issues in the Wind Rivers. One instructor reflected that it would be “an amazing professional development opportunity” if NOLS provided her with the opportunity to engage in new CS projects each summer season.
In interviews and open-ended online questionnaire responses (N = 35), instructors highlighted several key ingredients that may have resulted in the benefits described above. Nearly 23% of instructors (n = 8) named project simplicity — both the ease of implementation and approachability of content — as a desirable project attribute. In many instances, ease of implementation involves the quality of project materials such as identification guides, protocols, and datasheets. Over 52% of instructors (n = 19) remarked favorably on the materials associated with this project. As one instructor described, “… with the amount of tools and materials that were provided, it was very easy to make this a tool that benefitted the student experience.” Another instructor identified the benefit of having the supporting materials packaged for use in the backcountry: “… with laminated [information sheets], waterproof bags, I didn’t even have to think, you just put them in your pack and you pull them out when you want to do the survey.” Slightly over 34% percent of responding instructors (n = 12) commented favorably on the project training. Over 25% specifically mentioned the “energy” and “enthusiasm” they encountered at these trainings (n = 9). Interviews with NOLS administrators provided additional insight into positive project attributes. In the words of one program supervisor, energy during project trainings is a key currency that “translates to instructor excitement and … [then] they are way more willing to explore and try things with students.” Four NOLS administrators identified target species visibility as a key attribute for successful CS projects, a criterion that was also mentioned by nearly 9% of instructors (n = 3). When describing the attributes of a “good” citizen science project, one NOLS administrator said, “it’s something that students don’t have to work too hard to observe, like if something is tedious or hard to find, or not that interesting to students, they are not going to be that engaged.”
An instructor’s willingness to continue with a project was used as a tool to measure satisfaction with the project. In response to a multiple-choice item from the online questionnaire (N = 28), instructors overwhelming indicated a favorable outlook on their future participation (Figure 5). Over 64% of respondents (n = 18) described themselves as “very likely” to participate in the project in the future. The remaining 36% (n = 10) described themselves as “somewhat likely” to participate in the future. Twenty percent of responding instructors (n = 7) participated in the project multiple times in the summer with separate expedition groups, actively demonstrating their favorable response to the project.
It is important to note that not every participating instructor connected positively with the Clark’s Nutcracker Project, and in some instances, their experiences with the project changed over time. For example, one instructor who participated in data collection on two expeditions responded very favorably to the project when she led a highly motivated student group, who commemorated every nutcracker sighting with a signature dance move. Later in the same summer, this instructor had a lackluster experience when the students of her second expedition were largely disinterested in the project.
Though this research is exploratory, the rich information that was generated through interviews and online questionnaires sheds light on the experience of gateway participants within a new CS audience: The Outdoor Adventure Education (OAE) community. To provide a starting point for future CS projects in this setting, I compare the motivations, barriers, and benefits I identified among outdoor adventure educators to those described among other CS audiences and then offer a set of preliminary best practices based on these findings.
In the case of the Clark’s Nutcracker Project, participating instructors most frequently identified the motivation to enhance their environmental studies curriculum through CS participation. In contrast, among environmental monitoring projects, teaching-related intentions rarely surface as a primary motivation. Though multiple studies identify the desire to contribute—whether to scientific research or to the environment—as the top motivator for their volunteers (Land-Zandstra et al. 2016; Raddick et al. 2013), I suspect that this is not the top motivator among outdoor adventure educators who are first and foremost responsible for training and teaching their student groups. Other diverse studies point to a desire to learn as the primary driver behind participation both for volunteers engaged in pollinator observations (Domroese and Johnson 2016) and those participating in Master Naturalist programs (Merenlender et al. 2016). Even though many NOLS instructors described the project-related benefit of personal learning, their decision to participate is most clearly linked to their desire to provide quality environmental studies curricula to their students. Thus, it is critical to consider a project’s merits as a teaching tool when implementing CS in the OAE setting.
The second most selected motivation among participating instructors was “becoming interested in participating after hearing about the project in person.” Less than 15% of responding instructors indicated that they elected to participate in the project after hearing about it solely via email; the majority of participating instructors chose to do so after engaging directly with the author. In interviews and open-ended online questionnaire responses, instructors commented in more detail on the important role that in-person enthusiasm played in influencing their intention to participate. Evans et al. noted that personal, consistent, face-to-face communication between staff and participants was essential to the success of the Neighborhood NestWatch program (2005), and other researchers also attribute project outcomes and longevity to dedicated project staff and good leadership (Beaubien and Hamann 2011; Gooch 2005; Krasny and Bonney 2005). The need for personal connection and “passion” behind a CS project may be heightened within the OAE community (Droege 2007). For example, the Clark’s Nutcracker Project relied on a “train the trainer” model: After a mere one-hour training, instructors were entirely responsible for transmitting knowledge and facilitating data collection with their students. Without the motivating influence of face-to-face connection, many instructors would likely have declined to participate in the project given the time scarcity that prevails during the few, jam-packed days of expedition planning. I speculate that this would have been the unfortunate reality despite widespread interest in project content and point to the two other CS projects that launched concurrently at NOLS, which only garnered an extremely limited number of participants (Shannon Rochelle, personal communication). One of these projects involved filtering snowpack to capture particulates, and the other involved monitoring an alpine cushion plant; both projects were presented to NOLS by remote primary investigators and lacked the physical presence of a project manager. Bonney et al. (2009b) warned that “simply offering project materials rarely leads to adoption,” and this advice may warrant special consideration within the OAE setting. Clearly, there are tradeoffs that come with the need for project leaders to personally represent a project rather than merely supplying materials to their volunteers (Evans et al. 2005). Another crux of implementing CS within the OAE community involves the high amount of instructor turnover that may occur at an organization like NOLS (Shannon Rochelle, personal communication). Re-training and re-transmitting enthusiasm to new instructors year after year could become prohibitively time consuming and costly as it has in other citizen science projects (West and Pateman 2016). One potential solution to this conundrum relies on building the capacity of an entire OAE organization to sustain a project with internal trainings and word-of-mouth promotion.
I found that habitat-related data collection was perceived as the most prominent barrier to participation in this study. In this regard, my research further confirms the frequently described trade-off between study complexity and participant motivation (Bonney et al. 2009b; Droege 2007; Gallo and Waitt 2011; Hochachka et al. 2012). My data suggest that participants struggled with the subjectivity and length of the habitat survey as opposed to the content; tree species were, in fact, accurately identified by citizen scientists at 10 survey points re-visited by the author at the end of the field season. Specifically, many participants pinpointed percent cover estimates as a source of struggle, citing a lack of time, confidence, or patience with this duty. Though other authors have advised against asking CS participants to collect subjective measurements because of data quality concerns (Galloway, Tudor, and Haegen 2006), I would add participant reluctance as another reason to avoid these sorts of data requests in an OAE setting. Taken as a whole, the length of the habitat survey directly related to the second most identified barrier: Time limitations.
Though time is a limiting factor for volunteer audiences worldwide (Everett and Geoghegan 2016; Merenlender et al. 2016; West and Pateman 2016), CS projects designed for leaders within the OAE setting are in many ways considerably more constrained by time, weather, and equipment than projects in other settings. NOLS instructors have professional commitments to safety and curriculum goals that will always take priority over CS projects and often over environmental studies curricula in general. In the backcountry setting, a commitment to safety means that large amounts of time go into managing risk and facilitating group dynamics and self-care. Though other citizen science practitioners have described “wet weather” as a challenge for their participants (Domroese and Johnson 2016), an approaching thunderstorm on a high mountain pass is particularly non-negotiable. NOLS’ commitment to leadership education is similarly non-negotiable. The organization has been widely recognized for its highly effective leadership curriculum that often culminates in overnight to weeklong, student-led, independent expeditions (Paisley et al. 2008). This leadership progression fosters self-efficacy in students (Daniel et al. 2014), but is also time-consuming to implement; in order to ensure student safety and success within independent groups, there are many mandatory learning steps that may not be skipped or condensed.
Beyond bedrock curriculum objectives and safety requirements, students must first master the basics of living outdoors to be receptive to a CS project in the field. Similarly, for students of an ecological adventure course, Fry (2017) described self-empowerment and comfort in backcountry living as prerequisites to a favorable student response to ecology lessons. Though it’s easy to acknowledge a hierarchy of needs that might precede successful environmental programming (Maslow 1943), it is more difficult to plan for the full amount of time that student backcountry groups might require to meet these fundamental needs. The Clark’s Nutcracker Project typically asked for only a few hours of group time spread over the course of month-long expeditions. Nevertheless, several groups did not manage to conduct any surveys, even in the presence of highly motivated instructors. Project designers and managers working with the OAE community may need to accept that a number of participating expeditions will routinely return without data when faced with challenging weather conditions, medical evacuations, or difficult group dynamics.
Even though Clark’s Nutcrackers are noisy and commonly observed birds in the Wind Rivers, fewer than one third of the 10-minute surveys detected them, and nutcracker-less surveys were negatively received by students and instructors alike. Similarly, Trumbull et al. (2005) describe student and teacher disappointment over failing to attract birds to ill-placed feeders in an another avian CS project. Still other authors identify the challenge of maintaining volunteer motivation while generating absence data (Dickinson and Bonney 2012; Gommerman and Monroe 2012). In the case of the Clark’s Nutcracker Project, there are two readily apparent opportunities to overcome these stumbling blocks; one involves designing projects that rely entirely on opportunistic presence-only sightings or measurements, and the other involves strengthening the project curriculum and communication to increase participant awareness of the value and necessity of collecting absence data. The first avenue might lead to a heightened sense of accomplishment among participants, which could lead to greater participant retention, but at the same time, this approach may not meet the research needs behind the project. If successful, the second avenue may lead to a deeper participant understanding of ecological monitoring, which in turn may lead to the collection of higher quality data. This is important to note, because some of the participants identified concerns over data quality as a barrier to participation. Volunteer satisfaction and data quality have also been at odds in the context of other contributory projects (Gallo and Waitt 2011; Galloway, Tudor, and Haegen 2006).
The Clark’s Nutcracker Project was best received when instructors had positive experiences using the project as a teaching tool. Though future research would be valuable to confirm instructor perceptions of increased student awareness, similar gains have been reported in the context of other CS projects (Haywood, Parrish, and Dolliver 2016; Jordan et al. 2011). As far as instructor perceptions of improved student buy-in, the visible signs of habitat decline on the landscape, such as conspicuous swaths of dead trees, may have provided a sense of urgency or “mission” for students, as other research has confirmed the potential for ecological crises to spur environmental action (Vasi and Macy 2003). Perceptions of CS as “real” science have also been known to influence student agency (Ballard, Dixon, and Harris 2016). In the case of the Clark’s Nutcracker Project, potential increases in student buy-in may also be related to the project’s explicit connections to outside researchers. Another key benefit experienced by instructors occurred when the Clark’s Nutcracker Project gave structure and focus to existing curriculum objectives. Several authors describe the necessity of collaborating with teachers, museum personnel, or other youth leaders to enhance educational outcomes and shared value by matching CS activities to a specific audience (Bonney et al. 2009b; Dickinson et al. 2012; Galloway, Tudor, and Haegen 2006).
Only two of my self-selected respondents represent the 25% of instructors who trained with the project and yet failed to collect data with students in the field. Because I heard from so few of these “willing, but unable” instructors who may have had less-satisfying experiences with the project, I may not have captured the entire spectrum of attitudes toward the Clark’s Nutcracker Project. Furthermore, respondents self-reported and could have been swayed to respond positively in interviews in the presence of the author or could have been otherwise biased in their positive perceptions surrounding CS. It is worth noting, however, that without any requirement or push from the NOLS administration, more than 30% of NOLS instructors working in the Wind River Mountains in the summer of 2016 chose to participate in the project.
Through this exploratory research, I have identified five lynchpin requirements for practitioners considering CS projects in the OAE context. These non-hierarchical guidelines are listed below:
Given the demands of backcountry travel and other non-negotiable curriculum priorities, CS projects in the OAE setting should be 1) very easy to implement in terms of required time commitment and equipment. On the other hand, I learned that CS in the OAE setting doesn’t need to be “easy” in terms of content and setting. Almost all NOLS instructors are college educated, and each is a professional educator at home in the outdoors. Furthermore, the OAE community is uniquely poised to access locations that would be categorically too rugged, mosquito-ridden, or otherwise “hard” for many other volunteer audiences.
CS projects should be 2) well-aligned with the existing curriculum of each OAE partner organization; a notion that is reinforced by other research (Bonney et al. 2009b; Dickinson and Bonney 2012; Trumbull, Bonney, and Grudens-Schuck 2005). By highlighting the connections between a project’s content and an organization’s learning objectives, instructors are more likely to grasp (and be motivated by) the project’s potential as a teaching tool. In order to be a viable teaching tool within OAE, a project must also be fun or meaningful for students.
To encourage heightened student awareness and buy-in, CS projects in the OAE setting should 3) focus on a readily visible study subject and be 4) defined by a compelling need. In the case of this project, the common and conspicuous squawks of nutcrackers at high elevations made environmental lessons more accessible and allowed students to exercise new environmental awareness. This project also was defined by the nutcracker’s important role in an imperiled and richly interconnected ecosystem. Droege highlighted the need for a “mission” and “story” behind successful CS projects (2007), and according to instructor testimony, these same elements were often responsible for bringing the Clark’s Nutcracker Project to life for students.
Lastly, in an ideal scenario, CS projects in the OAE context should 5) connect to students’ lives outside the backcountry. Projects with direct, visible impact on students’ lives in their home communities may be one way to generate student interest more consistently (Fry 2017). This transference of environmental learning will be key if CS participation in the OAE context is to lead to greater societal gains.
In the case of the Clark’s Nutcracker Project, I witnessed a multiplicative effect by engaging an OAE organization; by training 56 NOLS instructors I was able to expose more than 30 expeditions, representing some 300 students, to the ecological context of the project and to engage over 200 of them in hands-on data collection. I am cautiously optimistic that the project might have a further multiplicative effect by virtue of NOLS’ emphasis on self-agency and leadership. CS partnerships with OAE have the potential to generate data in rugged and remote locations. This opportunity can benefit researchers who may otherwise lack access to remote locations, but it may also be a critical arena in which to promote responsible environmental behavior. In many cases, OAE organizations have the potential to shape the wider outdoor recreation community as industry leaders and as an entry point for new community members (Friese, Hendee, and Kinziger 1998). In the USA alone, an estimated 10.1 million people went backpacking in 2017 (Outdoor Foundation 2017). The activities of this growing population may increasingly overlap with critical pockets of biodiversity. There is a growing body of research that points to the potential for non-consumptive outdoor recreation activities to cause negative ecological impacts (Marzano and Dandy 2012; Pickering 2010; Rösner et al. 2014). Though outdoor recreation groups often align with conservation interests, there are scenarios in which conservation and outdoor recreation groups may be in direct opposition (Davis et al. 2010). With the end goal of negotiating these tensions, well-designed citizen science projects with OAE organizations may play a role in creating an outdoor recreation community that is as equally engaged in species conservation as it is in protecting land access. Careful implementation and consideration of best practices will be crucial to achieving this vision.
The supplementary file for this article can be found as follows:Supplementary File
Questionnaire and Interview Questions. DOI: https://doi.org/10.5334/cstp.127.s1
All interview and questionnaire procedures were reviewed by University of Vermont’s Committee on Human Subjects and approved under IRB Exemption Determination CHRBSS #16-581.
The author thanks the instructors, administrators, and students of NOLS who contributed to the Clark’s Nutcracker Project, especially Research Manager Shannon Rochelle. In addition, Taza Schaming, Kris Stepenuck, Jennifer Pontius, Walter Poleman, Hallie Schwab, Amy Lorenz, and Dana Gulley were critical mentors in the pursuit of this research. This manuscript also benefitted from valuable commentary provided by several reviewers and by the Citizen Science: Theory and Practice editorial team.
The author has no competing interests to declare.
Ballard, HL, Dixon, CGH and Harris, EM. 2016. Youth-focused citizen science: Examining the role of environmental science learning and agency for conservation. Biological Conservation, DOI: https://doi.org/10.1016/j.biocon.2016.05.024
Beaubien, EG and Hamann, A. 2011. Plant phenology networks of citizen scientists: recommendations from two decades of experience in Canada. International Journal of Biometeorology, 55(6): 833–841. DOI: https://doi.org/10.1007/s00484-011-0457-y
Bonney, R, Ballard, H, Jordan, R, McCallie, E, Phillips, T, Shirk, J and Wilderman, CC. 2009a. Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education. A CAISE Inquiry Group Report. Online Submission.
Bonney, R, Cooper, CB, Dickinson, JL, Kelling, S, Phillips, T, Rosenberg, KV and Shirk, J. 2009b. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience, 59(11): 977–984. DOI: https://doi.org/10.1525/bio.2009.59.11.9
Callaway, RM. 1998. Competition and Facilitation on Elevation Gradients in Subalpine Forests of the Northern Rocky Mountains, USA. Oikos, 82(3): 561. DOI: https://doi.org/10.2307/3546376
Center for Outdoor Ethics. 2012. Seven Principles Overview | Leave No Trace. 2012. Available at https://lnt.org/learn/seven-principles-overview [Last accessed 12 August 2017].
Chang, T, Hansen, AJ and Piekielek, N. 2014. Patterns and Variability of Projected Bioclimatic Habitat for Pinus albicaulis in the Greater Yellowstone Area. Bond-Lamberty, B. (ed.) PLoS ONE, 9(11): e111669. DOI: https://doi.org/10.1371/journal.pone.0111669
Corbin, JM, Strauss, AL and Strauss, AL. 2008. Basics of qualitative research: techniques and procedures for developing grounded theory. 3rd ed. Los Angeles, Calif: Sage Publications, Inc. DOI: https://doi.org/10.4135/9781452230153
Crall, AW, Jordan, R, Holfelder, K, Newman, GJ, Graham, J and Waller, DM. 2013. The impacts of an invasive species citizen science training program on participant attitudes, behavior, and science literacy. Public Understanding of Science, 22(6): 745–764. DOI: https://doi.org/10.1177/0963662511434894
Daniel, B, Bobilya, AJ, Kalisch, KR and McAvoy, LH. 2014. Autonomous Student Experiences in Outdoor and Adventure Education. Journal of Experiential Education, 37(1): 4–17. DOI: https://doi.org/10.1177/1053825913518892
Davis, CA, Leslie, DM, Walter, WD and Graber, AE. 2010. Mountain Biking Trail Use Affects Reproductive Success of Nesting Golden-Cheeked Warblers. The Wilson Journal of Ornithology, 122(3): 465–474. DOI: https://doi.org/10.1676/09-184.1
Dickinson, JL and Bonney, R. (eds.), 2012. Citizen science: public participation in environmental research. Ithaca: Comstock Pub. Associates.
Dickinson, JL, Shirk, J, Bonter, D, Bonney, R, Crain, RL, Martin, J, Phillips, T and Purcell, K. 2012. The current state of citizen science as a tool for ecological research and public engagement. Frontiers in Ecology and the Environment, 10(6): 291–297. DOI: https://doi.org/10.1890/110236
Domroese, MC and Johnson, EA. 2016. Why watch bees? Motivations of citizen science volunteers in the Great Pollinator Project. Biological Conservation, DOI: https://doi.org/10.1016/j.biocon.2016.08.020
Droege, S. 2007. Just because you paid them doesn’t mean their data are better. In: Citizen Science Toolkit Conference. Cornell Laboratory of Ornithology. 2007 pp. 13–26.
Ellison, AM, Bank, MS, Clinton, BD, Colburn, EA, Elliott, K, Ford, CR, Foster, DR, Kloeppel, BD, Knoepp, JD, Lovett, GM, Mohan, J, Orwig, DA, Rodenhouse, NL, Sobczak, WV, Stinson, KA, Stone, JK, Swan, CM, Thompson, J, Von Holle, B and Webster, JR. 2005. Loss of foundation species: consequences for the structure and dynamics of forested ecosystems. Frontiers in Ecology and the Environment, 3(9): 479–486. DOI: https://doi.org/10.1890/1540-9295(2005)003[0479:LOFSCF]2.0.CO;2
Ellwood, ER, Crimmins, TM and Miller-Rushing, AJ. 2017. Citizen science and conservation: Recommendations for a rapidly moving field. Biological Conservation, 208: 1–4. DOI: https://doi.org/10.1016/j.biocon.2016.10.014
Erb, PL, McShea, WJ and Guralnick, RP. 2012. Anthropogenic Influences on Macro-Level Mammal Occupancy in the Appalachian Trail Corridor. Fenton, B. (ed.) PLoS ONE, 7(8): e42574. DOI: https://doi.org/10.1371/journal.pone.0042574
Evans, C, Abrams, E, Reitsma, R, Roux, K, Salmonsen, L and Marra, PP. 2005. The Neighborhood Nestwatch Program: Participant outcomes of a citizen-science ecological research project. Conservation Biology, 19(3): 589–594. DOI: https://doi.org/10.1111/j.1523-1739.2005.00s01.x
Everett, G and Geoghegan, H. 2016. Initiating and continuing participation in citizen science for natural history. BMC Ecology, 16(S1). DOI: https://doi.org/10.1186/s12898-016-0062-3
Friese, G, Hendee, JC and Kinziger, M. 1998. The Wilderness Experience Program Industry in the United States: Characteristics and Dynamics. Journal of Experiential Education, 21(1): 40–45. DOI: https://doi.org/10.1177/105382599802100109
Fry, N. 2017. The Ala Archa Ecological Leadership Project: Evaluating the Effectiveness of Environmental Adventure Education in Growing Responsible Environmental Leaders in the Krygyz Republic. Rubenstein School Masters Project Publications, 13.
Gallo, T and Waitt, D, 2011. Creating a Successful Citizen Science Model to Detect and Report Invasive Species. BioScience, 61(6): 459–465. DOI: https://doi.org/10.1525/bio.2011.61.6.8
Galloway, AW, Tudor, MT and Haegen, WMV. 2006. The reliability of citizen science: a case study of Oregon white oak stand surveys. Wildlife Society Bulletin, 34(5): 1425–1429. DOI: https://doi.org/10.2193/0091-7648(2006)34[1425:TROCSA]2.0.CO;2
Gommerman, L and Monroe, MC. 2012. Lessons learned from evaluations of citizen science programs. Institute of Food and Agricultural Sciences, University of Florida.
Gooch, M. 2005. Voices of the Volunteers: an Exploration of the Experiences of Catchment Volunteers in Coastal Queensland, Australia. Local Environment, 10(1): 5–19. DOI: https://doi.org/10.1080/1354983042000309289
Hanna, G. 1995. Wilderness-Related Environmental Outcomes of Adventure and Ecology Education Programming. The Journal of Environmental Education, 27(1): 21–32. DOI: https://doi.org/10.1080/00958964.1995.9941968
Hattie, J, Marsh, HW, Neill, JT and Richards, GE. 1997. Adventure Education and Outward Bound: Out-of-Class Experiences That Make a Lasting Difference. Review of Educational Research, 67(1): 43–87. DOI: https://doi.org/10.2307/1170619
Haywood, BK, Parrish, JK and Dolliver, J. 2016. Place-based and data-rich citizen science as a precursor for conservation action: Citizen Science and Conservation Action. Conservation Biology, 30(3): 476–486. DOI: https://doi.org/10.1111/cobi.12702
Hochachka, WM, Fink, D, Hutchinson, RA, Sheldon, D, Wong, W-K and Kelling, S. 2012. Data-intensive science applied to broad-scale citizen science. Trends in Ecology & Evolution, 27(2): 130–137. DOI: https://doi.org/10.1016/j.tree.2011.11.006
Hoffman, SW and Potts, WK. 1981. Emigration Behavior of Clark’s Nutcracker. The Condor, 83(2): 162–170. DOI: https://doi.org/10.2307/1367421
Hungerford, HR and Volk, TL. 1990. Changing Learner Behavior Through Environmental Education. The Journal of Environmental Education, 21(3): 8–21. DOI: https://doi.org/10.1080/00958964.1990.10753743
Jackson, MM, Gergel, SE and Martin, K. 2015. Citizen science and field survey observations provide comparable results for mapping Vancouver Island White-tailed Ptarmigan (Lagopus leucura saxatilis) distributions. Biological Conservation, 181: 162–172. DOI: https://doi.org/10.1016/j.biocon.2014.11.010
Jordan, RC, Gray, SA, Howe, DV, Brooks, WR and Ehrenfeld, JG. 2011. Knowledge Gain and Behavioral Change in Citizen-Science Programs: Citizen-Scientist Knowledge Gain. Conservation Biology, 25(6): 1148–1154. DOI: https://doi.org/10.1111/j.1523-1739.2011.01745.x
Kolb, D. 1984. Experiential Learning: Experience As The Source Of Learning And Development.
Krasny, ME and Bonney, R. 2005. Environmental education through citizen science and participatory action research. In: Environmental education and advocacy: changing perspectives of ecology and education. Cambridge: Cambridge University Press. pp. 302–316.
Land-Zandstra, AM, Devilee, JL, Snik, F, Buurmeijer, F and van den Broek, JM. 2016. Citizen science on a smartphone: Participants’ motivations and learning. Public Understanding of Science, 25(1): 45–60. DOI: https://doi.org/10.1177/0963662515602406
Logan, JA, Macfarlane, WW and Willcox, L. 2010. Whitebark pine vulnerability to climate-driven mountain pine beetle disturbance in the Greater Yellowstone Ecosystem. Ecological Applications, 20(4): 895–902. DOI: https://doi.org/10.1890/09-0655.1
Macfarlane, WW, Logan, JA and Kern, WR. 2013. An innovative aerial assessment of Greater Yellowstone Ecosystem mountain pine beetle-caused whitebark pine mortality. Ecological Applications, 23(2): 421–437. DOI: https://doi.org/10.1890/11-1982.1
Marzano, M and Dandy, N. 2012. Recreationist behaviour in forests and the disturbance of wildlife. Biodiversity and Conservation, 21(11): 2967–2986. DOI: https://doi.org/10.1007/s10531-012-0350-y
Maslow, AH. 1943. A theory of human motivation.Psychological Review, 50(4): 370–396. DOI: https://doi.org/10.1037/h0054346
Mavris, C, Furrer, G, Dahms, D, Anderson, SP, Blum, A, Goetze, J, Wells, A and Egli, M. 2015. Decoding potential effects of climate and vegetation change on mineral weathering in alpine soils: An experimental study in the Wind River Range (Wyoming, USA), Geoderma, 255–256: 12–26. DOI: https://doi.org/10.1016/j.geoderma.2015.04.014
McCaffrey, RE. 2005. Using citizen science in urban bird studies. Urban habitats, 3(1): 70–86.
McDonough MacKenzie, C, Murray, G, Primack, R and Weihrauch, D. 2017. Lessons from citizen science: Assessing volunteer-collected plant phenology data with Mountain Watch. Biological Conservation, 208: 121–126. DOI: https://doi.org/10.1016/j.biocon.2016.07.027
McKinley, DC, Miller-Rushing, AJ, Ballard, HL, Bonney, R, Brown, H, Cook-Patton, SC, Evans, DM, French, RA, Parrish, JK, Phillips, TB, Ryan, SF, Shanley, LA, Shirk, JL, Stepenuck, KF, Weltzin, JF, Wiggins, A, Boyle, OD, Briggs, RD, Chapin, SF, Hewitt, DA, Preuss, PW and Soukup, MA. 2017. Citizen science can improve conservation science, natural resource management, and environmental protection. Biological Conservation, 208: 15–28. DOI: https://doi.org/10.1016/j.biocon.2016.05.015
McKinney, ST, Fiedler, CE and Tomback, DF. 2009. Invasive pathogen threatens bird–pine mutualism: implications for sustaining a high-elevation ecosystem. Ecological Applications, 19(3): 597–607. DOI: https://doi.org/10.1890/08-0151.1
McKinney, ST and Tomback, DF. 2007. The influence of white pine blister rust on seed dispersal in whitebark pine. Canadian Journal of Forest Research, 37(6): 1044–1057. DOI: https://doi.org/10.1139/X06-305
Merenlender, AM, Crall, AW, Drill, S, Prysby, M and Ballard, H. 2016. Evaluating environmental education, citizen science, and stewardship through naturalist programs: Naturalists and Citizen Science. Conservation Biology, 30(6): 1255–1265. DOI: https://doi.org/10.1111/cobi.12737
Miles, MB, Huberman, AM and Saldaña, J. 2014. Qualitative data analysis: a methods sourcebook. Edition 3. Los Angeles, London, New Delhi, Singapore, Washington DC: Sage.
NOLS. 2017. NOLS | About NOLS. 2017. Available at https://www.nols.edu/en/about/ [Last accessed 12 August 2017].
Outdoor Foundation. 2017. Outdoor Recreation Participation Topline Report.
Paisley, K, Furman, N, Sibthorp, J and Gookin, J. 2008. Student Learning in Outdoor Education: A Case Study From the National Outdoor Leadership School. Journal of Experiential Education, 30(3): 201–222. DOI: https://doi.org/10.5193/JEE.30.3.201
Pauli, H, Gottfried, M and Grabherr, G. 2014. Effects of climate change on the alpine and nival vegetation of the Alps. Journal of mountain ecology, 7.
Pickering, CM. 2010. Ten Factors that Affect the Severity of Environmental Impacts of Visitors in Protected Areas. AMBIO, 39(1): 70–77. DOI: https://doi.org/10.1007/s13280-009-0007-6
Priest, S. 1986. Redefining Outdoor Education: A Matter of Many Relationships. The Journal of Environmental Education, 17(3): 13–15. DOI: https://doi.org/10.1080/00958964.1986.9941413
Purcell, K, Garibay, C, Louv, R and Fitzpatrick, JW. 2012. A Gateway to Science for All: In: Citizen Science. Public Participation in Environmental Research. 1st ed. Cornell University Press. pp. 191–200. DOI: https://doi.org/10.7591/cornell/9780801449116.003.0014
Raddick, MJ, Bracey, G, Gay, PL, Lintott, CJ, Cardamone, C, Murray, P, Schawinski, K, Szalay, AS and Vandenberg, J. 2013. Galaxy Zoo: Motivations of Citizen Scientists. Astronomy Education Review, 12(1). DOI: https://doi.org/10.3847/AER2011021
Rösner, S, Mussard-Forster, E, Lorenc, T and Müller, J. 2014. Recreation shapes a ‘landscape of fear’ for a threatened forest bird species in Central Europe. Landscape Ecology, 29(1): 55–66. DOI: https://doi.org/10.1007/s10980-013-9964-z
Rotman, D, Preece, J, Hammock, J, Procita, K, Hansen, D, Parr, C, Lewis, D and Jacobs, D. 2012. Dynamic changes in motivation in collaborative citizen-science projects. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. 2012 ACM. pp. 217–226. DOI: https://doi.org/10.1145/2145204.2145238
Saldaña, J. 2013. The coding manual for qualitative researchers. 2nd ed. Los Angeles: SAGE.
Schaming, TD. 2015. Population-Wide Failure to Breed in the Clark’s Nutcracker (Nucifraga columbiana), PLOS ONE, 10(5). DOI: https://doi.org/10.1371/journal.pone.0123917
Schaming, TD. 2016. Clark’s Nutcracker breeding season space use and foraging behavior. PLOS ONE, 11(2): e0149116. DOI: https://doi.org/10.1371/journal.pone.0149116
Shirk, JL, Ballard, HL, Wilderman, CC, Phillips, T, Wiggins, A, Jordan, R, McCallie, E, Minarchek, M, Lewenstein, BV, Krasny, ME and Bonney, R. 2012. Public Participation in Scientific Research: a Framework for Deliberate Design. Ecology and Society, 17(2). DOI: https://doi.org/10.5751/ES-04705-170229
Sibthorp, J, Paisley, K and Gookin, J. 2007. Exploring Participant Development Through Adventure-Based Programming: A Model from the National Outdoor Leadership School. Leisure Sciences, 29(1): 1–18. DOI: https://doi.org/10.1080/01490400600851346
Stepenuck, KF and Green, LT. 2015. Individual- and community-level impacts of volunteer environmental monitoring: a synthesis of peer-reviewed literature. Ecology and Society, 20(3). DOI: https://doi.org/10.5751/ES-07329-200319
Tomback, DF. 1982. Dispersal of Whitebark Pine Seeds by Clark’s Nutcracker: A Mutualism Hypothesis. The Journal of Animal Ecology, 51(2): 451. DOI: https://doi.org/10.2307/3976
Tomback, DF and Achuff, P. 2010. Blister rust and western forest biodiversity: ecology, values and outlook for white pines: Blister rust and western forest biodiversity. Forest Pathology, 40(3–4): 186–225. DOI: https://doi.org/10.1111/j.1439-0329.2010.00655.x
Trumbull, DJ, Bonney, R and Grudens-Schuck, N. 2005. Developing materials to promote inquiry: Lessons learned. Science Education, 89(6): 879–900. DOI: https://doi.org/10.1002/sce.20081
Vasi, IB and Macy, M. 2003. The Mobilizer’s Dilemma: Crisis, Empowerment, and Collective Action. Social Forces, 81(3): 979–998. DOI: https://doi.org/10.1353/sof.2003.0047
Wells, NM, Lekies, KS, Louv, R and Fitzpatrick, JW. 2012. Children and Nature: In: Citizen Science. Public Participation in Environmental Research. 1st ed. Cornell University Press. pp. 201–213. DOI: https://doi.org/10.7591/cornell/9780801449116.003.0015
West, S and Pateman, R. 2016. Recruiting and Retaining Participants in Citizen Science: What Can Be Learned from the Volunteering Literature? Citizen Science: Theory and Practice, 1(2). DOI: https://doi.org/10.5334/cstp.8
Wilson, C. 2014. Interview techniques for UX practitioners: a user-centered design method. Waltham, MA: Morgan Kaufmann.
Wright, DR, Underhill, LG, Keene, M and Knight, AT. 2015. Understanding the Motivations and Satisfactions of Volunteers to Improve the Effectiveness of Citizen Science Programs. Society & Natural Resources, 28(9): 1013–1029. DOI: https://doi.org/10.1080/08941920.2015.1054976