Introduction
Effective engagement is central to participation in community and citizen science (CCS) projects (e.g., ; ; ; ; ; ; ). While these and other studies have explored best practices for engagement in CCS, there is little literature on how to effectively engage participants when expanding a project to include new geographic regions. The motivation for this paper arose as we embarked on a process of growth from a single-region hydrology-focused CCS project to multiple regions to encompass greater hydroclimatic diversity. Scaling in the context of citizen science has been defined as “the extension of existing approaches from a smaller geographical area to a larger one” (). For this work, we consider engagement for CCS to encompass the multitude of ways that observers interact with a CCS project, such as how they become involved, their experiences with the project, and how two-way communication takes place. We define engagement strategy as the collection of planned approaches that helps to meet the goals of the project through two-way interaction and support between project organizers and community observers.
Numerous studies have considered aspects of effective engagement in CCS to initiate and sustain participation through the project lifecycle. For example, De Moor, Rijpma, and Prats López () showed that training and feedback influenced the accuracy of volunteer contributions and helped maintain participation. Similarly, projects that provide trainings tend to observe higher agreement between data collected by citizen scientists and project leaders (). Additionally, Phillips et al. () found social connection to be central to participant experience on CCS projects, including relationships with the projects’ leaders, a sense of community, and being able to pass knowledge to others within the project. Social interactions and communication with project organizers through time can also be a factor in maintaining participation in CCS projects (), as can an effective communication plan that includes opportunities for dialogue between organizers and participants and dissemination of project results (). Studies have also examined factors that help sustain CCS initiatives. For example, Deutsch and Ruiz-Córdova () suggest that the 20-year longevity of their freshwater monitoring project was due in part to devoting time to interacting with volunteers in person, giving recognition for submitting highly credible data, and simplifying data collection by implementing an online database. Cunha et al. () described barriers to long-term participation in a freshwater monitoring project in Brazil, specifically, volunteer time and resources, and the sample sites’ distance or cleanliness.
Scale and scalability have been considered in the CCS literature mainly from the perspective of data collection and scientific outcomes. One of the early motivations for embracing citizen science approaches in research was to enable data collection across large spatial or temporal scales (). Internet and mobile phone technologies have enabled projects to scale up spatially, temporally, and institutionally, for example, by enabling the “platformization” of citizen science (). Recent work affirms the power of scalability in CCS data collection, and highlights actionable benefits for early detection of mosquito-borne disease in public health (; ; ).
However, relatively little has been written in the CCS literature on effective engagement through phases of pivotal growth, and navigating these processes skillfully is important for sustainability and adequate support for participants. A notable exception is Maccani et al. (), who highlighted key factors that support scaling and spreading of citizen science: The impact of the project is clear, there is consistent communication material, and the project engages champions to help attract others to participate. Scale was an important factor that shaped engagement on air quality monitoring projects, and scaling can be supported by information and communication technologies ().
Other fields such as education (e.g., ; ) and public health () have examined the process of scaling programs to maintain their effectiveness and impact. Having a strategic plan for scaling up is a key factor influencing success (), and partnerships are central in scaling new innovations (; ). It is also recommended to pilot or validate a program on a small scale before implementing more widely ().
The challenges of maintaining effective engagement while scaling up are particularly relevant to our CCS project: Mountain Rain or Snow. In our work, we seek to improve the technology used by satellites and hydrologic models to detect and predict precipitation phase (e.g., rain, snow, or mixed) by using crowdsourced, ground-based observations. We collect these data because it is a challenge to accurately model and monitor precipitation phase, particularly at temperatures close to freezing (e.g., ; ; ; ), as falling snow can remain frozen at air temperatures > 0°C because of atmospheric humidity and other environmental factors. We work closely with community observers to crowdsource visual observations of precipitation phase using a browser-based app, which we have found is an effective way of monitoring precipitation patterns (; ). The app automatically timestamps and geotags user-submitted observations of precipitation phase (https://rainorsnow.app/), and sends these data to a secure database. Most relevant to this manuscript is the growth of our project from a small amount of seed funding in a single geographic region to an order of magnitude increase in funding that covers multiple hydroclimatic regimes across the United States (US).
Given our review of the literature and our project’s growth, we identified the following research question to guide our work: In what ways was our engagement strategy—initially developed for a small, single geographic region—effective in meeting project goals when scaling to a larger multi-region project? To address this question, we established goals for effectiveness for each component of the engagement strategy at the outset of the scaling process. We used a mixed-methods approach to collect data to evaluate the project’s ability to meet those goals. We share qualitative factors that help contextualize our results and help understand variations across regions. Using these results, we share lessons from our scaling process that are relevant to the wider CCS community.
Methods
We assessed each component of the strategy, namely recruitment, training, activation, and retention, with respect to goals that were defined at the start of the scaling process. This section describes the four components of our engagement strategy, our programmatic goals for effective engagement, and how we assessed effectiveness of these activities. The scaling period and timeframe of implementation of the engagement strategy was October 7, 2021 to May 3, 2022.
Engagement activities
The project began in 2019 in the Sierra Nevada, focusing primarily on the Lake Tahoe region of California and Nevada, USA (). After two winter seasons, the project received funding to expand to include a greater diversity of hydroclimates, elevations, topographic settings, and snow regimes, providing an opportunity to evaluate the effectiveness of our approach while scaling up. We continued our work in the Sierra Nevada and established three new project regions: the Cascade Mountains in Oregon, the Front Range of the Colorado Rocky Mountains, and the Green Mountains of Vermont (henceforth Sierra Nevada, Cascades, Rocky Mountains, and Northeast, respectively).
Our overarching approach was to replicate the engagement strategy that was used in the Sierra Nevada in the new regions, while crafting a place-based approach to messaging and partnership-building. Single-region engagement involved recruitment, training, activation, and retention, and we note below any changes that were made when this approach was replicated at the multi-region scale. The components of the engagement strategy are integrated as a pipeline, where we intended observers to seamlessly transition from recruitment to training to activation. Below we describe each component of this strategy.
Recruitment
We employed a two-way text message alert system (SimpleTexting) that enabled observers to sign up for the project via text, opt into weather messaging specific for their region, and reply directly to the project team with questions. Observers signed up via text message by sending a region-specific keyword to our project number. To our knowledge, this place-based, text-based recruitment and activation approach is unique for CCS engagement.
When scaling up, our aim was to recruit large numbers of observers (50 or more in each project region) to account for attrition and differing levels of commitment (). Because of the ongoing COVID-19 pandemic, budget limitations, and the challenge of expanding in-person engagement across a very wide geographic area, we took an approach that optimized resources through partnerships, the internet, and other media summarized in Table 1. Our team worked with local weather and winter recreation groups in each region (e.g., OpenSnow, National Weather Service offices, local avalanche forecast teams, and other citizen science projects) to amplify our invitation to participate, meaning those groups shared our invitation with their respective networks via email, e-newsletter, or social media. We also worked with NASA Earth Science, SciStarter, and the Community Collaborative Rain, Hail, and Snow Network (CoCoRaHS; ) for national-scale recruitment via social media and email. Additionally, we offered webinars and in-person presentations related to hydrology and weather, and we engaged with local printed and broadcast news outlets (e.g., News 9 Denver, News 4 Reno).
Table 1
Recruitment activities by region between October 2021–May 2022.
| NORTHEAST | ROCKY MOUNTAINS | SIERRA NEVADA | CASCADE MOUNTAINS | NATIONAL | |
|---|---|---|---|---|---|
| Amplification via email or e-newsletter | 1 | 1 | 2 | ||
| Amplification via social media | 1 | 2 | 2 | 1 | 2 |
| News article or radio | 1 | 1 | 2 | 2 | |
| Webinars and presentations | 1 | 1 | 3 | 1 | |
| Total number of recruitment activities | 3 | 5 | 7 | 2 | 7 |
Training
When observers subscribed to the weather alert system, they automatically received three text messages with instructions on how to submit quality observations through the web app. The training text messages were designed to be simple and to enable observers to learn to submit data in less than three minutes:
Training text message 1: “Welcome to the Mountain Rain or Snow team! Send us observations any time it is raining, snowing, or a wintry mix. To get started, sign up on the web app: https://RainOrSnow.app. This is the first of 3 texts in 3 days to explain how to participate.”
Training text message 2: “You can send observations from anywhere (not just the mountains), just make sure your location services are ON. You can reply to these messages at any time with questions and reach a real human. https://RainOrSnow.app. Text 2 of 3.”
Training text message 3: “How often should you send observations? Whenever it starts precipitating, changes type (rain, snow, or mixed), or every 1–2 hrs. Submit here: https://RainOrSnow.app.”
Observers have the option of replying directly to text messages with questions for one-on-one communication with the project team. For this geographically expansive project, we were not able to conduct in-person trainings with each participant, and a text-based approach disseminated important training information while retaining the option for near real time communication with us.
Activation
For the purposes of this project, we defined activation as the two-way communication that motivates community observers to submit observations once they have signed up. Converting observers from recruitment to actively submitting data is essential for Mountain Rain or Snow to collect sufficient data. We consider this phase distinct from recruitment because we cannot assume that all individuals who sign up go on to submit observations. Our primary activation approach was to issue weather alert text messages 12–48 hours prior to the start of weather events that were forecast to produce precipitation 5–10°C above or below freezing. This allowed us to maintain the visibility of the project for observers, as we presumed many people access their mobile phones throughout the day. After scaling up, our text message lists were geographically segmented, enabling observers to receive messages about the specific weather in their region. An example weather alert from the 2021–2022 season was: “With snow and rain making multiple appearances in the forecast this week, we are looking at an exciting roller coaster of weather. As the precipitation changes, keep us updated through the Mountain Rain or Snow app: https://rainorsnow.app. Thanks!” We also encouraged observers to reply to texts with questions or comments, to which we could respond in near real time.
Retention
Our retention strategy focused on optimizing the volume of communication to strike a balance between meaningful reminders and communication fatigue. We did this by limiting text alerts to one per incoming weather event, no more than once per week. We also increased capacity when scaling up by dedicating a part-time team member to observer support through email, phone calls, and text messages for technology troubleshooting.
Evaluation data collection and analysis
Our mixed-methods assessment used two forms of data collection: (1) a participant feedback survey and (2) participation analytics, that is, data on observer interactions with the project. This mixed-method approach () was designed to answer specific questions about the engagement strategy for Mountain Rain or Snow relative to the program goals established at the outset of the study period (Table 2). The participant feedback survey and participation analytics were intended to complement one another by pairing observers’ self-reported input with their behavior (i.e., what they said about the project, as well as what they did). Additionally, we made qualitative observations of the process that may have shaped outcomes in each region.
Table 2
Programmatic goals and data collection mechanisms for each component of the engagement strategy.
| ENGAGEMENT STRATEGY COMPONENT | PROGRAMMATIC GOAL | DATA COLLECTION MECHANISMS |
|---|---|---|
| Recruitment | R1: Recruit 200 or more community observers project-wide. | Participation analytics: Project-wide total of observer sign-ups. |
| R2: Recruit 50 or more community observers in each project region. | Participation analytics: Region-level observer sign-ups. | |
| R3: Understand how participants found out about the project. | Participant feedback survey questions: How observers heard about the project. | |
| Training | T1: Community observers feel that the series of three introduction messages are helpful to their participation. | Participant feedback survey: Observer perceptions of the helpfulness of the text message training. |
| T2: Community observers submit accurate data following the text-based training. | Participation analytics: Percentage of observations that passed the quality control process. | |
| Activation | A1: Text message weather alerts support observers to submit at least 1,000 observations project-wide. | Participation analytics: Number of project-wide observations submitted through project web app. |
| A2: Text message weather alerts support observers to submit at least 200 observations per target ecoregion. | Participation analytics: Observations submitted through project web app per ecoregion. | |
| A3: Community observers feel that place-based weather alerts are helpful to know when and how to send observations. | Participant feedback survey: Observer perceptions of the helpfulness of the text message alerts. | |
| A4: Determine if there are barriers to submitting observations. | Participant feedback survey: Open-ended question inquiring about barriers to submitting observations. | |
| Retention | Re1: Maximize retention of project participants relative to prior years. | Participation analytics: Percent of observers who remain subscribed to text message alerts project-wide. |
| Re2: Optimize volume of text message communication based on observer satisfaction with text message frequency. | Participant feedback survey: Observer perceptions of frequency of text message alerts. | |
A note on terms: R2 and all participant feedback survey questions use “project region,” which refers to the areas into which observers self-selected to sign up for weather alert text messages. A2 uses “ecoregion,” which denotes the Environmental Protection Agency (EPA) Level III geographic unit of hydrologic analysis () used to categorize the geotagged locations of submitted observations in the target ecoregions (Northeastern Highlands, Sierra Nevada, Southern Rockies, and Cascades). Some project regions are comprised of more than one ecoregion, for example, the Northeast project region encompasses the Northeastern Highlands and the Eastern Great Lakes Lowlands.
Participant feedback survey
The survey collected participant feedback on goals R3, T1, A3, A4, and Re2 (Table 2). All subscribers to weather alerts (n = 1,147) and those on our project newsletter list (n = 1,038) were invited to complete the online survey (Table 3). A large fraction of observers are subscribed to both lists; however, we are not able to quantify the overlap (observers use different identifiers on each platform, e.g., phone numbers for the weather alerts and emails for the newsletter list). The survey was open from April 15–May 15, 2022 on the SurveyMonkey platform. Survey data with a combination of closed- and open-ended questions provide insight into experiences and preferences () of participants in this project. We received an Institutional Review Board exemption for this survey through the University of Nevada, Reno. We analyzed categorical data using frequency tables in Microsoft Excel. We collected regionally segmented responses to track responses of observers in each region. To identify salient themes from the open-ended survey questions, we used inductive coding methods (), which allow the themes to emerge from the comments themselves. Those themes are described in the results. Where needed, we calculated the frequency of each theme to yield percentage of respondents in each code category.
Table 3
Participant survey questions, response options, response types, and evaluation goals to which each question pertains.
| SURVEY QUESTION AND RESPONSE OPTIONS | RESPONSE TYPE | CORRESPONDING GOAL |
|---|---|---|
In which region are you located?
| Discrete-answer with open-ended option for “Other” | Data were used to segment responses to all other questions. |
How did you first hear about the project?
| Discrete-answer with open-ended option for “Other” | Recruitment (R3) |
Were these messages helpful to provide information about how to make observations?
| Discrete-answer with open-ended option to explain how they could be improved | Training (T1) |
We send text alerts when a winter storm is approaching. Are these alerts helpful for you to know when and how to submit observations?
| Discrete-answer with open-ended option to explain how they could be improved | Activation (A3) |
| Are there any barriers to submitting observations? If so, what? | Open-ended | Activation (A4) |
| What are your thoughts on the frequency of the alerts? Just enough
| Discrete-answer with open-ended option for comments | Retention (Re2) |
Participation analytics
Our participation analytics tracked observer interactions with the project, specifically sign-ups (recruitment) and data submitted (activation) for goals R1, R2, A1, A2, and Re1. Analytics is a widely-used approach in digital marketing to understand interactions with a digital product or site, for example, web traffic, page views, and conversion rates (). These tools have also been used outside of marketing to understand, for example, civic engagement on social media () and volunteer recruitment for CCS (). For Mountain Rain or Snow, the use of simple analytics tools through SimpleTexting and our web app enabled us to track interactions over time. Regionally segmented sign-ups to SimpleTexting were downloaded for our project for October 7, 2021 to May 3, 2022. Geotagged and timestamped precipitation phase observations were downloaded from the web app’s admin database for the same time period.
As our objective was to evaluate our activities relative to our goals for engagement, we did not use experimental methods to isolate the effect of engagement activities on observer behavior (a controlled engagement study with large enough sample size would have required time, resources, and personnel beyond what was available). Our results are not intended to be generalizable; however, we believe this experience is relevant to inform the growing body of literature on CCS engagement strategies and recommendations for practice ().
Quality control
To assess goal T2, that our training supports observers to submit quality data, we followed the quality control (QC) process described in our previous work () and report the percentage of observations that passed QC.
Results
There were 443 responses to the participant feedback survey, which represents approximately 38% of observers who were subscribed to the weather alerts at the end of the survey period. Respondents represented all project regions. In total, 35% of respondents were in the Northeast (n = 156), 30% were in the Rocky Mountains (n = 131), 26% were in the Sierra Nevada (n = 114), 4% were in the Cascades (n = 19), and 5% were outside of these regions (“Other”) (n = 23).
The dataset for participation analytics during the study period was comprised of 877 new sign-ups and 13,107 observations submitted. For each of the four components of the engagement strategy, we provide data related to its effectiveness goal for the study period.
Recruitment
In the study period, we exceeded R1, the project-wide goal for recruitment. There were 877 new individuals who signed up for weather alerts. We met goal R2 for recruitment in three of the four project regions in the scaling-up period, recruiting 318 in the Northeast, 310 in the Rocky Mountains, and 219 new observers in the Sierra Nevada. We did not meet the goal (n = 50) in the Cascades, where 30 observers signed up.
Figure 1 shows a time series of observer sign-ups broken down by project region and color-coded to show local and national recruitment events. The majority of recruitment took place in early winter. There were differences in the impact of national- and regional-level organizations on recruitment. National-level recruitment resulted in sign-ups, however the steepest growth followed regional-scale events.
Sign-ups by project region for the study period: (a) Northeast, (b) Rocky Mountains, (c) Sierra Nevada, and (d) Cascades. Vertical yellow lines indicate local-level recruitment efforts, and blue vertical lines indicate national-level recruitment efforts.
Participation feedback survey data provide insight for goal R3 to understand how observers heard about the project, and they demonstrate regional differences (Table 4). Certain recruitment events were exceptionally strong, such as an article in the VTDigger digital newspaper: Within one week of publishing, 255 individuals signed up, which represents 80% of the Northeast observer base. In the Rocky Mountains, 180 individuals signed up within one week of the NWS Grand Junction amplification of a Facebook post (58% of recruitment that season). In the Sierra Nevada, 115 individuals signed up in the week following a Facebook post by the Eastern Sierra Avalanche Center (53% of recruitments that season). Notably, of the 103 respondents who heard about the project through “Other” means, 21% commented that they heard about the project through a partner citizen science project (CoCoRaHS) that amplified the invitation to participate. In the Cascades, we did not have the opportunity to engage with media, and partnerships with local organizations to enable amplification of our posts were limited.
Table 4
How observers heard about the project, by project region (n = 432).
| NORTHEAST (n = 154) | ROCKY MOUNTAINS (n = 123) | SIERRA NEVADA (n = 113) | CASCADE MOUNTAINS (n = 19) | OTHER (n = 23) | ALL REGIONS | |
|---|---|---|---|---|---|---|
| Email or e-newsletter | 21% | 21% | 19% | 26% | 13% | 20% |
| News article or radio | 29% | 22% | 6% | 5% | 4% | 19% |
| Social media (organization) | 17% | 19% | 44% | 42% | 26% | 26% |
| Social media (friend) | 6% | 4% | 11% | 0% | 4% | 6% |
| Word of mouth | 6% | 4% | 5% | 5% | 4% | 5% |
| Other | 21% | 30% | 15% | 21% | 48% | 24% |
Training
Community observers overwhelmingly felt that the series of three introduction messages were helpful to their participation. Of the total respondents, 85% found the training texts helpful (Table 5) to meeting goal T1 in the scaling period.
Table 5
Helpfulness of the text message training guidance by project region (n = 429).
| NORTHEAST (n = 144) | ROCKY MOUNTAINS (n = 122) | SIERRA NEVADA (n = 112) | CASCADE MOUNTAINS (n = 16) | OTHER (n = 21) | ALL REGIONS | |
|---|---|---|---|---|---|---|
| Helpful | 81% | 88% | 90% | 59% | 81% | 85% |
| Helpful but could be improved | 13% | 7% | 6% | 29% | 14% | 10% |
| Not helpful | 7% | 5% | 4% | 12% | 5% | 5% |
The optional open-ended question on how the training texts could be improved revealed that many individuals were not signed up for the weather alerts (47% of the 112 comments received), meaning that these observers were participating in the project without receiving activation messages to remind them to send observations.
To assess goal T2 on data quality, QC was conducted for observations within the targeted ecoregions submitted between October 7, 2021 to May 3, 2022, which constituted 11,906 of the 13,017 total observations. Cumulatively, 96.7% (11,517 of 11,906) passed all QC criteria meeting T2 for the scaling period.
Activation
Place-based weather alerts sent via text message were effective for activating observers in the scaling period. We exceeded A1 for the project-wide activation goal, having received 13,017 observations in total and 11,906 within the target ecoregions. These alerts also supported observers to meet A2 (200 or more observations per ecoregion) in all target ecoregions in all but the Cascades (Table 6).
Table 6
Total number of observations by ecoregion in target areas.
| ECOREGION | TOTAL |
|---|---|
| Northeastern Highlands | 4,159 |
| Sierra Nevada | 1,779 |
| Southern Rockies | 1,746 |
| Central Basin and Range | 1,509 |
| Eastern Great Lakes Lowlands | 1,046 |
| High Plains | 607 |
| Colorado Plateaus | 585 |
| Willamette Valley | 55 |
| Cascades | 31 |
We also received more than 200 observations in three additional ecoregions, namely the Central Basin and Range, Eastern Great Lake Lowlands, and High Plains. The highest concentration of observations occurred in populated areas of target ecoregions (Figure 2). Observers submitted a mean of 12, with a maximum of 340.
Map showing the spatial distribution of crowdsourced observations of rain, snow, and mixed precipitation submitted via the Mountain Rain or Snow web app. The gray dashed outlines denote the ecoregions of interest in our expanded study domain, while the shading represents the number of observations per hexagonal grid cell.
Figure 3 shows time series of the cumulative number of observations in each ecoregion with vertical lines indicating the timing of weather alerts. During the study period, the Northeast project region received 5 alerts, the Rocky Mountains received 9 alerts, the Sierra Nevada received 11 alerts, and the Cascades received 7 alerts. As a reminder, alert timing and frequency were based on the forecast storms in the project region.
Time series of observations per ecoregion: (a) Northeastern Highlands, (b) Southern Rockies, (c) Sierra Nevada, and (d) Cascades. Observations for snow are denoted in black, rain in blue, and mixed precipitation in pink. Vertical orange lines indicate weather alerts.
Goal A3 was met as 82% of community observers felt that place-based weather alerts were helpful to know when and how to send observations (Table 7).
Table 7
Helpfulness of the weather alerts by project region (n = 415).
| NORTHEAST (n = 144) | COLORADO (n = 122) | SIERRA NEVADA (n = 112) | CASCADE MOUNTAINS (n = 16) | OTHER (n = 21) | ALL REGIONS | |
|---|---|---|---|---|---|---|
| Helpful | 75% | 75% | 97% | 88% | 86% | 82% |
| Helpful but could be improved | 9% | 12% | 2% | 0% | 5% | 7% |
| Not helpful | 16% | 12% | 1% | 13% | 10% | 10% |
Of the 113 comments on how alerts could be improved, 73% noted that they were not aware that they could sign up to receive alerts. The remaining comments dealt with recommendations for communication (18%), technology challenges (3%), and requests for more educational content (3%).
To assess A4, we sought to determine if there were barriers to submitting observations in the scaling period. Of the 215 comments, 33% reported no barriers, 22% reported challenges with technology, 15% reported forgetting to submit observations, 10% reported lack of cell coverage or connection, 6% requested educational guidance, 4% made data collection suggestions, 2% noted that precipitation sometimes falls at night while sleeping, and 9% were unable to be coded.
Retention
Of the 877 new individuals who subscribed during the project period, 115 unsubscribed. This yields an 87% project-wide retention rate, and we determined this was satisfactory to meet Re1 to maximize retention for the scaling period.
To assess the volume of text messaging, the survey included a question on satisfaction with the frequency of weather alerts. A large majority of respondents felt the frequency of messages was “Just enough,” meeting our goals for Re2 to optimize the volume of communication, and 1% of all respondents noted that there were too many alerts (Table 8).
Table 8
Satisfaction with the frequency of weather alerts by project region (n = 411).
| NORTHEAST (n = 141) | COLORADO (n = 122) | SIERRA NEVADA (n = 112) | CASCADE MOUNTAINS (n = 16) | OTHER (n = 20) | ALL REGIONS | |
|---|---|---|---|---|---|---|
| Just enough | 66% | 65% | 83% | 81% | 85% | 72% |
| Not enough | 33% | 34% | 16% | 13% | 15% | 27% |
| Too much | 1% | 1% | 1% | 6% | 0% | 1% |
Discussion
This paper reports on a mixed methods approach to evaluating the ways in which our engagement strategy was effective in meeting project goals while geographically scaling up from one to four regions across the US. By focusing on the scaling (or growth) phase, which can be pivotal in shaping projects’ success, this study adds to the growing body of work using mixed methods to examine the effectiveness of engagement strategies for CCS (e.g., ; ; ). It also provides lessons from our scaling process that are relevant to the wider CCS community. Although these results were not intended to be generalizable, they provide insights for the growth process that can be adapted and tested with other projects in other contexts. Our discussion below focuses on themes in this evaluation—as opposed to numerical comparisons that are specific to the project’s unique circumstances—to maintain relevancy for CCS practitioners across diverse topic areas.
The results of the participant feedback survey and participation analytics are highly complementary: Together, they allowed us to holistically assess our engagement strategy relative to the goals that we set at the outset of the process, highlighting quantitative and qualitative factors associated with success across project regions. While scaling up, Mountain Rain or Snow replicated its structure across project regions for each component of the engagement strategy, using place-based messaging and a localized approach to building partnerships. One key lesson is that this approach was effective to meet our project-wide goals for recruitment and activation, but it did not enable us to meet regional recruitment and activation goals in all four project regions (specifically, the Cascades). Replicating the fundamental structure in new regions was important; however, it was not a recipe that we could follow to meet project goals with predictable levels of success. Additionally, the evaluation shows that this novel text-based communication approach was effective to meet project-wide goals for recruitment, training, and activation. This form of communication helped to integrate these three phases as a pipeline, and observer feedback also revealed a gap in that pipeline.
Recruitment
To establish the project in new regions, we used a diversity of recruitment approaches (Table 1), namely, email and e-newsletters, media engagement, webinars and other educational events, and social media, many of which relied heavily on amplification by key partners and sharing of social capital. Prior studies have explored dynamics of recruitment (; ) and reported that broad promotion through diverse media was effective, as confirmed here (Table 4). By tracking national and regional-level recruitment efforts, our work offers an additional approach that can be used for programmatic use evaluation while scaling up.
The qualitative strength of partnerships was a feature of the project regions with highest success, specifically where we were able to work with partners to set up amplification opportunities. Our differing levels of recruitment across regions (Figure 1) may be explained by the reach of different partners’ amplification efforts and audiences of the media outlets with whom we engaged. Collaborating with other CCS projects for recruitment was particularly effective for Mountain Rain or Snow, likely because of alignment in interests with their existing participant base. The central role of partnerships in scaling this project is consistent with previous work in CCS (), education (), and public health ().
Our recruitment approach targeted low-cost internet-based tools such as email and social media. This was effective in areas such as the Sierra Nevada where relationships with partners were already strong; however, it did not enable Mountain Rain or Snow to get traction in the Cascades where did not have prior connections with many organizations or media. For our work, we concluded that in-person recruitment and partnership-building is important for these areas. This is a consideration for other projects facing budget allocation decisions.
Training
Our training strategy provided an expedient way to reach large numbers of observers in new locations. The text-based approach was appreciated by observers while using training time and resources wisely. Training was intended to immediately follow observer sign-up; however, qualitative survey data revealed a leak in this pipeline: some observers accessed the web app directly to submit observations and did not receive the training. As a result, we determined that it would be helpful to duplicate the training content through multiple channels.
Regarding the goal related to data quality, 96.7% of the observations submitted in the study period in the ecoregions of focus passed QC, comparable to the rate our previous study of 96.5% (). This study confirms that a text-based training method was effective to support Mountain Rain or Snow observers to provide quality data while scaling up.
Activation
Converting observers from sign-up to data collection is essential for Mountain Rain or Snow to gather sufficient precipitation phase observations. By tracking activation of users as distinct from the recruitment process, this study offers an evaluation approach that sheds light on participants’ levels of activity through time. This enabled us to develop targeted approaches for our goals in each phase (i.e., reach new observers and then energize them to submit observations).
When it comes to data collection and the process of scaling up, a text-based approach enabled place-based communication with large numbers of people while reaching increasingly diverse hydroclimatic zones and ecoregions. An important lesson from this evaluation is that a text-based approach retains the option for one-on-one communication with volunteers in near real time. For projects with limited resources, this medium can stretch time or funds invested in communication without sacrificing opportunities for direct connection with project participants. This work reinforces the important role of messaging and timing to support CCS volunteers ().
This work offers a valuable reminder that CCS participant audiences are not monolithic. The large majority of respondents to the survey noted that the place-based weather alerts were helpful to know how and when to submit observations. However, not all participants relied on place-based weather alerts to remind them to send observations, as some respondents reported that they were not signed up for text alerts. Some observers were not dependent on these messages to send observations, while others underscored the importance of text-based reminders to participate. There is a need for further exploration of the heterogeneity of CCS participant groups and tools to meet their needs.
Additionally, we used qualitative survey data to understand barriers to submitting observations. Of those who reported barriers, challenges with technology and remembering to submit observations were most frequent. Complicated or unclear technological tools have also been reported as a participation barrier for community members involved in an ecology-based CCS program (). One study found that, after making online user registration optional, participants who were registered submitted proportionally more reports than previously (). Other factors, such as time and knowledge, have been reported as barriers to participation (). For our project, we concluded that user-facing technologies are a priority area of focus through the scaling process, as recurring glitches or technologies that are not user-friendly pose barriers to participation. These findings underscore the importance of considering CCS participants’ design and workflow preferences when selecting and developing technology tools to optimize useability (). Likewise, open-ended responses to the survey also requested educational content and guidance, which points to the need for exploration of observer preferences on the science behind the project and reporting back scientific results.
Retention
For this work, we defined retention as the percent of individuals who remained subscribed to text alerts. Project-wide retention was 87%, a level consistent with the 88% retention rate that we reported following local-scale project implementation in the Sierra Nevada (). One possible reason for this high retention rate is that remaining subscribed to alerts is convenient and simple, and participation in the project does not involve a large time investment. We note that our retention numbers do not provide insight into levels of activation through time, and the relationship between activation and retention is worth investigating in the future.
Messaging frequency was found to be “Just enough” by a large majority of Mountain Rain or Snow observers. Frequency of communication with observers is important for CCS projects (; ). Mountain Rain or Snow text message updates aim to align the frequency of communication with the phenomena of interest (winter weather). Interestingly, 15% of open-ended comments to this question noted that remembering to submit observations was a challenge, reinforcing the importance of reminders for a CCS project like this. We speculate that alignment with the pace of change in weather and the convenience of text reminders were helpful for observers to remember and to understand how to continue participating as the project scaled up.
Another lesson from this study is the key role that feedback on communication preferences plays in CCS engagement. Given that nearly one third of observers noted that there were “Not enough” text messages, we underestimated the number of weather alerts that many observers would like to receive. This survey input helped to challenge our assumption that “less is more” when it comes to observer appetite for communication (i.e., that we should limit to one text alert per week per winter weather event) and helped us to fine-tune the frequency of communication with our observers. We recommend that projects seek input from observers on their communication preferences and adjust accordingly, especially if more educational or scientific report-back content is requested.
Conclusions and Lessons for Practice
Expansion of the Mountain Rain or Snow project as it scaled up from one region to four provided an opportunity to evaluate each component of our engagement strategy, namely recruitment, training, activation, and retention. This work used two sources of data, a participant feedback survey of 443 observers and participant analytics of sign-ups and observations submitted, to understand the ways in which our engagement strategy was effective. We offer the following conclusions and lessons for practice that may be relevant to the growth and expansion of other CCS projects.
In this pivotal growth period, it was valuable to create a strategy dedicated to engagement with goals and mechanisms to assess the effectiveness of our approach. Mountain Rain or Snow’s strategy involved replicating the project structure of our previous single-region effort in new regions while simultaneously customizing messaging and partnerships in a place-based way. In three of four regions, this approach was effective in scaling up Mountain Rain or Snow, based on meeting project-wide goals for recruitment, training, activation, and retention.
Lessons from our recruitment experience underscore the value of partnerships and diverse approaches to amplification, which made a remarkable difference toward meeting recruitment goals. Many respondents noted that they heard about Mountain Rain or Snow through a partner CCS project, and as such, reciprocal support and sharing social capital can be helpful to reach audiences with aligned interests.
This evaluation shows that text message communication with participants was an effective and cost-efficient way to provide training and support observer participation over large geographic areas without losing the opportunity for one-on-one communication. Survey data confirmed that this novel approach to training was helpful and appreciated by observers, and they overwhelmingly found activation text alerts helpful to understand when and how to participate.
Directly asking participants for feedback on communication preferences can provide valuable insights for optimizing the volume and frequency of messaging. In our case, it also revealed unexpected results that challenged our assumptions about our messaging frequency and the need to send a greater number of activation messages. Qualitative responses identified a gap in how we were reaching our observer audience with training and activation messages and the need for reinforcing training through multiple channels.
This study identified several areas for future investigation. First, retention as defined here does not provide insight into levels of activation through time, and the relationship between recruitment and activation may be a rich area of study for CCS engagement. Likewise, activation differed across regions in ways that we did not uncover in this study. What energizes people to participate, and what value can projects provide to participants to sustain activation? Additionally, respondents to open-ended questions noted a need for science communication and educational content. It is important to explore observers’ demands for science communication as part of giving back, including for scientific report-backs and educational content. Finally, controlled or A/B testing of engagement strategies such as these could quantitatively and determine the effect of activities on project outcomes in a more robust way.
This type of evaluation that assesses engagement relative to project goals can support evidence-based design of engagement strategies for project growth and expansion. We anticipate that these lessons contribute to best practices for engagement for growth and sustainability of CCS programs.
Data Accessibility Statement
All precipitation observation data are available at: https://github.com/SnowHydrology/MountainRainOrSnow.
The quality control process and observations by study region are available here: https://github.com/SnowHydrology/MountainRainOrSnow/tree/main/manuscripts/collins_et_al_cstp.