Recruiting and retaining participants have repeatedly been identified as the greatest challenges for citizen science programs (West and Pateman 2016; Frensley et al. 2017; De Moor et al. 2019). A growing body of research, designed to maximize recruitment and engagement efforts, has investigated participants’ motivations for initially joining citizen science projects as well as the factors that keep them engaged over time (Eveleigh et al. 2014; Nov et al. 2014; West and Pateman 2016; Frensley et al. 2017). In a similar vein, increasing focus has been placed on identifying effective strategies for engaging participants and sustaining their activity in a project. Typical strategies recommended to sustain engagement in citizen science projects include providing participants with prompt and regular feedback, offering support, creating a sense of community, and expressing gratitude to participants for their contributions (Ohrel and Register 2006; Cooper et al. 2007; Prysby and Super 2007; Crall et al. 2017, Frensley et al. 2017; De Moor et al. 2019; Davis et al. 2020).
Citizen science programs are frequently managed by small teams, often engaging volunteers across a large region. As such, email or text messages are a commonly used tactic for reminding participants to take a desired action such as collecting an observation (e.g., Black 2009; Beaubien and Hamann 2011; Birkin and Goulson 2015; Arienzo et al. 2021; Lopez 2021). Reminder messages prompting volunteer participants to log observations are especially important when the phenomena under observation require attention at a specific time, such as immediately following a weather event or when plants and animals are undergoing seasonal transitions. Crimmins et al. (2014) reported an increase of nearly 200% in observations of biological phenomena submitted following email messages. Similarly, Arienzo et al. (2021) reported a “marked increase” in the number of precipitation observations reported following text notifications to report, as well as a low error rate in reports. A clear understanding of the impact of messages on data quantity and quality, as well as the associated costs, can help citizen science programs determine how many resources to expend on these activities.
Through the plant and animal phenology observing program, Nature’s Notebook, the USA National Phenology Network (USA-NPN) engages volunteer and professional observers in documenting the timing of seasonal events such as leaf-out, flowering, and egg hatch in plants and animals. The primary objective is to document when individual plants or animals at a location transition from not expressing a phenological state, such as open flowers, to expressing that state, and then back to no longer expressing that state, over the course of a season. The phenology observations contributed to Nature’s Notebook are used to document how plants and animals and ecosystems are responding to rapidly changing climate conditions (Howard 2018; Brenskelle et al. 2019) to determine the specific conditions that cue species to transition from not expressing to expressing a state (Mazer et al. 2015; Crimmins et al. 2017a; Elmendorf et al. 2019) and to guide the timing of management activities in a range of applications (Wallace et al. 2016; Emery et al. 2020).
Science and management applications strongly benefit from a high degree of accuracy and precision in the timing of phenological transitions. One approach USA-NPN staff use to encourage precise and accurate records of phenological transitions contributed through Nature’s Notebook is to message participating observers immediately prior to when their plants are expected to undergo phenological transitions. Surveys of Nature’s Notebook participants have indicated that the most common motivations among observers for persisting with the program were the desire to contribute to a valuable, national-scale effort, personal enjoyment for observing plants and animals, and an interest in learning (Crimmins et al. 2010; Goldsmith et al. 2019). Consequently, our messages emphasize the importance of regular observations, the significance of negative data (“no” reports), and the value of all observations to science. Further, our assumption has been that sending email messages to observers just prior to when an event such as leaf-out or bloom is expected to occur leads to more accurate and precise reports of the events. We were able to test this assumption because of unplanned interruptions in the messages sent to observers. In each of three recent years we sent messages to observers, scripts responsible for sending the messages stopped functioning at some point in the season, resulting in dozens of observers not receiving messages.
In this study, we evaluate the impact of carefully timed messages on two aspects of data quality: accuracy and precision. We ask: 1) Does the accuracy or precision of leaf-out and flowering observations vary depending on whether the volunteer observer receives an email-based prompt? 2) Does it matter if the participant opens the message, or is simply seeing the email reminder in their inbox sufficient to influence their observing behavior? Our focus for this study was observers tracking lilacs through Nature’s Notebook. Unique in comparison to other studies addressing the effects of messaging program participants, we focus on how the messages impacted facets of data quality and quantity.
The USA-NPN generates forecasts of leaf-out and flowering in lilacs up to six days in the future as a part of the Spring Leaf and Bloom Index models (Schwartz 1997; Schwartz et al. 2006; Crimmins et al. 2017b). These models use daily temperature and weather events as inputs to predict when individual lilacs will first undergo leaf-out and bloom at a location (Schwartz et al. 2006; Schwartz et al. 2013; Ault et al. 2015). The Spring Leaf Index is the average of three individual models, one of which predicts leaf-out in lilacs. Similarly, the Spring Bloom Index is the average of three individual models, one of which predicts bloom in lilacs. Each night, short-term gridded minimum and maximum temperature forecast maps are downloaded and used to update short-term forecasts of spring activity—as reflected in the Spring Leaf and Bloom Index models—and lilac leaf-out and flowering.
Participants in Nature’s Notebook collect repeated observations of what they see on individual plants over the course of the season. The steps to tracking plant phenology using Nature’s Notebook are: create a user account, register a site, and then register one or more individual plants from the list of species available for monitoring (Rosemartin et al. 2014). Once these steps have been achieved, participants can submit phenology observations.
The observation protocols in Nature’s Notebook are status protocols, in that each time a participant records an observation, they indicate whether they do or do not see a phenophase, such as leaf-out or flowering, being expressed (Denny et al. 2014). Observers are encouraged to report frequently when things are changing quickly, to capture the date of transition—when a plant shifts from the phenophase not being expressed to when it is being expressed—with as much precision as possible. We recommend at least weekly observations, and encourage observers to report two to three times a week when things are changing quickly. Participants may join the program at any time and may track the phenology on as many plants as they wish, though the majority of participants tracking lilacs record observations on a single lilac plant.
Hundreds of Nature’s Notebook participants track leaf-out and flowering in lilacs across the United States. To prompt Nature’s Notebook observers to capture the transitions from leaf buds not broken to leaf buds broken and flower buds not broken to flower buds broken with as much accuracy and precision as possible, we email observers who have registered a lilac in their Nature’s Notebook accounts three days prior to when leaf-out is expected to occur based on the lilac Spring Leaf Index model. We also message these observers three days prior to when flowering is expected to occur based on the lilac Spring Bloom Index model. In each message, we emphasize the importance of documenting “no” observations (lilacs not yet leafed out, not flowering) as well as capturing the transition from the phenophase not yet present (“no”) to the phenophase present (“yes”) in as few days as possible. To prepare observers to receive these messages, we send a message at the beginning of the season explaining this campaign and the importance of documenting the timing of transition accurately and precisely.
Nature’s Notebook observers that register a lilac are automatically added to the list of message recipients, in accordance with terms to which they agree when they join the program. However, participants can easily and permanently opt out of the messages using the unsubscribe link that appears at the bottom of every message.
Forecasts of lilac leaf-out and bloom are updated nightly based on daily minimum and maximum temperature data (Crimmins et al. 2017b). Locations of registered lilac plants under observation in Nature’s Notebook are intersected with these forecast maps through a nightly process. When a map pixel containing one or more lilacs is predicted to reach the conditions associated with either leaf-out or flowering within the next three days, an email message is sent to the email address associated with that lilac plant. Messages are sent using Constant Contact email marketing software. This workflow requires a script on USA-NPN servers to connect with the Constant Contact application programming interface (API) to send messages to the proper individuals on a daily basis. Leaf messages always precede bloom messages, as the environmental conditions associated with leaf-out are always met prior to conditions associated with bloom (Schwartz 1997). Leaf-out messages are typically sent between mid-January and the end of May, and bloom messages are typically sent between the end of January and the end of June.
In this study, we focus on participants that submitted phenology observations in the years of the study (2018, 2019, and 2020) (USA National Phenology Network 2021). Like many citizen science programs, we struggle to sustain active participants in Nature’s Notebook, and participants leave the program for a wide range of reasons. Here, we focus on how participants that were active in the program were affected by messages encouraging them to observe.
Our assumption in this evaluation is that large differences between predicted and observed dates for the phenomena reflect inconsistent reporting on the behalf of observers rather than poor forecast performance. The Spring Leaf and Bloom Index models perform well at predicting leaf-out and flowering in lilacs. A recent evaluation comparing observations of leaf-out and flowering in cloned lilacs (1981–2017) to Spring Index predictions of leaf-out and flowering in lilacs reported a root mean squared error (RMSE) of 11.97 days for leaf-out and 6.46 days for bloom (Gerst et al. 2020). Further, 47% of reports of leaf-out in cloned and common lilacs submitted to Nature’s Notebook (1981–2021) were within one week of the predicted date of leaf-out made with the lilac model; for bloom, this figure climbed to 60% (USA National Phenology Network, unpublished data). Bias in both models is less than two days. Accordingly, we assess accuracy in observers’ reports by comparing their reports of when leaf-out and flowering occurred to when the events were predicted to occur within three days. We evaluate precision in the report of onset by evaluating the number of days between when an observer previously reported the phenomenon (leaf-out or flowering) as not occurring and when they first reported it as occurring.
We constrained our analysis to focus only on instances when the model predicted leaf-out or bloom prior to when the observer reported the event. If observers logged observations of leaf-out or bloom prior to receiving the messages, the evaluation does not answer the question of whether messages prompting participants to observe their plants have the intended effect of improving accuracy and precision in reports of leaf-out or flowering.
To test for differences in accuracy, we calculated the number of days between the predicted and observed day of leaf-out or flowering for each report submitted in each year. Because we constrained our analyses to instances where the model predicted leaf-out or bloom prior to the observer’s report, the calculated values were either zero or positive, with the greatest number of values at or close to zero, and a decreasing number of values farther from zero. We used the outer fence method (3 × the interquartile range) to identify and remove outliers. As this resulted in a highly skewed dataset, we compared the values among the three groups (received and opened the message; received but did not open the message; did not receive the message) using a Kruskall-Wallis rank sum test. We then used the Wilcoxon rank sum test for multiple pairwise comparisons to determine statistically significant pairs of means.
To test for differences in precision, we calculated the number of days since the last reported “no” for the “breaking leaf buds” or “open flowers” phenophases and the first reported “yes” for each report. As above, we used the outer fence method to identify and remove outliers. Next, for both the leaf and bloom datasets, we compared the values among the three groups (received and opened the message; received but did not open the message; did not receive the message) using a Kruskall-Wallis rank sum test. We then used the Wilcoxon rank sum test for multiple pairwise comparisons to determine statistically significant pairs of means.
The total number of observers that contributed lilac leaf-out observations was 257 in 2018, 305 in 2019, and 250 in 2020. The total number of observers that contributed lilac bloom observations was 200 in 2018, 241 in 2019, and 195 in 2020. After constraining the records to instances when the model predicted leaf-out or bloom prior to when the observer reported the event, the number of lilac observers that received and opened, received but did not open, and did not receive messages varied between leaf-out and bloom messages and across the three years (Table 1). However, the number of observers was fairly balanced across the three categories when all years were combined.
|Leaf messages||Received and opened||53 (58%)||48 (48%)||21 (62%)||122|
|Received but did not open||39 (38%)||51 (50%)||13 (13%)||103|
|Did not receive||41 (32%)||39 (30%)||50 (38%)||130|
|Bloom messages||Received and opened||66 (77%)||34 (47%)||6 (100%)||106|
|Received but did not open||20 (34%)||38 (66%)||0 (0%)||58|
|Did not receive||0 (0%)||46 (37%)||77 (63%)||123|
Message open rates, defined as the proportion of individuals who received the messages and opened them, ranged from 47% to 100% for the leaf and bloom cohorts in each year. Email open rate is a commonly used measure of engagement among recipients of targeted email messages. In 2021, industry standard email open rates for nonprofits were 25.5% for agriculture, forestry, fishing, and hunting (Campaign Monitor 2021).
For the leaf messages, there was a highly significant difference in the duration between when leaf-out was predicted and reported among the three groups (X2 = 14.56, p < 0.001, df = 2, Kruskall-Wallis rank sum test). The duration in days between when leaf-out was predicted and reported was nearly five days shorter in the “opened” group (mean = 8.1 days ± SD = 8.5 days) than the “did not open” group (13.4 ± 13.6 days, p = 0.019) and over a week shorter than the “did not receive group (16.0 ± 16.0 days, p < 0.001, Wilcoxon multiple comparisons test, Figure 1a).
There was a marginally significant difference in the duration between when bloom was predicted and reported among the three groups (X2 = 05.33, p = 0.07, df = 2, Kruskal-Wallis rank sum text). The number of days between when leaf-out was predicted and reported was slightly more than one day longer in the “opened” group (5.5 ± 4.6 days) than the “did not open” group (4.2 ± 3.1 days, p = 0.085, and the “did not receive” group (4.9 ± 4.6 days, p = 0.085, Wilcoxon multiple comparisons test, Figure 1b).
The number of days between when the observer reported a “no” and when they first reported “yes” to leaf-out was significantly different among the three groups (X2 = 15.62, p < 0.001, df = 2, Kruskal-Wallis rank sum test). The duration between when the observer reported a “no” and when they first reported “yes” to leaf-out was nearly two days shorter in the “opened” group (5.2 ± 5.5 days) than in the “did not open” group (7.0 ± 5.4 days, p < 0.001, Tukey multiple comparisons test) and about one and a half days shorter than the “did not receive” group (6.5 ± 5.9 days, p = 0.01, Wilcoxon multiple comparisons test, Figure 2a).
The number of days between when the observer reported a “no” and when they first reported “yes” to open flowers was marginally significantly different among the three groups (X2 = 5.01, p = 0.08, df = 2, Kruskall-Wallis rank sum test), though the duration between when the observer reported a “no” and when they first reported “yes” to open flowers was not significantly different among the three groups (Figure 2b).
In this study, we took advantage of an unplanned interruption in email prompts sent to participants in a citizen science program to formally test whether these messages impacted participants’ activity in the program. The leaf-out messages, which are the first messages observers receive in the spring, led to significant improvements in both accuracy and precision of observations (Figures 1a, 2a). Accuracy and precision of leaf-out reports submitted to Nature’s Notebook was improved when observers received and opened the messages. We conclude from these results that the content of the messages—accessed by individuals that both received and opened them—had substantial impact on the observers’ behavior, motivating them to carefully document and report on the leaf status of their lilacs.
Results were slightly different for the bloom messages, which were sent several weeks after the leaf-out messages. Accuracy values were notably better for bloom messages across all groups compared with leaf-out messages (Figure 1); the number of days between predicted and reported leaf-out for the “opened” group was nearly twice that of the bloom messages, and for the “did not open” group was nearly four times higher in the leaf-out messages. One reason for these patterns could be that observing leaf-out earlier in the season served to focus observers’ attention and led to more accurate reports of flowering, regardless of message status. It is possible that by the time bloom messages are received, observers are in the habit of checking their lilacs, and the messages have a diminished impact on their behavior. Accuracy improved from leaf-out to bloom regardless of message status, suggesting that observers paid closer attention to flowering in their lilacs after they started to see the appearance of leaves rather than because of the bloom messages. This finding is consistent with other studies evaluating the quality of observations reported by Nature’s Notebook participants, which showed that observers demonstrate greater performance reporting phenophases later in the season than the first phenophases (Fuccillo et al. 2015, Crimmins et al. 2017a).
Interestingly, accuracy was the lowest among observers that opened the bloom reminder messages (Figure 1b); we do not have a ready answer to explain this result. Upon reflection, we realized that our message prompting observers to submit reports of blooming in their lilacs included a photo with mostly open flowers. We propose that perhaps observers that opened this message and saw this photo interpreted that their lilac needed to be fully in flower to report a “yes” to blooming. We will reconsider our choice of photos and phrasing in future bloom messages.
The precision values for reports of bloom were similar in magnitude to those of leaf-out and were not statistically significant among the three groups (Figure 2). This finding suggests that observers may not increase the frequency of their data collection efforts even when asked repeatedly to do so. It seems that the first messages sent—pertaining to leaf-out—had a positive impact, but subsequent messages did not. Citizen science programs implementing reminder messages to prompt activity in participants may wish to place special emphasis on the content of their first message, as subsequent messages may have a lesser impact.
Overall, the messages had a clear and positive impact on the accuracy and precision of reports of leaf-out and flowering in lilacs. Implementing this approach entailed the up-front investments of staff time to craft the messages and to create and maintain the scripts that populated and sent the messages each day as well as the cost of the email marketing software subscription. We estimate the time spent on these activities to be approximately 10 hours to create the script that interfaces between the daily lilac leaf- out and bloom maps and the registered lilac observer locations, and approximately 20 hours each year to maintain the script, test the Constant Contact API, and draft and send the email messages.
Open rates for our directed email messages were much higher than the standards for the most aligned industries: our rates ranged from 47% to 100%, compared with industry standards around 25%. This may stem from the fact that participants in Nature’s Notebook engage in the program primarily to contribute to science and to learn; such motivations may naturally lead to a greater likelihood of opening messages. This result suggests that the messages are appreciated by Nature’s Notebook participants and are worth the costs involved to prepare and send. The high open rates indicate that we are sharing content with the recipients that they wish to receive. Further, because participants can opt out of the messages at any time, we can rest easy that we are not bombarding our participants with unwanted emails.
Because this approach results in a positive impact on data quality at a relatively low cost, we plan to continue to send messages when we can predict the timing of an event using a model like the Spring Indices. However, as this study reveals, this approach is not failsafe. In some seasons of our study, the scripts failed multiple times, resulting in dozens of participants not receiving messages. One approach to mitigate this issue would be to implement a nightly script that sent a notification to our team if no emails were sent on a particular day; this would provide a prompt to ensure scripts were still functioning properly. Other citizen science programs planning to rely on automated scripts should consider similar failsafe measures.
Short message service (SMS) text messages are one alternative approach to the email-based messages we are currently sending. Recent surveys suggest growing preference for text messages over email, especially for messages coming from businesses or other institutions (PC Magazine 2020). This pattern appears to hold for programs individuals voluntarily join as well. For example, participants in a smoking cessation program expressed more positive feelings toward text messages than email messages (Abroms et al. 2012). Text-based notifications offer direct contact through devices that are nearly always on and in one’s possession. However, with the proliferation of mobile apps, users may receive over 60 notifications each day (Pielot et al. 2014), which can lead to message fatigue and even to users unsubscribing.
Another option for engaging program participants, particularly for those programs with mobile apps, are app-based notifications, or push notifications that can be sent by mobile app, even when the app is not open. Nature’s Notebook participants have the option to either log observations on paper datasheets and transcribe these into an online interface or to use the Nature’s Notebook mobile application. Use of the Nature’s Notebook mobile app in place of the web-based interface to log phenology observations has grown rapidly in recent years, from just over 20% of observations submitted via the app in 2017 to nearly 60% in 2021. Accordingly, sending app-based notifications to Nature’s Notebook participants to go out and log observations at a particular time as a result of a forecast, is one alternative we may consider for engaging observers. In a study of diet apps, participants were tolerant of receiving multiple push notifications a day, though engagement with the messages declined over time (Freyne et al. 2017), suggesting push notifications may offer an effective solution for short-term campaigns.
The question of whether email-based messages have a positive impact on participants’ actions—and therefore, data quality—has applicability to many citizen science programs, especially in cases where documenting the precise time an event occurred is important. Findings from the healthcare field demonstrate that program participants’ reactions and feelings regarding email or text-based messages prompting a particular action are variable. For example, Woolford et al. (2012) found that text messages sent to adolescents in a weight-loss program were very appreciated, and the recipients were “very enthusiastic” about such forms of communication (p. 2382). In contrast, Cherubini et al. (2020) found that app-based messages intended to motivate people to take a walk did not result in increased physical activity and annoyed participants. However, within the citizen science realm, reminder messages are generally welcomed: Project participants have indicated that such reminders are helpful (Tang and Prestopnik 2019; Martin and Greig 2019; Shelton et al. 2020), and participants in one project even requested regular reminders to observe (Martin and Greig 2019).
A key takeaway from this analysis is that the first messages participants received during the active data collection season had the largest impact on their activity. This finding is consistent with other research that has reported declining engagement over time with subsequent messages (Freyne et al. 2017). We do not know whether participants that opened the messages read through to the end; the truth may be that that simple, brief notifications that could be achieved via SMS text messages or app-based notifications can have the intended effect. However, the significant differences in both accuracy and precision between participants that opened the messages and those that received but did not open the messages suggests that the content of the messages had some influence on observer behavior. It may still be possible to achieve the desired effect in these situations through brief text messages or app-based notifications, though the content of these messages should be carefully considered to contain the critical information needed by participants. Establishing the optimal format and content of messages encouraging citizen science project participants to take a particular action is an area ripe for further work.
In this analysis, we assume that the models used to predict the timing of leaf-out and flowering in lilacs reflect real-world conditions well enough that differences between predicted and reported values can be interpreted as observer error. In reality, model performance is unknown. Previous studies (e.g., Gerst et al. 2020) have endeavored to evaluate model performance, though this has been achieved by comparing the predictions to reports of leaf-out and flowering contributed by volunteer observers. As such, neither dataset can be considered correct.
This limitation has the potential to affect our accuracy results. If the model predictions are not correct, then evaluating whether the number of days between the predicted and reported values for leaf-out or flowering, as we do in the present study, is fraught. The difference in accuracy values among the three groups in this study—that is, the differences in the number of days between the predicted and reported dates of an event—are equal to or greater than the model RMSE, suggesting that despite the potential error present in the predicted values, we still see an impact of messaging on when observers report the event to Nature’s Notebook. Both the model performance and the impact of messaging on observer behavior could be assessed through alternative approaches with more rigor, such as by documenting leaf-out and flowering in individual lilac plants using automated repeat photographs (e.g., Crimmins and Crimmins 2008).
In this study, we take advantage of a hiccup in our scripts to ask and answer questions pertaining to the impact of messaging program participants. We did not impose a formal sampling design or randomize the treatments of who received a message and who did not. Accordingly, our findings are correlational, and as such, we cannot establish that the messages (or lack thereof) caused participants to observe more frequently or with greater accuracy. The patterns we report here may be the result of social dynamics or other confounding variables not tracked in this study. A formally designed evaluation with a randomized study design would be better suited to establish these relationships with more rigor.
In this study, we took advantage of unplanned interruptions in auto-generated emails to evaluate their impact on influencing the behavior of citizen science program participants. In our project, we message participants three days prior to when lilac leaf-out and bloom are expected to occur based on forecast models. We found that these messages had a positive impact on observers’ accuracy and precision in capturing onsets of breaking leaf buds and open flowers in lilacs, and in our assessment, are worth the time and effort involved in preparing and sending the messages.
Specifically, we found that the accuracy in observers’ reports of leaf-out and bloom was improved even if the observers did not open the messages. However, the precision of their reports was significantly improved only when they opened the messages, suggesting that they benefited from nuanced content contained in the message.
Our findings underscore the importance of communication with project participants, and of timely messages to prompt action. Our findings also suggest that different forms of messaging (SMS text, in-app notifications) may be viable alternative approaches for prompting an intended action. If the aim of a message is to simply prompt a participant to log an observation, a brief message may suffice. However, in other cases where more nuanced detail is necessary to guide the participant’s actions, email messaging, which allows for the inclusion of more information and photos, may be the best choice of mechanism.
Data files and code are available at https://doi.org/10.5281/zenodo.6618222.
We are very grateful to the hundreds of Nature’s Notebook observers tracking the phenology of lilacs and other plants and animals across the country. Lilac observations were provided by the USA National Phenology Network and the many participants who contribute to its Nature’s Notebook program. We thank R. Lee Marsh for assistance in preparing data files.
This work was supported through a Cooperative Agreement from the U.S. Geological Survey [G18AC00135], a Cooperative Agreement from the U.S. Fish & Wildlife Service [F19AC00168], and National Science Foundation awards [DEB-2017831, DEB-2017848, & DEB-2017815].
The authors have no competing interests to declare.
TMC and EEP conceived of and designed the study; EEP led data collection and management; TMC led analysis; TMC drafted the manuscript, and EEP and TMC reviewed and revised the manuscript.
Abroms, LC, Ahuja, M, Kodl, Y, Thaweethai, L, Sims, J, Winickoff, JP and Windsor, RA. 2012. Text2Quit: Results from a pilot test of a personalized, interactive mobile health smoking cessation program. Journal of Health Communication, 17: 44–53. DOI: https://doi.org/10.1080/10810730.2011.649159
Arienzo, MM, Collins, M and Jennings, KS. 2021. Enhancing engagement of citizen scientists to monitor precipitation phase. Frontiers in Earth Science, 9: 617594. DOI: https://doi.org/10.3389/feart.2021.617594
Ault, T, Schwartz, MD, Zurita-Milla, R, Weltzin, JF and Betancourt, JL. 2015. Trends and natural variability of spring onset in the coterminous United States as evaluated by a new gridded dataset of spring indices. Journal of Climate, 28(21): 8363–8378. DOI: https://doi.org/10.1175/JCLI-D-14-00736.1
Beaubien, EG and Hamann, A. 2011. Plant phenology networks of citizen scientists: recommendations from two decades of experience in Canada. International Journal of Biometeorology, 55: 833–841. DOI: https://doi.org/10.1007/s00484-011-0457-y
Birkin, L and Goulson, D. 2015. Using citizen science to monitor pollination services. Ecological Entomology, 40: 3–11. DOI: https://doi.org/10.1111/een.12227
Black, JM. 2009. River otter monitoring by citizen science volunteers in Northern California: social groups and litter size. Northwestern Naturalist, 90(2): 130–135. DOI: https://doi.org/10.1898/NWN08-22.1
Brenskelle, L, Stucky, BJ, Deck, J, Walls, R and Guralnick, RP. 2019. Integrating herbarium specimen observations into global phenology data systems. Applications in Plant Sciences, 7(3): e01231. DOI: https://doi.org/10.1002/aps3.1231
Campaign Monitor. No date. “Ultimate Email Marketing Benchmarks for 2021: By Industry and Day.” https://www.campaignmonitor.com/resources/guides/email-marketing-benchmarks/ Accessed Aug 4, 2021.
Cherubini, M, Villalobos-Zuniga, G, Boldi, M-O and Bonazzi, R. 2020. The unexpected downside of paying or sending messages to people to make them walk: Comparing tangible rewards and motivational messages to improve physical activity. ACM Transactions on Computer-Human Interaction, 27(2): 1–44. DOI: https://doi.org/10.1145/3365665
Cooper, CB, Dickinson, J, Phillips, T and Bonney, R. 2007. Citizen science as a tool for conservation in residential ecosystems. Ecology and Society, 12(2): 11–21. DOI: https://doi.org/10.5751/ES-02197-120211
Crall, A, Kosmala, M, Cheng, R, Brier, J, Cavalier, D, Henderson, S and Richardson, A. 2017. Volunteer recruitment and retention in online citizen science projects using marketing strategies: lessons from Season Spotter. Journal of Science Communication, 16: 1–29. DOI: https://doi.org/10.22323/2.16010201
Crimmins, MA and Crimmins, TM. 2008. Monitoring plant phenology using digital repeat photography. Environmental Management, 41: 949–958. DOI: https://doi.org/10.1007/s00267-008-9086-6
Crimmins, TM, Crimmins, MA, Gerst, KL, Rosemartin, AH and Weltzin, JF. 2017a. USA National Phenology Network’s volunteer-contributed observations yield predictive models of phenological transitions. PLoS ONE, 12(8): e0182919. DOI: https://doi.org/10.1371/journal.pone.0182919
Crimmins, TM, Marsh, RL, Switzer, J, Crimmins, MA, Gerst, KL, Rosemartin, AH and Weltzin, JF. 2017b. USA National Phenology Network gridded products documentation. U.S. Geological Survey Open-File Report 2017–1003. DOI: https://doi.org/10.3133/ofr20171003
Crimmins, TM, Rosemartin, A, Lincicome, A and Weltzin, JF. 2010. USA National Phenology Network 2009 Observer Survey Report. USA-NPN Technical Series 2010–003. https://www.usanpn.org/files/shared/files/2009_Observer_Survey_Report_final.pdf.
Crimmins, TM, Weltzin, JF, Rosemartin, AH, Surina, EM, Marsh, RL and Denny, EG. 2014. Targeted campaign increases activity among participants in Nature’s Notebook, a citizen science project. Natural Sciences Education, 43: 64–72. DOI: https://doi.org/10.4195/nse2013.06.0019
Davis, LF, Ramirez-Andreotta, MD and Buxner, SR. 2020. Engaging diverse citizen scientists for environmental health: recommendations from participants and promotoras. Citizen Science: Theory and Practice, 5(1): 1–27. DOI: https://doi.org/10.5334/cstp.253
De Moor, T, Rijpma, A and Lopez, MP. 2019. Dynamics of engagement in citizen science: results from the “Yes, I do!” project. Citizen Science: Theory and Practice, 4(1): 1–17. DOI: https://doi.org/10.5334/cstp.212
Elmendorf, SC, Crimmins, TM, Gerst, KL and Weltzin, JF. 2019. Time to branch out? Application of hierarchical survival models in plant phenology. Agriculture and Forest Meteorology, 279: 107694. DOI: https://doi.org/10.1016/j.agrformet.2019.107694
Emery, N, Roth, K and Pivovaroff, AL. 2020. Flowering phenology indicates plant flammability in a dominant shrub species. Ecological Indicators, 109: 105745. DOI: https://doi.org/10.1016/j.ecolind.2019.105745
Eveleigh, A, Jennett, C, Blandford, A, Brohan, P and Cox, AL. 2014. Designing for dabblers and deterring dropouts in citizen science. CHI ‘14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Toronto. Canada, 26 May–1 Jun 2014, 2985–2994. DOI: https://doi.org/10.1145/2556288.2557262
Frensley, T, Crall, A, Stern, M, Jordan, R, Gray, S, Prysby, M, Newman, G, Hmelo-Silver, C, Mellor, D and Huang, J. 2017. Bridging the benefits of online and community supported citizen science: a case study on motivation and retention with conservation-oriented volunteers. Citizen Science: Theory and Practice, 2(1): 1–14. DOI: https://doi.org/10.5334/cstp.84
Freyne, J, Yin, J, Brindal, E, Hendrie, GA, Berkovsky, S and Noake, M. 2017. Push Notifications in Diet Apps: Influencing Engagement Times and Tasks. International Journal of Human–Computer Interaction, 33: 833–845. DOI: https://doi.org/10.1080/10447318.2017.1289725
Fuccillo, KK, Crimmins, TM, DeRivera, C and Elder, TS. 2015. Assessing accuracy in citizen science-based plant phenology monitoring. International Journal of Biometeorology, 59: 917–926. DOI: https://doi.org/10.1007/s00484-014-0892-7
Gerst, KL, Crimmins, TM, Posthumus, EE, Rosemartin, AH and Schwartz, MD. 2020. How well do the Spring Indices predict phenological activity across plant species? International Journal of Biometeorology, 64: 889–901. DOI: https://doi.org/10.1007/s00484-020-01879-z
Goldsmith, GR, Crimmins, TM, Haas, D, Roush, M and Verbeke, M. 2019. Demographics, interests, and motivations of participants in a long-term citizen science project. In AGU Fall Meeting Abstracts, 2019: ED33B–1013.
Howard, AF. 2018. Asclepias syriaca (common milkweed) flowering date shift in response to climate change. Scientific Reports, 8: 17802. DOI: https://doi.org/10.1038/s41598-018-36152-2
Lopez, C. 2021. Motives for Citizen Science Program Participation and the Role of the Organization: Lessons from Water Quality Monitors in Texas. Citizen Science: Theory and Practice, 6(1): 3. DOI: https://doi.org/10.5334/cstp.341
Martin, VY and Greig, EI. 2019. Young adults’ motivation to feed wild birds and influences on their potential participation in citizen science: An exploratory study. Biological Conservation, 235: 295–307. DOI: https://doi.org/10.1016/j.biocon.2019.05.009
Mazer, SJ, Gerst, KL, Matthews, ER and Evenden, A. 2015. Species-specific phenological responses to winter temperature and precipitation in a water-limited ecosystem. Ecosphere, 6(6): 1–27. DOI: https://doi.org/10.1890/ES14-00433.1
Nov, O, Arazy, O and Anderson, D. 2014. Scientists@Home: what drives the quantity and quality of online citizen science participation? PLoS ONE, 9(4). DOI: https://doi.org/10.1371/journal.pone.0090375
Pielot, M, Church, K and de Oliveira, R. 2014. An in-situ study of mobile phone notifications. In: Quigley, A, Diamond, S, Irani, P and Subramanian, S (eds.), Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services; 2014 Sep 23–26, 233–242. Tornoto, ON, Canada. New York, NY: ACM. DOI: https://doi.org/10.1145/2628363.2628364
Rosemartin, AH, Crimmins, TM, Enquist, CAF, Gerst, KL, Kellermann, JL, Posthumus, EE, Weltzin, JF, Denny, EG, Guertin, P and Marsh, L. 2014. Organizing phenological data resources to inform natural resource conservation. Biological Conservation, 173: 90–97. DOI: https://doi.org/10.1016/j.biocon.2013.07.003
Schwartz, MD, Ahas, R and Aasa, A. 2006. Onset of spring starting earlier across the Northern Hemisphere. Global Change Biology, 12: 343–351. DOI: https://doi.org/10.1111/j.1365-2486.2005.01097.x
Shelton, JMG, Fisher, MC and Singer, AC. 2020. Campaign-based citizen science for environmental mycology: the science solstice and summer soil-stice projects to assess drug resistance in air- and soil-borne Aspergillus fumigatus. Citizen Science: Theory and Practice, 5(1): 20. DOI: https://doi.org/10.5334/cstp.325
Tang, J and Prestopnik, NR. 2019. Exploring the impact of game framing and task framing on user participation in citizen science projects. Aslib Journal of Information Management, 71(2): 260–280. DOI: https://doi.org/10.1108/AJIM-09-2018-0214
USA National Phenology Network. 2021. Plant Phenology Data. Data type: Status and Intensity, 2018–2029 for the continental United States. USA-NPN, Tucson, Arizona, USA. Data set accessed Jan 19, 2021. DOI: https://doi.org/10.5066/F78S4N1V
Wallace, CSA, Walker, JJ, Skirvin, SM, Patrick-Birdwell, C, Weltzin, JF and Raichle, H. 2016. Mapping presence and predicting phenological status of invasive buffelgrass in southern Arizona using MODIS, climate and citizen science observation data. Remote Sensing, 8(7): 524. DOI: https://doi.org/10.3390/rs8070524
West, S and Pateman, R. 2016. Recruiting and retaining participants in citizen science: what can be learned from the volunteering literature? Citizen Science: Theory and Practice, 1(2): 1–10. DOI: https://doi.org/10.5334/cstp.8
Woolford, SJ, Barr, KLC, Derry, HA, Jepson, CM, Clark, SJ, Clark, VJ and Stretcher, KR. 2012. OMG do not say LOL: Obese adolescents’ perspectives on the consent of text message to enhance weight loss efforts. Obesity, 19(12): 2382–2387. DOI: https://doi.org/10.1038/oby.2011.266