Lilac leaf-out and bloom forecasts

The USA-NPN generates forecasts of leaf-out and flowering in lilacs up to six days in the future as a part of the Spring Leaf and Bloom Index models (Schwartz 1997; Schwartz et al. 2006; Crimmins et al. 2017b). These models use daily temperature and weather events as inputs to predict when individual lilacs will first undergo leaf-out and bloom at a location (Schwartz et al. 2006; Schwartz et al. 2013; Ault et al. 2015). The Spring Leaf Index is the average of three individual models, one of which predicts leaf-out in lilacs. Similarly, the Spring Bloom Index is the average of three individual models, one of which predicts bloom in lilacs. Each night, short-term gridded minimum and maximum temperature forecast maps are downloaded and used to update short-term forecasts of spring activity—as reflected in the Spring Leaf and Bloom Index models—and lilac leaf-out and flowering.

Tracking phenology in Nature’s Notebook

Participants in Nature’s Notebook collect repeated observations of what they see on individual plants over the course of the season. The steps to tracking plant phenology using Nature’s Notebook are: create a user account, register a site, and then register one or more individual plants from the list of species available for monitoring (Rosemartin et al. 2014). Once these steps have been achieved, participants can submit phenology observations.

The observation protocols in Nature’s Notebook are status protocols, in that each time a participant records an observation, they indicate whether they do or do not see a phenophase, such as leaf-out or flowering, being expressed (Denny et al. 2014). Observers are encouraged to report frequently when things are changing quickly, to capture the date of transition—when a plant shifts from the phenophase not being expressed to when it is being expressed—with as much precision as possible. We recommend at least weekly observations, and encourage observers to report two to three times a week when things are changing quickly. Participants may join the program at any time and may track the phenology on as many plants as they wish, though the majority of participants tracking lilacs record observations on a single lilac plant.

Cuing participants to observe

Hundreds of Nature’s Notebook participants track leaf-out and flowering in lilacs across the United States. To prompt Nature’s Notebook observers to capture the transitions from leaf buds not broken to leaf buds broken and flower buds not broken to flower buds broken with as much accuracy and precision as possible, we email observers who have registered a lilac in their Nature’s Notebook accounts three days prior to when leaf-out is expected to occur based on the lilac Spring Leaf Index model. We also message these observers three days prior to when flowering is expected to occur based on the lilac Spring Bloom Index model. In each message, we emphasize the importance of documenting “no” observations (lilacs not yet leafed out, not flowering) as well as capturing the transition from the phenophase not yet present (“no”) to the phenophase present (“yes”) in as few days as possible. To prepare observers to receive these messages, we send a message at the beginning of the season explaining this campaign and the importance of documenting the timing of transition accurately and precisely.

Nature’s Notebook observers that register a lilac are automatically added to the list of message recipients, in accordance with terms to which they agree when they join the program. However, participants can easily and permanently opt out of the messages using the unsubscribe link that appears at the bottom of every message.

Forecasts of lilac leaf-out and bloom are updated nightly based on daily minimum and maximum temperature data (Crimmins et al. 2017b). Locations of registered lilac plants under observation in Nature’s Notebook are intersected with these forecast maps through a nightly process. When a map pixel containing one or more lilacs is predicted to reach the conditions associated with either leaf-out or flowering within the next three days, an email message is sent to the email address associated with that lilac plant. Messages are sent using Constant Contact email marketing software. This workflow requires a script on USA-NPN servers to connect with the Constant Contact application programming interface (API) to send messages to the proper individuals on a daily basis. Leaf messages always precede bloom messages, as the environmental conditions associated with leaf-out are always met prior to conditions associated with bloom (Schwartz 1997). Leaf-out messages are typically sent between mid-January and the end of May, and bloom messages are typically sent between the end of January and the end of June.

Statistical analyses

In this study, we focus on participants that submitted phenology observations in the years of the study (2018, 2019, and 2020) (USA National Phenology Network 2021). Like many citizen science programs, we struggle to sustain active participants in Nature’s Notebook, and participants leave the program for a wide range of reasons. Here, we focus on how participants that were active in the program were affected by messages encouraging them to observe.

Our assumption in this evaluation is that large differences between predicted and observed dates for the phenomena reflect inconsistent reporting on the behalf of observers rather than poor forecast performance. The Spring Leaf and Bloom Index models perform well at predicting leaf-out and flowering in lilacs. A recent evaluation comparing observations of leaf-out and flowering in cloned lilacs (1981–2017) to Spring Index predictions of leaf-out and flowering in lilacs reported a root mean squared error (RMSE) of 11.97 days for leaf-out and 6.46 days for bloom (Gerst et al. 2020). Further, 47% of reports of leaf-out in cloned and common lilacs submitted to Nature’s Notebook (1981–2021) were within one week of the predicted date of leaf-out made with the lilac model; for bloom, this figure climbed to 60% (USA National Phenology Network, unpublished data). Bias in both models is less than two days. Accordingly, we assess accuracy in observers’ reports by comparing their reports of when leaf-out and flowering occurred to when the events were predicted to occur within three days. We evaluate precision in the report of onset by evaluating the number of days between when an observer previously reported the phenomenon (leaf-out or flowering) as not occurring and when they first reported it as occurring.

We constrained our analysis to focus only on instances when the model predicted leaf-out or bloom prior to when the observer reported the event. If observers logged observations of leaf-out or bloom prior to receiving the messages, the evaluation does not answer the question of whether messages prompting participants to observe their plants have the intended effect of improving accuracy and precision in reports of leaf-out or flowering.

To test for differences in accuracy, we calculated the number of days between the predicted and observed day of leaf-out or flowering for each report submitted in each year. Because we constrained our analyses to instances where the model predicted leaf-out or bloom prior to the observer’s report, the calculated values were either zero or positive, with the greatest number of values at or close to zero, and a decreasing number of values farther from zero. We used the outer fence method (3 × the interquartile range) to identify and remove outliers. As this resulted in a highly skewed dataset, we compared the values among the three groups (received and opened the message; received but did not open the message; did not receive the message) using a Kruskall-Wallis rank sum test. We then used the Wilcoxon rank sum test for multiple pairwise comparisons to determine statistically significant pairs of means.

To test for differences in precision, we calculated the number of days since the last reported “no” for the “breaking leaf buds” or “open flowers” phenophases and the first reported “yes” for each report. As above, we used the outer fence method to identify and remove outliers. Next, for both the leaf and bloom datasets, we compared the values among the three groups (received and opened the message; received but did not open the message; did not receive the message) using a Kruskall-Wallis rank sum test. We then used the Wilcoxon rank sum test for multiple pairwise comparisons to determine statistically significant pairs of means.

Do email notifications result in more accurate reports of leaf-out and flowering?

For the leaf messages, there was a highly significant difference in the duration between when leaf-out was predicted and reported among the three groups (X² = 14.56, p < 0.001, df = 2, Kruskall-Wallis rank sum test). The duration in days between when leaf-out was predicted and reported was nearly five days shorter in the “opened” group (mean = 8.1 days ± SD = 8.5 days) than the “did not open” group (13.4 ± 13.6 days, p = 0.019) and over a week shorter than the “did not receive group (16.0 ± 16.0 days, p < 0.001, Wilcoxon multiple comparisons test, Figure 1a).

There was a marginally significant difference in the duration between when bloom was predicted and reported among the three groups (X² = 05.33, p = 0.07, df = 2, Kruskal-Wallis rank sum text). The number of days between when leaf-out was predicted and reported was slightly more than one day longer in the “opened” group (5.5 ± 4.6 days) than the “did not open” group (4.2 ± 3.1 days, p = 0.085, and the “did not receive” group (4.9 ± 4.6 days, p = 0.085, Wilcoxon multiple comparisons test, Figure 1b).

Do email notifications result in more precise reports of leaf-out and flowering?

The number of days between when the observer reported a “no” and when they first reported “yes” to leaf-out was significantly different among the three groups (X² = 15.62, p < 0.001, df = 2, Kruskal-Wallis rank sum test). The duration between when the observer reported a “no” and when they first reported “yes” to leaf-out was nearly two days shorter in the “opened” group (5.2 ± 5.5 days) than in the “did not open” group (7.0 ± 5.4 days, p < 0.001, Tukey multiple comparisons test) and about one and a half days shorter than the “did not receive” group (6.5 ± 5.9 days, p = 0.01, Wilcoxon multiple comparisons test, Figure 2a).

The number of days between when the observer reported a “no” and when they first reported “yes” to open flowers was marginally significantly different among the three groups (X² = 5.01, p = 0.08, df = 2, Kruskall-Wallis rank sum test), though the duration between when the observer reported a “no” and when they first reported “yes” to open flowers was not significantly different among the three groups (Figure 2b).

Are email prompts worth the cost and effort?

Overall, the messages had a clear and positive impact on the accuracy and precision of reports of leaf-out and flowering in lilacs. Implementing this approach entailed the up-front investments of staff time to craft the messages and to create and maintain the scripts that populated and sent the messages each day as well as the cost of the email marketing software subscription. We estimate the time spent on these activities to be approximately 10 hours to create the script that interfaces between the daily lilac leaf- out and bloom maps and the registered lilac observer locations, and approximately 20 hours each year to maintain the script, test the Constant Contact API, and draft and send the email messages.

Open rates for our directed email messages were much higher than the standards for the most aligned industries: our rates ranged from 47% to 100%, compared with industry standards around 25%. This may stem from the fact that participants in Nature’s Notebook engage in the program primarily to contribute to science and to learn; such motivations may naturally lead to a greater likelihood of opening messages. This result suggests that the messages are appreciated by Nature’s Notebook participants and are worth the costs involved to prepare and send. The high open rates indicate that we are sharing content with the recipients that they wish to receive. Further, because participants can opt out of the messages at any time, we can rest easy that we are not bombarding our participants with unwanted emails.

Because this approach results in a positive impact on data quality at a relatively low cost, we plan to continue to send messages when we can predict the timing of an event using a model like the Spring Indices. However, as this study reveals, this approach is not failsafe. In some seasons of our study, the scripts failed multiple times, resulting in dozens of participants not receiving messages. One approach to mitigate this issue would be to implement a nightly script that sent a notification to our team if no emails were sent on a particular day; this would provide a prompt to ensure scripts were still functioning properly. Other citizen science programs planning to rely on automated scripts should consider similar failsafe measures.

Alternatives to email-based messages

Short message service (SMS) text messages are one alternative approach to the email-based messages we are currently sending. Recent surveys suggest growing preference for text messages over email, especially for messages coming from businesses or other institutions (PC Magazine 2020). This pattern appears to hold for programs individuals voluntarily join as well. For example, participants in a smoking cessation program expressed more positive feelings toward text messages than email messages (Abroms et al. 2012). Text-based notifications offer direct contact through devices that are nearly always on and in one’s possession. However, with the proliferation of mobile apps, users may receive over 60 notifications each day (Pielot et al. 2014), which can lead to message fatigue and even to users unsubscribing.

Another option for engaging program participants, particularly for those programs with mobile apps, are app-based notifications, or push notifications that can be sent by mobile app, even when the app is not open. Nature’s Notebook participants have the option to either log observations on paper datasheets and transcribe these into an online interface or to use the Nature’s Notebook mobile application. Use of the Nature’s Notebook mobile app in place of the web-based interface to log phenology observations has grown rapidly in recent years, from just over 20% of observations submitted via the app in 2017 to nearly 60% in 2021. Accordingly, sending app-based notifications to Nature’s Notebook participants to go out and log observations at a particular time as a result of a forecast, is one alternative we may consider for engaging observers. In a study of diet apps, participants were tolerant of receiving multiple push notifications a day, though engagement with the messages declined over time (Freyne et al. 2017), suggesting push notifications may offer an effective solution for short-term campaigns.

Applicability to other citizen science programs

The question of whether email-based messages have a positive impact on participants’ actions—and therefore, data quality—has applicability to many citizen science programs, especially in cases where documenting the precise time an event occurred is important. Findings from the healthcare field demonstrate that program participants’ reactions and feelings regarding email or text-based messages prompting a particular action are variable. For example, Woolford et al. (2012) found that text messages sent to adolescents in a weight-loss program were very appreciated, and the recipients were “very enthusiastic” about such forms of communication (p. 2382). In contrast, Cherubini et al. (2020) found that app-based messages intended to motivate people to take a walk did not result in increased physical activity and annoyed participants. However, within the citizen science realm, reminder messages are generally welcomed: Project participants have indicated that such reminders are helpful (Tang and Prestopnik 2019; Martin and Greig 2019; Shelton et al. 2020), and participants in one project even requested regular reminders to observe (Martin and Greig 2019).

A key takeaway from this analysis is that the first messages participants received during the active data collection season had the largest impact on their activity. This finding is consistent with other research that has reported declining engagement over time with subsequent messages (Freyne et al. 2017). We do not know whether participants that opened the messages read through to the end; the truth may be that that simple, brief notifications that could be achieved via SMS text messages or app-based notifications can have the intended effect. However, the significant differences in both accuracy and precision between participants that opened the messages and those that received but did not open the messages suggests that the content of the messages had some influence on observer behavior. It may still be possible to achieve the desired effect in these situations through brief text messages or app-based notifications, though the content of these messages should be carefully considered to contain the critical information needed by participants. Establishing the optimal format and content of messages encouraging citizen science project participants to take a particular action is an area ripe for further work.

Study limitations

In this analysis, we assume that the models used to predict the timing of leaf-out and flowering in lilacs reflect real-world conditions well enough that differences between predicted and reported values can be interpreted as observer error. In reality, model performance is unknown. Previous studies (e.g., Gerst et al. 2020) have endeavored to evaluate model performance, though this has been achieved by comparing the predictions to reports of leaf-out and flowering contributed by volunteer observers. As such, neither dataset can be considered correct.

This limitation has the potential to affect our accuracy results. If the model predictions are not correct, then evaluating whether the number of days between the predicted and reported values for leaf-out or flowering, as we do in the present study, is fraught. The difference in accuracy values among the three groups in this study—that is, the differences in the number of days between the predicted and reported dates of an event—are equal to or greater than the model RMSE, suggesting that despite the potential error present in the predicted values, we still see an impact of messaging on when observers report the event to Nature’s Notebook. Both the model performance and the impact of messaging on observer behavior could be assessed through alternative approaches with more rigor, such as by documenting leaf-out and flowering in individual lilac plants using automated repeat photographs (e.g., Crimmins and Crimmins 2008).

In this study, we take advantage of a hiccup in our scripts to ask and answer questions pertaining to the impact of messaging program participants. We did not impose a formal sampling design or randomize the treatments of who received a message and who did not. Accordingly, our findings are correlational, and as such, we cannot establish that the messages (or lack thereof) caused participants to observe more frequently or with greater accuracy. The patterns we report here may be the result of social dynamics or other confounding variables not tracked in this study. A formally designed evaluation with a randomized study design would be better suited to establish these relationships with more rigor.