Visual foliage survey

Visual foliage surveys conducted at ground level have been used for decades to characterize foliage arthropod availability to birds throughout the forest canopy (Holmes and Schultz 1988). For one survey, an observer examines both the upper- and undersides of 50 leaves and associated petioles and twigs on a branch of woody vegetation typically 1–2 m above the ground. All arthropods observed greater than 2 mm in length are identified, generally to order (but in some cases suborder or family; Table 1), and their body length (not including legs or antennae) is recorded to the nearest millimeter. Arthropods smaller than 2 mm are ignored both because of their lesser importance as food items as well as the increased difficulty and therefore time required for identification. A single visual foliage survey takes 2–6 minutes depending upon the density of arthropods, experience of the observer, and degree of clustering of leaves on a branch.

Beat sheet survey

As an alternative to the visual foliage survey, participants may choose instead to conduct a beat sheet survey in which the survey branch is beaten with a stick ten times in rapid succession over a white 60 × 60 cm sheet. As with the visual survey, all arthropods are identified to the relevant order/group (Table 1) and length is recorded to the nearest millimeter. In addition, the participant records the total number of leaves that were positioned above the beat sheet during beating which is expected to vary from branch to branch. A single beat sheet survey typically takes 2–3 minutes depending on the density of arthropods and the experience of the observer.

Data collection

Foliage arthropod surveys were conducted at Prairie Ridge Ecostation (35.8117° N, 78.7139° W), an outdoor nature center in Raleigh, NC. This site features a narrow strip of deciduous forest, with a diverse tree community including Liquidambar styraciflua, Acer negundo, and Diospyros virginiana, alongside an open prairie. Sixty survey locations were established in 12 groups (“circles”) of five surveys each (Figure 1).

In both 2015 and 2016, members of the Hurlbert Lab at the University of North Carolina (hereafter “trained scientists”) conducted visual and beat sheet surveys twice per week from mid-May through July at all survey locations (Table 2). With the exception of Hurlbert, trained scientists were undergraduate students with no significant entomological expertise prior to participating. Hurlbert conducted surveys himself but also provided extensive training before and during all foliage survey activities throughout the season, ensuring that team members properly followed survey protocols and were capable of documenting potentially cryptic arthropods and of identifying arthropods to the relevant groups. Visual surveys were conducted first at each survey location followed by a beat sheet survey on an adjacent branch of the same plant species. Surveys were typically conducted between 0830 and 1200 hrs. In 2015, trained scientists additionally conducted beat sheet surveys once per week on Thursday afternoons, typically between 1300 and 1400, at a fixed subset of 40 of the 60 total survey locations. This was done to check for an effect of time of day on the observed phenological patterns, but no strong effect on phenological timing was found (Figure 2).

Also in both 2015 and 2016, volunteers (hereafter “citizen scientists”) were recruited to conduct foliage arthropod surveys at the fixed subset of 40 survey locations (Table 2). Citizen scientists were recruited through the volunteer program at the North Carolina Museum of Natural Sciences and included both men and women varying in age from 22 to 50 years in age. Volunteers were trained by CLG, who worked with the volunteers the first three times they conducted surveys and focused heavily on arthropod identification skill building. After the third survey, the volunteers conducted the surveys on their own. Thus, while citizen scientists and trained scientists received similar training initially, they differed in the duration of supervision in the field. In 2015, seven different citizen scientists conducted visual foliage surveys, some on Thursdays between 1300 and 1500 and others on Saturdays between 0900 and 1100 hours most weeks. In 2016, four citizen scientists were recruited, and they conducted beat sheet surveys once per week on average, typically between 0800 and 1200 hrs. We were thus able to compare citizen scientist and trained scientist observations based on visual surveys in 2015, and based on beat sheet surveys in 2016. Citizen scientists conducted 166–185 surveys per participant on average over the course of each season, while trained scientists conducted 1111–1662 surveys per participant on average and so had more experience on top of the increased training and supervision (Table 2).

Finally, while the Caterpillars Count! survey methodology focused on foliage 1–2 m above ground for logistical reasons, we would ideally like to make inferences about arthropod phenology throughout the entire canopy. To validate this comparison between foliage strata, we collected caterpillar frass falling from the canopy in 2015 to compare with observed phenology from the ground level foliage surveys. Although citizen scientists did not collect frass data, it is crucial to understand whether this citizen science protocol actually captures the phenology reflected by the frass. Frass traps consisted of a 20 cm diameter plastic funnel mounted onto a garden stake 30 cm above ground level and lined with a 40 cm diameter piece of filter paper folded into a cone. Each frass trap samples a cross-sectional area of 434 cm². Frass traps were located within existing survey circles (1 trap per circle) such that they spanned the same locations as the arthropod surveys. Although frass traps were collected and reset every 3–4 days, data were unusable on dates where there had been major rainstorms since the traps were deployed.

All data were recorded on paper data sheets (although a free mobile app is now available and recommended), and later entered into the project database via a web form. The form limited the entry of extreme values or outliers and also ensured standardization of arthropod names. At the end of each survey day, automated algorithms tallied the number of surveys entered and flagged instances of potentially missing data sheets that had not yet been entered.

Data analysis

Although all arthropods at least 2 mm in length were recorded by our protocols, we only used observations of arthropods 5 mm long or longer in analyses. This reduces the incidence of misidentification of very small individuals, and also minimizes the effect of error in estimating the 2 mm cutoff. Comparisons of relative arthropod composition between survey methods and between survey participant groups was conducted using chi-squared analyses, while comparisons of absolute density (number observed per survey) across all arthropod groups were conducted using Pearson’s correlation coefficients.

Phenology was characterized by the fraction of surveys (occurrence) on which a focal arthropod group was detected on a given date. We used occurrence rather than mean density estimates because the latter are sensitive to outliers, and we had a few instances in which a large number of gregarious caterpillars were observed in a single survey. Because citizen scientists typically collected data only once per week, we averaged the bi-weekly samples of trained scientists into weekly estimates in order to visually compare phenology and calculate Pearson’s correlation coefficients across weeks.

To assess the impact of sampling intensity and sampling frequency on estimates of peak caterpillar phenology date, we used data from 2015 where trained scientists conducted 60 beat sheet surveys twice per week from mid-May through mid-July. We fit a Gaussian curve to these data (excluding the last two dates in July which reflect a late season peak less relevant for the avian breeding season; see Figure 5a below) and assumed the estimated mean of this curve reflected the “true” peak date (julian day 172). We then randomly subsampled the full dataset by manipulating both the number of surveys examined per sampling date (10, 20, 30, 40, 50, or 60 out of the 60 surveys) and the sampling frequency (every sampling date used, every other, every third, every fourth, and every fifth). For each combination of survey number and sampling frequency we conducted 60 replicate subsamples evenly split across potential starting dates (i.e., if sampling frequency was set at every other sampling date, we subsampled using the 1st, 3rd, 5th, etc. dates, but as another replicate also the 2nd, 4th, 6th, etc.). We estimated the peak date from Gaussian fits to the subsampled data. Fits were used only if the mean date was between julian days 100 and 200, and if the R² for the fit was >0.2 (89% of all fits).

Beat sheet versus visual surveys

Relative and absolute density estimates for each arthropod group depended upon survey method (Figure 3; 𝜒² = 284.73, df = 6, p < 10^–16). Beat sheet surveys by trained scientists revealed a greater proportion of Coleoptera (beetles) and a lower proportion of Diptera (flies) compared to visual surveys by trained scientists at the same sites and years. A comparison of absolute densities reveals the same discrepancy with respect to the rates at which beetles and flies are observed using the two methods, but also illustrates that density estimates are comparable for most other arthropod groups (r = 0.82, p = 0.0004). Notably, caterpillar density estimates were similar using both methods (0.077 versus 0.083 caterpillars/survey for beat sheet and visual surveys, respectively).

Citizen scientist versus trained scientist observations: Density

Perceived arthropod composition differed between citizen scientist- and trained scientist- conducted visual surveys (Figure 4a, 𝜒² = 44.94, df = 6, p < 5e10^–8). Citizen scientists reported a greater proportion of flies and beetles and a smaller proportion of Auchenorrhyncha (leafhoppers, planthoppers, etc.), Orthoptera (grasshoppers and crickets), and caterpillars compared to the trained scientists, however all differences were within +/– 6%. Absolute density estimates across arthropod taxa were positively correlated between the two groups (Figure 4c triangles, r = 0.84, p < 0.0002), and although citizen scientists overestimated fly and beetle density and underestimated caterpillar density relative to trained scientists, these differences were all within 0.05 arthropods/survey.

Using beat sheet surveys, the difference between citizen scientists and trained scientists was less pronounced (Figure 4b, 𝜒² = 18.34, df = 6, p = 0.005), with citizen scientists reporting a slightly greater proportion of Diptera and Araneae and a slightly lower proportion of Auchenorrhyncha and Coleoptera relative to trained scientists. Again, all differences were within +/– 6%. Absolute density estimates were even more strongly correlated across arthropod taxa between the two groups than in the visual survey comparison (Figure 4c, rectangles, r = 0.94, p < 0.0001). There was much better congruence in estimates of caterpillar and Orthopteran density in particular using beat sheet surveys compared to visual surveys.

Citizen scientist versus trained scientist observations: Phenology

A primary goal of the Caterpillars Count! project is to characterize the seasonal fluctuations in arthropods over the spring and summer. The phenology of caterpillars as captured by visual and beat sheet surveys near ground level mirrored the phenology of frass falling from the canopy (Figure 5a, dotted line).

As expected, arthropods like caterpillars and orthopterans that depend on leaves for food and shelter exhibited low densities in early spring and then increased over the summer (Figure 5a–d). Orthopterans continued to increase through mid- to late-July, while caterpillars exhibited a peak in occurrence in mid-June, followed by another in early July. Foliage arthropods in aggregate (caterpillars, orthopterans, beetles, spiders, leafhoppers, and true bugs) exhibit a general positive trend over the dates examined, with less pronounced seasonal peaks due to the more consistent occurrence of some of those other groups like spiders.

In 2015 using visual surveys, citizen scientists underestimated the occurrence of foliage arthropods early in the season relative to trained scientists, but estimates converged later in the season (Figure 5a, c, e). Citizen scientists did not observe many caterpillars at all until July. As such, they missed the peak in caterpillar occurrence documented by trained scientists in mid-June, although their observations of a decline in mid-July and subsequent recovery in late July were generally consistent (Figure 5a). Similarly, citizen scientists in 2015 also missed the late June peak in orthopterans but captured the peak in July (Figure 5c).

In 2016, using beat sheet surveys, the phenology recorded by citizen scientists was much more strongly correlated with trained scientist observations (0.50 < r < 0.95; Figure 5b, d, f). In particular, citizen scientists identified the same increase and mid-June peak in caterpillar occurrence as trained scientists. Citizen scientists did not actually conduct surveys the week of Julian day 186 when trained scientists identified a second seasonal peak in caterpillars.

Sampling frequency and intensity

Estimates of peak caterpillar date were unbiased with respect to the “true” value (Julian day 172) even at low sampling intensity or frequency (Figure 6). However, as expected, 95% confidence intervals around the estimated value were tightest when conducting many surveys at high frequency, or with a low sampling interval. Estimates of peak caterpillar date based on only a small number of surveys or a low frequency of sampling resulted in estimates that were often weeks from the “true” value. In this particular dataset, sampling 30 surveys on a weekly basis led to 95% of estimated peak dates falling within one week of the true date. Increasing the number of surveys conducted per sampling date typically yielded a greater increase in accuracy of the peak date estimate compared to increasing the sampling frequency (Figure 6). For example, doubling the number of weekly surveys from 20 to 40 reduced the confidence interval width by more than 50% (19 days to 9), compared to conducting 20 surveys at double the frequency (19 days to 12).

Implications for survey methods

We evaluated two methods for conducting foliage arthropod surveys, visual surveys and beat sheet surveys. In general, the two survey methods yielded very similar results with respect to relative and absolute estimates of arthropod group density based on data collected by trained scientists. As expected, however, each method had its own biases. Flies (Diptera) were underrepresented on beat sheet surveys compared to visual surveys as they tended to fly immediately up and away as soon as a branch was first struck. In contrast, beetles (Coleoptera) were more numerous in beat sheet surveys than in visual surveys. Many of the beetles observed in beat sheets were narrow brownish click beetles (family Elateridae) which rest flat along twigs. This comparison suggests that observers may frequently be overlooking these beetles in visual surveys, although they are quite obvious when lying in a beat sheet. Density estimates for most other groups, including caterpillars, were similar using the two methods. This is interesting given anecdotal observations that some caterpillars, especially those in leaf rolls or sewn between two leaves, are not dislodged by beating, while caterpillars that are extremely cryptic in appearance are more likely to be missed in visual surveys. Although these two groups seemed to be of equivalent abundance such that our two density estimates were comparable, this may not always be the case. Researchers using these data specifically for density estimates will certainly want to take survey method and associated biases into account during analysis. However, phenological metrics of timing which rely on relative, not absolute, indices of abundance should be unbiased.

We found a striking concordance between our ground level survey-based estimates of caterpillar phenology and the canopy level frass-based phenology, suggesting that foliage arthropod surveys conducted near ground level can be used to assess the phenology of higher vegetation strata as well. This correspondence in phenology is consistent with other studies that have found a correlation between lower and upper canopy caterpillar density across trees, years, and season (Cooper 1988; Holmes and Schultz 1988). Monitoring frass phenology at sites where the Caterpillars Count! project is implemented will continue to improve our understanding of where and when phenology varies across forest strata, and we are currently exploring how frass monitoring may form the basis for a complementary citizen science project.

Agreement between citizen scientists and trained scientists

For both survey methods, citizen scientists generated relative density estimates that were, on average, within 6% of those generated by trained scientists. Beat sheet surveys yielded stronger agreement between citizen scientists and trained scientists with respect to density estimates and phenology compared to visual surveys. This was especially true for caterpillars: In 2015 citizen scientists entirely missed the mid-June peak in caterpillar occurrence when conducting visual surveys, while the citizen scientists in 2016 documented patterns similar to the trained scientists using beat sheet surveys. The individual citizen scientist participants differed between 2015 and 2016 and numbered only 10 unique individuals in total, indicating that this effect is just as likely to be a participant effect as a survey method effect. Anecdotally, one participant in 2015 was notably less engaged and motivated compared to participants in 2016, highlighting the need to further validate the use of visual surveys in this project. Certainly, not all participants would necessarily have missed the caterpillar peak in 2015. Nevertheless, the ideal arthropod survey methodology is one that is robust to variation in participant ability and motivation. The task of detecting arthropods against a white beat sheet is presumably less subject to error than that of detecting arthropods on an often similarly colored branch. Given a choice, we therefore encourage citizen science participants to use beat sheets if possible.

Another advantage of beat sheet surveys in the context of citizen science is the ability to engage and involve younger participants. Although children are not the target participant group for this project, beat sheet surveys require considerably less time and patience than visual surveys, and may be better for youth education programs. Beat sheets are also useful for displaying interesting arthropods to a group, providing an unobstructed view and avoiding the need to have observers step up to a branch one at a time. Although constructing a homemade beat sheet is fairly simple and cheap (approximately $5 in fabric and hardware), it still represents a potential barrier for participants or environmental education centers with limited resources. For that reason alone, we expect that some local project coordinators will choose to conduct visual surveys. Our comparison of the two methods provides an initial suggestion of how to compare data obtained in each, but conducting this methods comparison in other habitats and regions would be useful.

Implications for sampling frequency and intensity

Finally, we examined how variation in sampling intensity and frequency influenced the perceived date of peak caterpillar occurrence. This is an important question because citizen scientist participants have finite time and resources to dedicate to any particular project. While estimates of phenology become more precise with increased data collection, the number of participants willing to meet those increased data collection requirements will be smaller (Sauermann and Franzoni 2015). We found that conducting 30 foliage surveys on a weekly basis provided estimates of peak caterpillar occurrence typically within 1 week of the “true” peak, and we recommend this level of effort as a best practice. If a greater sampling effort is possible, increasing the number of surveys conducted per sampling date yields a greater increase in precision of phenological estimates compared to investing an equivalent amount of effort in increased sampling frequency and so should be preferred. Put another way, conducting 40 surveys once a week is better than visiting the same 20 surveys twice each week. A smaller number of surveys may still be useful in assessing phenology in a qualitative sense (e.g., determining whether it’s an “early” or “late” year). We will more rigorously evaluate this possibility as we accumulate more years of survey data.

Because a single foliage survey by an untrained individual conservatively takes about 6 minutes (including sharing observations with others, walking between surveys, etc.), our recommended effort (30 surveys) requires 3 person-hours per week. While some dedicated and interested individuals may participate at this level, they will be in the minority. For this reason, Caterpillars Count! will be most easily carried out at centralized locations such as environmental education centers that frequently host thousands of visitors each season and have groups of dedicated, regular volunteers eager to contribute toward projects at the site. At centers like these, the data collection effort can be divided among several people such that, for example, a group of 5 could conduct 30 surveys in less than forty minutes. In this way, individuals interested in participating for only a single day may still contribute to the project within a discrete amount of time and with the assistance of trained and experienced participants. This distributed effort strategy still requires one individual at the site who can coordinate the efforts of other participants, and our experience at Prairie Ridge Ecostation suggests that this will require 2 hours per week once the project is up and running.

Sources of error and bias

Data collection for this project involves three potential sources of error in the context of phenology estimation. First, participants must detect arthropods on survey branches or beat sheets. As discussed above, detectability is expected to be a greater problem for visual surveys due to crypsis, although detectability on beat sheets may still be an issue for arthropods that fly, jump, or run out of the sheet before they can be observed. Nevertheless, for detectability to bias phenological signal, it must vary systematically over time. This may be less of an issue for beat sheet surveys, however, the ability to detect insects on branches via visual surveys almost certainly increases with experience. For sites at which the same individual or individuals conduct visual surveys each week, one might expect observations in the first few survey periods to underestimate arthropod occurrence relative to later in the season. Quantifying exactly how arthropod searching ability improves over time will help to determine whether this bias is mostly eliminated after a single day of conducting 5–10 surveys, or if it is likely to persist over a longer period. Nevertheless, to the extent that seasonal arthropods decline in late summer (e.g., July for caterpillars at our study sites), this phenomenon should be well captured by observers regardless of any increases in searching competence.

Second, participants must properly identify arthropods to the appropriate group (Table 1). For groups like caterpillars and spiders, this task will be straightforward. Distinguishing beetles from true bugs and leafhoppers may be more prone to error. We have developed outreach materials including identification keys and cheatsheets to assist participants while they are in the field). We have also developed an arthropod photo identification quiz, which is on our website (https://caterpillarscount.unc.edu/arthropodQuiz/). The quiz may be taken repeatedly with different photos of common foliage arthropods each time. Finally, although not used for this study, Caterpillars Count! now features a free mobile app for data submission. With the app, users may optionally photograph the arthropods they encounter, and these photographs will be automatically submitted to the crowdsourcing identification website iNaturalist.org. This feature allows those who are interested to pursue lower taxonomic level identification by experts.

Third, participants must estimate the body length of arthropods to the nearest millimeter. Although much of the US public is less familiar with metric units, having participants calibrate familiar objects like the width of a fingernail or a pencil is fairly straightforward, and simple rulers can be drawn on the supports of a beat sheet or included in the mobile app and arthropod identification guides. Regardless, errors in length estimation will not impact phenology patterns based on occurrence or density. Even in the event that arthropod lengths are used to calculate biomass phenology via length-weight regressions, length estimates are not expected to be biased seasonally in one direction or the other.

Incentives for participation

Robust survey protocols are necessary but insufficient for ensuring a citizen science project’s success. Equally important are considerations about the motivations and incentives for participating (Hobbs and White 2012), both from the perspective of potential one-time contributors like weekend visitors to an environmental education center, as well as new potential site coordinators and their regular volunteers. Caterpillars Count! provides a context for interested individuals to learn about the natural world around them and to contribute to a broader scientific understanding of arthropod phenology and its consequences in a changing world. The project also provides participants who have affinities to particular Caterpillars Count! sites the ability to contribute to something meaningful at that site which advances broader citizen science outcomes (Jordan et al. 2011; McKinley et al. 2017). We hope the availability of arthropod identification resources, mobile apps for easy data collection, data visualization tools on the project website, and structured learning activities associated with the project will provide additional incentives for environmental educators and others to initiate a Caterpillars Count! monitoring scheme that integrates citizen science volunteers. It is our hope that Caterpillars Count! will yield robust data on arthropod phenology over broad spatial scales that can ultimately be leveraged with other existing datasets to provide new insights into potential mismatches between vegetation, arthropods, and birds.