Oak selection

We designed a simple protocol that was applied by both schoolchildren and trained and untrained professional scientists. The protocol was written by scientists in collaboration with science instructors and communication officers. It was available in French, English, German, Spanish, and Portuguese (Castagneyrol et al. 2019).

In early 2018, 58 teachers with their students and 27 scientists from 16 European countries participated in the project. Each school and scientist selected a minimum of 1 and maximum of 18 mature pedunculate oak trees with lower branches accessible from the ground (schoolchildren: 1 to 8 oak trees, median = 2; scientists: 1 to 18 oak trees, median = 6). We imposed no restrictions on oak tree location, age, or size, but professional scientists were asked to choose oaks in woods larger than 1 ha. All partners measured oak tree circumference at 1.30 m from the ground and recorded oak coordinates with the GPS function of their smartphones.

All partners installed dummy caterpillars on lower branches of their selected oak trees to estimate attack rate, and haphazardly collected fresh leaves from the same trees to estimate insect herbivory. Although most of the schools estimated attack rates, none assessed herbivory. We also set up a complementary experiment to evaluate precision and accuracy of estimating insect herbivory by schoolchildren and professional scientists (see section entitled Insect herbivory below).

Attack rate

To control for latitudinal variation in environmental conditions, we matched the start of the experiment to the local phenology of the oak trees. Six weeks after oak budburst, partners installed 20 dummy caterpillars per tree, i.e., five caterpillars on each of four branches (facing north, south, east, and west) with a minimum distance of 15 cm between caterpillars. Caterpillars were made of the same green plasticine (Staedler, Noris Club 8421, green[5]) provided to all partners by the project coordinators (B. Castagneyrol, EVC). To standardize caterpillar size among partners, caterpillars were made from a ball of plasticine of 1 cm diameter, and gently pressed/rolled onto the middle of a 12 cm-long metallic wire until a 3 cm-long caterpillar was obtained. Partners were instructed to attach the caterpillars to branches using wire, and leave the caterpillars on trees for 15 days prior to recording predation marks. Schoolchildren counted predation marks and attributed them to birds, mammals, arthropods, or reptiles. In 2018, they tagged and photographed every caterpillar with the suspected predation marks from any potential predator taxa. To minimise the probability of false negative results, we also advised the schoolchildren to send photographs of marks that were not clearly recognized as predation marks. Photos were taken from three different angles to show the observed damage and were labeled in such a way that the file name indicated both tree and caterpillar ID. Professional scientists were asked to gently remove all caterpillars from the trees and send them back to the project coordinators. One school also returned caterpillars, although this was not requested. A second survey using the same procedure immediately followed the first one. In 2019, both schoolchildren and professional scientists were instructed to send caterpillars back to the project coordinators. Photos and actual caterpillars were used by EVC to double-check and to standardize the predation assessment made by individual partners.

Every partner received a field bite guide containing a collection of photos illustrating predation marks left by different types of predators as well as false positive marks on plasticine surfaces that were made by leaves, buds, or finger nails. The different predator guilds that can be easily identified from their typical marks left on plasticine include passerine birds, rodents, snakes, lizards, and insects—mainly beetles and bush-crickets (Lövei and Ferrante 2017). The bite guide was available online and accessible to all partners through a hyperlink from the protocol (Castagneyrol et al. 2019), and teachers were invited to contact the scientific coordinator or local scientific partners in cases of uncertainty regarding the marks.

All partners were required to record their observations in the same standardized recording form. Partners indicated (a) the total number of caterpillars installed; (b) the number of caterpillars with any type of predation marks, (c) the number of caterpillars without predation marks; and (d) the number of caterpillars with predation marks left by birds (typically V-shaped beak marks and holes), arthropods (mandible marks), mammals (parallel teeth marks), or lizards (ellipse-shaped line of small teeth marks). Therefore, the same attacked caterpillar made a minimum of two entries in the recording form. We intentionally asked for redundant information to limit the risk of error in data reporting.

Data and biological material were collected by both schoolchildren and professional scientists during the same time period (from May through July). Project partners filled in the recording form and sent it to the project coordinators with the photos or the caterpillars. A single observer (EVC) with expertise in identifying predation marks on model caterpillars (Valdés-Correcher et al. 2019) screened every photo or caterpillar to verify observations reported by partners. It must be noted that false positives were more likely to be identified from the photos than false negatives. False positives are caterpillars classified by project partners as having been attacked when they were not. Because of previous reports (Low et al. 2014) and our own experience with undergraduate students trained to identify predation marks on artificial prey, we anticipated that schoolchildren and their teachers would be overly enthusiastic, making false positives more likely than false negatives. Schoolchildren were instructed to take photos of caterpillars with suspected predation marks, even marks they could not attribute to any predator type. It is therefore possible that they did not notice real predation marks on caterpillars that were photographed because they had marks left by buds, leaves, or finger nails. Such cases would represent false negatives. The probability of detecting false negative was not an issue when project partners returned caterpillars to the project coordinators.

For each oak tree and survey period, we assessed attack rate as the proportion of dummy caterpillars with at least one predation mark. Although we asked partners to record predation marks left by different types of predators (in particular birds and arthropods), this level of precision could not be reached on photos because of low resolution. Therefore, we quantified overall attack rate, regardless of predator type.

We estimated the precision and accuracy of attack-rate assessments by schoolchildren and untrained professional scientists by running two separate linear mixed-effect models with attack rate estimated by schoolchildren or professional scientists as a dependent variable, attack rate estimated by a single trained professional scientist and year (as factor) as independent variables, and Partner ID and Tree ID nested within Partner ID as random factors (Johnson, Bertrand, and Turcotte 2016). From each regression, we quantified the bias (a deviation between attack rate estimated by partners and a single trained observer) as the intercept (β₀). Positive deviation from β₀ = 0 indicates an overestimation of attack rate by partners. We quantified accuracy as the regression slope (β₁), where β₁ = 1 indicates high accuracy and β₁ ≠ 1 indicates that accuracy in attack-rate assessment varied with actual attack rate. We used parametric bootstrapping with 1,000 simulations to compute 95% confidence interval (CI) around β₀ and β₁ and estimate how they deviated from 0 and 1, respectively. The null hypotheses were that β₀ = 0 and β₁ = 1. We considered that the null hypothesis was rejected if the 95% CI did not bracket zero or one. The significance of the fixed effect of year was tested based on the F-distribution and estimating degrees of freedom with Kenward-Roger methods (Kuznetsova, Brockhoff, and Christensen 2017).

Insect herbivory

To compare insect herbivory estimated by schoolchildren versus trained and untrained professional scientists, we set up a complementary survey (administered by AB). In April 2019, we prepared 12 sets of 5 oak leaves randomly drawn from a large sample of oak leaves collected in September 2018 on 162 oak trees around Bordeaux city (SW France) and stored in paper bags at –18°C. For each set of leaves, five trained professional scientists with previous experience in scoring insect herbivory on oak leaves (BC, EVC, AB, TD, and YK [see acknowledgements]) estimated insect herbivory as the percentage of leaf area removed or impacted by insect herbivores by giving each individual leaf a damage score: (0: 0%, A: 1–5%, B: 6–15%, C: 16–25%, D: 26–0%, E: 51–75%, F: > 75%; Castagneyrol et al. 2013). To reduce variability in estimates of herbivory due to observers, we created digital model leaves with given amounts of simulated herbivore damage that were used as examples for the seven damage classes (Castagneyrol et al. 2019). Leaf chewers were the main source of insect herbivory on oak leaves, but because leaves were drawn at random from a large pool of leaves, some were attacked by leaf miners, although none had galls. We asked participants to score total insect herbivory, regardless of damaging agents. As a result, the damage score incorporated leaf area removed by chewers as well as covered by leaf mines.

We invited schoolchildren 11 to16 years old (and their teachers) from six local secondary schools (equivalent US grades 6–10) to visit the first author’s research facilities (INRA research station of Pierroton, Bordeaux, France). Five groups of 10 to 12 students were introduced to the study of insect herbivory by the survey administrator, who challenged them to score insect herbivory as accurately as professional scientists would do. Students worked in groups of 2 or 3, with a total of 24 student groups. Each group was given 3 sets of 5 leaves, selected at random from the pool of 12 leaf sets. All students scored damage using the same digital model leaves as a template. In total, each of the 12 leaf sets was processed by six independent groups of students.

The same day (or the day after), we invited INRA permanent and non-permanent staff members to participate in the survey. The volunteers were researchers, engineers, technicians, and Master of Science students. They were considered untrained professional scientists). They received the same information from the survey administrator as secondary school students and used the same templates to score herbivory. Each of the nine volunteers processed every set of five leaves.

We did not keep records of individual leaves and we therefore averaged herbivory estimates across leaves for each set. We first tested whether individuals with a different background differed in their estimation of insect herbivory by running Linear Mixed-effects Models (LMM) with (log-transformed) insect herbivory as a response variable, observer type (Observer) as a fixed-effect factor, and leaf-set identity and observer identity as random effect factors. Because repeated handling of the same leaves may have caused some breakage, leading to a progressively increased estimation of herbivory, we added Time (number of hours since the first assessment) and Time × Observer interactions as additional fixed effects in the model. The model equation was

where β₀ was the model intercept (i.e., Observer_{schoolchildren}), β₁ and β₂ were the coefficients of the fixed effects of the treatment for trained (Observer_Trained) and untrained professional scientists (Observer_Untrained), β₃ was the effect of Time, β₄ and β₅ were the effects of the Time × Observer interaction, γ_j and δ_k were the random intercepts for the observer and leaf-set identities, and ε_ijk were the residuals. For γ_j, δ_k and ε_ijk, we assumed a normal distribution with zero mean and variance σ²_γ, σ²_δ, and σ²_ε, respectively. σ²_ε contained variation among observers in scoring different leaf sets, i.e., the Observer × Leaf set interaction, but also all other noise. The significance of fixed effects was tested based on the F-distribution and estimating degrees of freedom with Kenward-Roger methods (Kuznetsova, Brockhoff, and Christensen 2017).

Second, we used σ²_δ to quantify consistency among observers in rating herbivory. To do so, we ran an intercept only LMM for each group separately (i.e., for students and for trained and untrained professional scientists) and calculated intraclass correlation (ICC) for the Leaf set random factor (σ²_δ/(σ²_δ + σ²_γ + σ²_ε)). ICC represents the proportion of the total variance that is explained by Leaf set identity. It is a metric commonly used to estimate repeatability (Nakagawa and Schielzeth 2010). The greater the ICC, the greater rating consistency among observers scoring the same leaf set. We used parametric bootstrap with 1,000 random draws to estimate ICC 95% CI.

All analyses were done in R (R Core Team 2018) using packages lmerTest and car (Fox et al. 2016; Kuznetsova, Brockhoff, and Christensen 2015).

Attack rate

In total, 7,338 dummy caterpillars were installed on 195 oak trees by 58 schools and 27 scientists. Schools and scientists’ data came from from 8 and 14 countries throughout Europe, respectively (Figure 1). Schoolchildren installed and returned 3,289 dummy caterpillars. They counted 1,802 of them as attacked by predators (i.e., 55%), whereas EVC counted only 868 caterpillars with predation marks (26%). Professional scientists installed 4,045 caterpillars, 1,629 of which they identified as attacked by predators (40%); EVC counted 1,338 of these caterpillars as attacked by predators (33%).

Attack-rate estimates by schoolchildren were more biased (intercept estimate ± 95% bootstrap CI: β₀ = 40.63 ± [22.45, 59.27]) than those by professional scientists (β₀ = 23.41 ± [13.11, 33.17]). Detailed examination of pairwise comparisons at the tree level reveals that 81.5% of assessments made by schoolchildren were above the 1:1 line (Figure 2), thus indicating overestimation of attack rate as compared with assessments made by a single trained observer.

There was no relationship between attack rates estimated by schoolchildren versus a single trained observer (slope estimate ± 95% bootstrap 95% CI: β₁ = 0.43 ± [–0.02, 0.90]), whereas professional scientists made more accurate assessments (β₁ = 0.66 ± [0.54, 0.77], Figure 2). Attack rates estimated by schoolchildren and professional scientists did not differ between years (F_{1, 53.9} < 0.01, P = 0.952 and F_{1, 23.5} = 0.22, P = 0.644, respectively).

Insect herbivory

Insect herbivory estimates by trained professional scientists were the lowest (mean ± SE = 9.00% ± 0.51%, range 2.20% to 19.6%) (Figure 3; Figure S1 in the Supplemental File), whereas insect herbivory estimates by untrained professional scientists were the highest (14.65% ± 1.01%, range from 3.80% to 62.00%) (Figure 3; Figure S1 in the Supplemental File). Schoolchildren estimates of insect herbivory were intermediate (11.55% ± 0.64%, range from 2.20% to 27.40%) (Figure 3; Figure S1 in the Supplemental File). Both untrained professional scientists and schoolchildren consistently overestimated insect herbivory compared wiht trained professional scientists (Figure S1 in the Supplemental File), but this effect was not statistically significant at α = 0.05 (F_2,31.9 = 2.79, P = 0.076) (Figure 3). Herbivory did not vary significantly with time (Time: F_{1, 28.5} < 0.01, P = 0.954; Time × Observer: F_{2, 33.0} = 0.62, P = 0.544).

Interestingly, ICC revealed that the consistency of herbivory estimates was comparable between trained professional scientists (ICC ± 95% CI: 0.58 ± [0.31, 0.84]) and schoolchildren (0.54 ± [0.22, 0.76]), whereas estimates made by untrained professional scientists were less consistent (0.44 ± [0.13, 0.67]).

Can schoolchildren collect data of sufficient quality for ecological research?

The main strength of CS programs, from a research perspective, is the collection power achieved by volunteers (especially if the data are independently verified). However, our findings proved ambiguous with respect to whether the resulting data are of sufficient quality to yield scientifically robust results. On the one hand, we clearly show that schoolchildren overestimated attack rate compared with trained professional scientists (Figure 2). They also tended to overestimate insect herbivory, but this effect was not significant at the common α = 0.05 threshold (Figure 4). On the other hand, professional scientists with mixed expertise in these fields also tended to overestimate attack rate and insect herbivory (Figures 2 and 3).

Importantly for the interpretability of the data, overestimation of attack rates was consistent across schools, as overestimation occurred in 81% of observations. Attack rates as assessed by professional scientists were, on average, slightly higher than attack rates re-estimated by a single trained observer. However, pairwise comparisons revealed that over- and underestimation of attack rates were more balanced in this group. In sharp contrast, we also found schoolchildren assessed insect herbivory in a more consistent way than untrained professional scientists did. Collectively, our results indicate that data provided by schoolchildren should be considered with caution, but the same holds true for data provided by untrained professional scientists.

Why did (so) many schools overestimate attack rate?

Overestimation principally arose from partners scoring scratch marks left by contact with buds or leaves as signs of predation (Figure 4). Other sources of overestimation of predation cannot be ignored. Although no teachers mentioned vandalism of experiments, researchers should be aware of this possibility, particularly when caterpillars are placed on trees in urban environments. This may lead to missing caterpillars falsely scored as attacked. In addition, schoolchildren were told by teachers that the aim of the study was to determine “who protects oaks” against herbivores. It is possible that schoolchildren (and their teachers too) felt they had to see predation marks because this is what they perceived as the aim of the experiment. However, although confirmation bias is more likely to occur in schoolchildren and their teachers, it is important to stress that this type of cognitive bias is also common among trained professional scientists who may have interpreted small cracks (for example) on the caterpillar surface as predation marks (Forstmeier, Wagenmakers, and Parker 2017; Zvereva and Kozlov 2019).

Although the protocol clearly specified how to standardize caterpillar size and shape, and emphasized the importance of standardization, we noticed that the dimensions of dummy caterpillars varied widely, both within and among schools. In other studies, the probability of detecting predation marks left by avian or arthropod predators was found to be influenced by the length and width of artificial caterpillars (Lövei and Ferrante 2017). It is unlikely that variability in the dimension of artificial caterpillars has affected the comparison of attack rate as estimated by schoolchildren versus trained observers. However, the variation found should be regarded as a potential source of bias in large-scale multi-partner studies. As a potential mitigation procedure, researchers can provide pre-made caterpillars to project partners (Roslin et al. 2017). That said, making caterpillars according to a standard protocol is also an important dimension of student training. Despite potential biases in data collection, the pedagogical aspects of citizen science programs at schools must not be neglected, and scientists must recognize trade-offs between scientific and pedagogic objectives when planning mitigation procedures. As a compromise, scientists could provide partners with a reference caterpillar made of hardened undeformable clay. 3D-printed models of caterpillars attacked by different predator types may also be included as examples. In any case, we advise that project partners be instructed to carefully pack caterpillars when sending these to lead scientists for calibration of predation assessment. We also recommend that data collected by schoolchildren are not directly used in the project—their value lays in the pedagogical outcomes—but that trained professional scientists use their own scoring on the material provided by schoolchildren.

Schoolchildren scored insect herbivory in a more consistent way than untrained professional scientists did

Johnson et al. (2016) found that bias in herbivory assessment decreased with the number of years of experience in herbivory assessment. Assuming that being trained as a scientist increases accuracy and the sense of rigor, we expected that herbivory would have been scored more accurately by untrained professional scientists than by schoolchildren. Our findings do not support this prediction. Although both schoolchildren and untrained professional scientists ranked the different leaf sets in the same order, for a given leaf set, schoolchildren always overestimated herbivory compared with trained professional scientists, with only one exception (Figure S1 in the Supplemental File), and untrained professional scientists always overestimated herbivory compared with schoolchildren, with only one exception (Figure S1 in the Supplemental File). However, this tendency was not statistically clear (Figure 3).

Interestingly, both schoolchildren and trained professional scientists assessed herbivory in a more consistent way than untrained professional scientists did, as revealed by the greater ICC in estimates. Individuals may vary in their observational skills, but training likely reduces this variability. Schoolchildren formed groups of 2 to 3 participants, while untrained professional scientists were alone when estimating herbivory. It is possible that within-group discussion leveled out intrinsic variability in observational skills and therefore variability of estimates made by schoolchildren. An alternative explanation for this unexpected finding is that schoolchildren took the activity more seriously than untrained professional scientists did. Regardless of the cause, these results stress that schoolchildren are no less reliable than untrained professional scientists when it comes to estimating insect herbivory (on oak leaves).

How can we make data collected by schoolchildren more reliable?

CS programs can help to generate a large amount of data, but the quality has been questioned, especially when these big data are not based on standard protocols (Bayraktarov et al. 2019; Burgess et al. 2016). Few studies have evaluated the quality of data collected by schoolchildren participating in CS programs (Miczajka, Klein and Pufal 2015; Saunders et al. 2018; Steinke et al. 2017). It emerges from these studies that schoolchildren can actually provide data accurate enough to support ecological research, provided that the tasks they are requested to undertake are adapted to their skills and that they receive proper training (Miczajka, Klein and Pufal 2015; Ratnieks et al. 2016; Saunders et al. 2018). Although we could not provide face-to-face training sessions for every school partner involved in the Oak Bodyguards project, the project methodology was simple and based on a detailed protocol. Nonetheless, this simplicity did not suffice to guarantee unbiased data, as illustrated by the fact that schoolchildren consistently overestimated attack rates. We therefore emphasize that CS programs relying on data collected by schoolchildren should include several checks of data quality and appropriate mitigation procedures. In particular, training sessions undertaken face-to-face or at least remotely must be planned before data collection (Ratnieks et al. 2016). Finally, whenever possible, the researcher analyzing the data should recover the raw material collected by children, or at the very least access pictures that allow for the re-assessment of measurements (Ekholm et al. 2019; Steinke et al. 2017). Importantly, these recommendations also hold true for large multi-partner research programs, as we also detected bias in data collected by professional scientists (Zvereva and Kozlov 2019). Whether variability in observations made by schoolchildren is random or can be modelled using appropriate covariates is an important question deserving further attention.