Introduction

Data quality is central to citizen science projects producing trustworthy scientific outcomes. As the citizen science approach became more popular, some scientists voiced concerns about the potential for volunteers to produce data sets without large amounts of error (Dickinson, Zuckerberg, and Bonter 2010) that would be able to detect changes in the population status of wild species (Danielsen et al. 2014). In response to those challenges, many best practices in citizen science now feature mechanisms to assess the quality of data. Most of this development has focused on the accuracy and trustworthiness of their observations (e.g., Crall et al. 2011; Hunter et al. 2013; Kosmala et al. 2016). However, even if citizen observations are accurate, there is an additional risk for sampling bias because most are opportunistic samples (Bird et al. 2014).

Because citizen science projects rely on volunteers to collect data, it is difficult to follow a systematic sampling design. For example, in a study of two Canadian aquatic monitoring programs, Millar et al. (2019) found clustered and biased sampling around lakeshore houses, which they referred to as the “cottage effect.” This spatial bias can affect biological inference; for example, Weiser et al. (2020) found strong bias when building statistical models that included non-probabilistically selected sites to survey butterflies (also known as preferential sampling; Diggle, Menezes, and Su 2010), which are typical of citizen science, although they noted that this might be less of a problem with larger data sets. There is some evidence that larger sample sizes might help offset these biases, as spatial models with different sampling schemes were similar for large data sets on toads in the United Kingdom (Petrovan, Vale, and Sillero 2020). Similarly, Callaghan et al. (2020) found that citizen science data can be as good as professional records for continental diversity mapping once a minimum sample size is met. There has also been work on accounting for biases in data collection during analysis. For example, Johnston et al. (2020) showed how spatial bias can be corrected for in spatial models by weighting based on the probability of an area being sampled, and Isaac et al. (2014) showed how filtering and data corrections could improve power to detect trends. However, both of these studies highlight the limitations of post-hoc corrections, emphasizing the importance of reducing bias at the point of data collection.

Here we address the questions of spatial bias and sample-size requirements in a citizen science data set collected as part of the statewide North Carolina’s Candid Critters (NCCC) camera trapping wildlife survey. Recognizing these potential problems at the start of the project, we created an a priori study design. During the study, we monitored our progress and encouraged volunteers to run cameras in sites that would help meet those sampling goals. For the most poorly sampled areas, we supplemented the volunteer data with professional data collection, employing a hybrid citizen/professional scientist sampling program. We refer to this as the “Plan, Encourage, Supplement” strategy, which is similar to the strategies introduced by Callaghan et al. (2019).

The objective of the NCCC project was to document the distribution and ecological relationships of mammals in the state through statistical models that relate the occurrence of species in the camera traps to environmental data such as land cover, habitat type, or degree of human disturbance. In this paper, our goal is to assess potential biases in citizen science data by comparing the dimensions of habitat sampled by our cameras with those available in the state, using a minimum sample threshold approach (Callaghan et al. 2020). We also evaluate the sample-size robustness by comparing the relative bias and precision of wildlife metrics (i.e., species occupancy) calculated with subsets of the full data set. This subsetting approach provides insight into the extent to which finer-scale comparisons can be made between regions, seasons, or less common species that might naturally have fewer detections with a given survey effort. We expect these results will be broadly useful for professionals setting up citizen science projects who plan to use opportunistic data to study ecological relationships and want to ensure they end up with enough samples to statistically test hypotheses, and the samples are distributed to be representative of the larger study area.

Methods

The North Carolina’s Candid Critters project worked with citizen scientists and the North Carolina Wildlife Resources Commission personnel to survey wildlife with camera traps across the state over 3 years (Lasky et al. 2021a). Camera traps have advantages over some traditional wildlife data sources because they collect data across all seasons and times of day on a variety of warm-blooded species > 100 g. The photos allow species identifications to be verified (all NCCC pictures were reviewed by experts), and the duration a camera is deployed gives a clear record of sampling effort. The citizen science approach also allowed us to sample private land (52% of sites), for which it is typically difficult for researchers to obtain permission, even though it can represent a large part of a study area (here 86% in southern forests, Butler and Wear 2013).

However, because most (65%) of our cameras were set by volunteers, we had limited ability to determine where exactly they were set ahead of time. Nonetheless, as part of our Plan, Encourage, Supplement strategy, we had a priori sampling goals, and recruited and encouraged volunteers to set cameras in ways that would meet those goals. We supplemented by having staff cameras, as well. When planning the project, we used counties as an operational unit, aiming to get samples from all 100 counties. We initially quantified the land-cover types across the state as open, forested, or developed, and quantified the proportion of each primary land-cover type across counties (Lasky et al. 2021a). We then used this as a guideline for creating our study design on public lands, and for monitoring where cameras were run on private lands during the study, aiming for the simple goal of having samples in proportion to available land-cover types in each county. However, when evaluating if our data represents the state, we focused on whether we had an adequate sample of a given habitat type (to discover animal-habitat relationships) rather than an exact proportional match to what is available, since analyses were done at larger scales than county (i.e., state or ecoregion).

As the study progressed, we noticed areas that were being under sampled, especially rural areas, and designed campaigns to obtain more sampling in those locations by recruiting volunteers locally or by encouraging existing volunteers to travel outside their home county to set cameras on public land (e.g., visit a state park). This encouragement was done as part of our ongoing engagement activities with volunteers; it included newsletters, social media, and webinars (Lasky et al. 2021a), and is similar to the dynamic incentives suggested by Callaghan et al. (2019). If sampling was still lower than project goals, we supplemented the data by deploying cameras ourselves or by working with professional partners to run cameras in specific target areas.

Spatial bias

A recent empirical assessment of camera-trapping study design recommends a minimum of 40 to 60 camera points per habitat category (Kays et al. 2020). Based on this, we considered > 40 camera points to be an adequate sampling of a given habitat type, and > 60 to be a very adequate sample. Although this sample-size criteria might still not be sufficient for very rare species, it was derived from a wide variety of species and sites and serves as a good benchmark for our objectives.

Our approach (Box 1) to evaluate the representativeness of our sampling involved categorizing a variety of habitat types in the state, and evaluating whether each dimension was adequately sampled by our cameras. We did this first at the state level, but also at the finer-scale level of ecoregions. There are three primary ecoregions in North Carolina (mountains, piedmont, coastal), and using this division allowed us to also evaluate if ecological relationships could be different in these different regions. To describe the habitats of the state, we generated 10,000 randomly distributed points across the state, and after removing those in open water, used 9,586 random points in terrestrial habitats (Figure 1). At each of these points, we quantified the habitat with seven different covariates describing the natural and human infrastructures that are likely to affect animal distribution (Table 1). We then broke the variation in these each of these ecological covariates into bins, each representing 10% of the total statewide variation as measured by the random points (Figure 2). In the case of highly skewed distributions, we had fewer than 10 bin levels of available habitat because one category would have a very high proportion of the total variation (e.g., 60% of random points had 0 roads within 250 m). We used these categories to describe the dimensions of variation of habitat in the state, and in each ecoregion.

Box 1 Overview of our approach for determining spatial bias in NCCC camera sites Objective: Determine how representative the locations sampled by NCCC cameras were compared with all available habitat in the state. Criteria: If a habitat type was sampled by > 40 camera sites it was judged as adequate, while those with > 60 were very adequate. Habitat Availability: Habitat variation in the state was quantified using 9,586 random points. At each point, we measured seven different aspects of habitat (Table 1). For each habitat variable we split the statewide variation into 10 categories (bins in Figure 2). We then used the above criteria to evaluate if each bin of each habitat variable had been sufficiently sampled.

Figure 1 

Map of the 9,586 random points (grey) across North Carolina, USA used to quantify variation in available habitat types, and the 4,295 sites sampled with camera traps (blue). Black lines indicate the three primary ecoregions (from west to east): mountains, piedmont, coastal.

Table 1

Habitat variables considered for the analysis of how representative the data were of the state.

HABITAT	DESCRIPTION	UNIT	SOURCE
Cover type	Open, forest, developed, or other		(Homer et al. 2015)
Elevation	Elevation	m	USGS
Large core forests	Area within a 5 km radius consisting of continuous forest fragment (forest parcels >2 ha)	%	(Homer et al. 2015)
Developed	Area within a 5 km radius consisting of developed land use	%	(Homer et al. 2015)
Houses	Housing density within 1 km radius of the site	houses/km2	(Hammer et al. 2004)
Roads	Road density within a 250 m radius of site	km/km2	NCDOT
Tree Cover	Tree cover at 30 m pixel resolution	%	(Hansen et al. 2013)

Figure 2 

Distribution of elevational habitats available in North Carolina measured at 9,586 random points across the states. Colors show the 10 categories that each represent ~10% of the variation in available habitat. We used these categories to see if our camera samples were representative of the dimensions of a given habitat type.

For each of the seven habitat covariates (i.e., elevation, road density, etc.; Table 1), we then quantified the habitat at the 4,295 sites sampled by our camera traps and placed them into the same ten bins as the available habitat (Figure 2) to evaluate if we had an adequate (> 40) or very adequate (>60) sampling in each dimension. Because ecological relationships might be different across the state (i.e., animals in the mountains might be more or less likely to occupy a site based on local forest cover than animals on the coast), we also considered how many camera traps fit into these categories separately in the three major ecoregions of the state (coastal, piedmont, and mountains).

Results

Sample size robustness

Occupancy estimates were robust to changes in sample size, with error and bias falling below our goal of 0.1 quickly (i.e., 10%; Figure 3; Supplemental Figure 1). Not surprisingly, error and bias fell below our threshold for the more common deer, with smaller sample size than the less-common coyote. We also plotted the sensitivity of ecological relationships described by the model to changes in sample size in terms of error and bias (Supplemental Figure 2). The error associated with estimating these relationships approached, but did not meet, our 10% goal in most cases. Interestingly, for a given number of camera sites, error was lower for the more restrictive season-region models than the season-only or region-only (Figure 4), suggesting a fair amount of spatio-temporal variability in these relationships. Bias, on the other hand, decreased rapidly after few hundred sites in most cases (Supplemental Figure 2).

Figure 3 

Graphs showing relative bias (a measure how different the estimates were from the full data set estimate) of occupancy estimates for white-tailed deer and coyote. These were calculated with subsets of the full NCCC dataset for a) three ecoregions, b) four seasons, and c) ecoregion-seasons. Our estimates reached our goal for bias (<0.1) at very small sample sizes for the common deer and after sampling 250–300 sites for the less common coyote across all spatio-temporal divisions. The lack of change in these estimates with increasing sample size also indicates a stable, robust result. Results were similar for estimates of error (Supplemental Figure 1).

Figure 4 

Graphs showing the changes in the relative error (relative root mean square error [RRMSE], a measure of how variable the estimates of occupancy were across the replicates) with larger sample size for estimated ecological relationships for agriculture land cover in occupancy models for coyotes and deer. Models were run across regions (top), seasons (middle), and regions-seasons (bottom). Only significant model effects are shown. Error estimates approached our 10% goal more rapidly with more restricted models (i.e., region-season) suggesting spatio-temporal variability in these relationships added variation to the larger-scale models. Full results for changes in error and bias of all covariates are available in Supplemental Figure 2.

Discussion

The opportunistic nature of sampling within most citizen science projects puts them at risk of providing biased results, especially if sample size is low. We used a Plan, Encourage, Supplement strategy to try to obtain a balanced sample by having a priori sampling goals, by encouraging volunteers to help meet those goals, by setting additional cameras ourselves in underrepresented areas, and in general, by obtaining a large sample size. Here we present a unique approach to evaluating the representativeness of our data, comparing it with a random sample of seven environmental covariates (i.e., dimensions of habitat) at both the statewide and ecoregion levels. We show that we obtained an adequate sample across the spectrum of variation in these habitat dimensions at the state level, and for most of the variation at the ecoregion level. Dimensions that were under sampled were rare, such that we adequately sampled 99.2% of the area of the state at the ecoregion level, and very adequately sampled 96.4%. Furthermore, sample sizes were robust enough to estimate occupancy for important wildlife species, although some of the ecological relationships we discovered between covariates and occupancy were sensitive to reductions in sample size. Additionally, our subsetting exercise suggested these ecological relationships are variable over space and time (i.e., nonstationary).

This analysis shows that citizen scientists can obtain representative data at large scales and demonstrates how a hybrid study design can improve sample coverage. By having a priori sampling goals, we were able to encourage volunteers to sample in ways that gave us a more balanced sample, which was sufficient in the more populated piedmont ecoregion. This approach also would have obtained good coverage in the mountains and coastal ecoregions (92% and 98% coverage, respectively), but by supplementing this with data collected by staff, we were able to boost coverage to > 98% for both. This approach is also cost-effective over large areas (Lasky et al. 2021a), and provides the opportunity to sample on private land, which makes up 86% in the region (Butler and Wear 2013), but is difficult to sample without involving citizens. We think this Plan, Encourage, Supplement strategy could be useful to others designing citizen science projects, whether they are trying to get a representative sample of a region, as we did, or are targeting specific rare species or habitats.

As citizen science databases grow, they are more likely to be used in biodiversity modeling research projects. However, some question whether big unstructured, biodiversity data sets typical of citizen science actually mean more knowledge (Bayraktarov et al. 2019). This is less of a concern for sensor-based citizen science projects, including camera traps, because the primary data come from the sensor, and is thus more structured. In the case of camera traps, this means that effort is automatically calculated (time in the field), data quality can be checked (by examining photographs), and the likely absence of a species can be inferred (from lack of pictures). However, even with accurate measures from the sensors, there is still a risk of spatial bias based on where the sensors are actually deployed by citizens, especially if site choice is opportunistic (Weiser et al. 2020). Our approach for detecting spatial bias in relation to seven dimensions of habitat is unique, and the results provide more confidence in the utility of this data set specifically, and the potential for the Plan, Encourage, Supplement approach in general.

Our decision to categorize the habitat variation seen across the state into ten bins, and then check if each of those categories was adequately sampled, was somewhat arbitrary. Spreading the variation into more categories would certainly result in fewer being adequately sampled by our scheme. However, we felt that having each category represent 10% of the variation observed in the state was a biologically relevant measure. We also recognize that there are other more fine-scale ways to classify habitat types, for example, recognizing categorical difference in forest type based on tree-species composition. However, categorical fine-scale habitat classifications are not often used in large-scale ecological models, because most types are absent from a given area, requiring exhaustive sampling to get coverage. Such an analysis would undoubtably uncover that some rare habitat types were under sampled. In addition, there are some habitats that are not realistic to sample with camera traps set by volunteers or professionals (e.g., cliffs, pocosins, swamps).

We used occupancy models as our gauge of how sensitive ecological results from this data set would be to changes in sample size, because these are one of the most common metrics derived from this type of data. To allow for spatial and temporal variation in these relationships, we conducted separate analyses for three ecoregions, four seasons, and also twelve region-season combinations. Bias and error dropped below our goal of 0.1 with relatively small sample sizes for occupancy estimates. However, the precision of our estimation of ecological relationships was less robust. Interestingly, there was quite a lot of variability in which factors were significant depending on the spatiotemporal grouping of the data, suggesting substantial local differences in the ecology of these species (nonstationarity). For example, agriculture was a significant predictor of deer distribution in the costal and mountain regions but not the piedmont, and in the fall, winter, and spring, but not in the summer (Supplemental Figure 2). Likewise, the more restricted region-season model reached lower error with fewer samples than the larger-scale models, further suggesting nonstationary ecological relationships. These results show how larger data sets can enable more detailed research questions (e.g., seasonal or geographic differences in ecology of a species) and in our case, still estimate occupancies accurately, although with less confidence in specific ecological relationships. Although more data could help untangle these ecological nuances more precisely, this density of data could also be useful for more sophisticated spatiotemporal modeling approaches designed to accommodate this type of dynamic (Meehan, Michel, and Rue 2019).

In summary, we showed that sensor-based citizen science projects can obtain a robust sample size that is representative of habitat variation across large scales. We implemented the dynamic incentives strategy of Callaghan et al. (2019) as a three-part Plan, Encourage, Supplement strategy that can be useful for other citizen science projects, and we think that that our approach for assessing the representativeness of opportunistic samples could be applied to other data sets before using them in modeling exercises.

Data Accessibilty statement

The camera trap data used in these analyses is available in (Lasky et al. 2021b).

Supplementary Files

The supplementary files for this article can be found as follows: