Start Submission

Reading: Do You See What I See? Quantifying Inter-Observer Variability in an Intertidal Marine Citize...

Download

A- A+
Alt. Display

Research Papers

Do You See What I See? Quantifying Inter-Observer Variability in an Intertidal Marine Citizen Science Experiment

Authors:

Hannah Earp ,

Aberystwyth University, GB
X close

Siobhan Vye,

Bangor University, GB
X close

Katrin Bohn,

Natural England, GB
X close

Michael Burrows,

Scottish Association of Marine Sciences, GB
X close

Jade Chenery,

Newcastle University, GB
X close

Stephanie Dickens,

Newcastle University, GB
X close

Charlotte Foster,

Newcastle University, GB
X close

Hannah Grist,

Scotland’s Rural College, GB
X close

Peter Lamont,

Scottish Association of Marine Sciences, GB
X close

Sarah Long,

University of Portsmouth, GB
X close

Zoe Morrall,

University of Portsmouth, GB
X close

Jacqueline Pocklington,

Deakin University, AU
X close

Abigail Scott,

James Cook University, AU
X close

Gordon Watson,

University of Portsmouth, GB
X close

Victoria West,

ABPmer, GB
X close

Stuart Jenkins,

Bangor University, GB
X close

Jane Delany,

Newcastle University, GB
X close

Heather Sugden

Newcastle University, GB
X close

Abstract

Citizen science represents an effective means of collecting ecological data; however, the quality/reliability of these data is often questioned. Quality assurance procedures are therefore important to determine the validity of citizen science data and to promote confidence in conclusions. Here, data generated by a marine citizen science project conducted at 12 sites across the United Kingdom was used to investigate whether the use of a simple, low-taxonomic-resolution field-monitoring protocol allowed trained citizen scientists to generate data comparable to those of professional scientists. To do this, differences between field estimates of algal percentage cover generated by different observer units (i.e., trained citizen scientists, professional scientists, and combined units), and digitally derived baseline estimates were examined. The results show that in the field, citizen scientists generated data similar to those of professional scientists, demonstrating that training, coupled with the use of a simple, low-taxonomic-resolution protocol can allow citizen scientists to generate robust datasets in which variability likely represents ecological variation/change as opposed to observer variation. The results also show, irrespective of observer unit, that differences between field and digital baseline estimates of algal percentage cover were greatest in plots with medium levels of algal cover, highlighting that additional/enhanced training for all participants could be beneficial in this area. The approach presented can serve as a guide for existing and future projects with similar protocols to assess their data quality, to strengthen participant training/protocols, and ultimately to promote the incorporation of robust citizen science datasets into environmental research and management.

How to Cite: Earp, H., Vye, S., Bohn, K., Burrows, M., Chenery, J., Dickens, S., Foster, C., Grist, H., Lamont, P., Long, S., Morrall, Z., Pocklington, J., Scott, A., Watson, G., West, V., Jenkins, S., Delany, J. and Sugden, H., 2022. Do You See What I See? Quantifying Inter-Observer Variability in an Intertidal Marine Citizen Science Experiment. Citizen Science: Theory and Practice, 7(1), p.12. DOI: http://doi.org/10.5334/cstp.483
305
Views
98
Downloads
10
Twitter
  Published on 04 May 2022
 Accepted on 01 Apr 2022            Submitted on 14 Dec 2021

Introduction

Many marine ecosystems are at risk of degradation and decline as a result of multiple interacting global, regional, and local stressors, including climate change, pollution, and overfishing (Halpern et al. 2008; Pecl et al. 2017; Poloczanska et al. 2013). To detect and fully understand the impact of stressors on marine ecosystems, and to provide evidence that supports conservation measures, baseline information and continued monitoring over large spatiotemporal scales and including a diverse array of species is often required (Magurran et al. 2010; Mieszkowska et al. 2014). The establishment and persistence of large, long-term monitoring programmes however, requires sustained financial investment alongside a trained “workforce” (Cox et al. 2012).

In recent decades, citizen science, widely defined as public participation in scientific research, has emerged as a powerful and cost-effective means of generating extensive ecological data sets across a multitude of ecosystems (Bonney et al. 2009; Dickinson, Zuckerberg, and Bonter 2010; Earp and Liconti 2019). Involvement in such projects can increase environmental stewardship as participants develop a greater appreciation of natural environments (Cerrano, Milanese, and Ponti 2017; Evans, Gebbels, and Stockill 2008), and can enhance participant wellbeing through mental stimulation, social interaction, and improved fitness (O’Brien, Townsend, and Ebden 2010). In the marine realm, citizen scientists have participated, both in person and more recently online, in monitoring species and environmental conditions across ecosystems including coral reefs (Branchini et al. 2015; Marshall, Kleine, and Dean 2012; Mumby et al. 1995), seagrass meadows (Finn et al. 2010), temperate rocky reefs (Gillett et al. 2012; Rosenthal et al. 2018), and many more (Earp and Liconti 2020; Sandahl and Tøttrup 2020; Thiel et al. 2014). Yet, despite its capacity to broaden the scope of monitoring initiatives, citizen science has yet to be fully embraced as a valid means of scientific investigation (Bonney et al. 2014), primarily due to concerns regarding data quality (Balázs et al. 2021; Burgess et al. 2017; Freitag, Meyer, and Whiteman 2016; Penrose and Call 1995).

The accuracy of citizen science–generated data may vary depending on the knowledge and/or experience of the participants, the task level, and the ecosystem in question (Kosmala et al. 2016). Techniques suggested to minimize error and/or biases and increase the robustness of citizen science data include retaining trained and/or experienced participants, training participants to use standardized protocols, reducing the taxonomic resolution or complexity of the task/protocol, verifying data (i.e., the checking of record correctness by experts), and comparing data collected by citizen scientists and professional scientists (Baker et al. 2021; Clare et al. 2019; Cox et al. 2012; Kosmala et al. 2016; Parsons, Lukyanenko, and Wiersma 2011; Rambonnet et al. 2019; van der Velde et al. 2017; Zettler et al. 2017).

A growing body of research has compared data collected by citizen and professional scientists to demonstrate that citizen scientists can generate data of similar and potentially greater quality than professional scientists (Cox et al. 2012; Crall et al. 2011; Finn et al. 2010; Fore, Paulsen, and O’Laughlin 2001; Forrester et al. 2015; Koss et al. 2009; Rosenthal et al. 2018; van der Velde et al. 2017). However, Koss et al. (2009) found that on subtidal rocky shores, estimates of percentage cover of algal species differed significantly between citizen and professional scientists, potentially due to misidentification and/ or confusion between morphologically similar species. Here, we use data collected as part of an intertidal ecology experiment conducted by a marine citizen science program to examine whether the use of a simple, low-taxonomic-resolution algal monitoring protocol allows citizen scientists to generate data comparable to those of professional scientists. To do this, we provide a novel means of assessing the reliability of field estimates of algal percentage cover generated by different types of observers (e.g., citizen scientists, professional scientists, and combined units of citizen scientists working with professional scientists), by comparing them to digitally derived baseline estimates.

Firstly, we assessed differences between algal percentage cover estimates generated by a subgroup of citizen scientists in the field and digital baseline estimates generated by a professional scientist using three different digital analysis techniques. We then investigated differences between estimates of algal percentage cover generated by trained citizen scientists, professional scientists, and combined units in the field, and a single digital baseline estimate generated by a professional scientist (which represents a consistent method of estimation without observer bias). We also explored whether differences between field and digital baseline estimates of algal percentage cover were influenced by the level of algal coverage. This provided a simple means by which citizen science programmes with similar monitoring protocols may investigate the quality of their data, and ultimately, further evidence of the robustness of citizen science data to support its use in research and management.

Materials and Methods

Citizen scientists

The citizen scientists involved in this study were recruited and trained as part of the Capturing our Coast marine citizen science program (www.capturingourcoast.co.uk) that ran in the United Kingdom from 2015 to 2018. All citizen scientists attended a one-day training session delivered by professional marine scientists that covered rocky shore ecology, species identification, and monitoring methods. Training was supported by the distribution of reference materials and ongoing support (including field refreshers) from program staff to aid data quality and participant retention.

Study system

The study was conducted at moderately exposed to exposed shores (Burrows 2012, 2020) across the United Kingdom, including the west coast of Scotland (3 sites), north Wales (4 sites), northeast England (3 sites) and southwest England (2 sites) (Supplemental files 1 and 2: Appendix A). Sites were selected based on their proximity to Capturing our Coast training hubs, and thus trained citizen scientists, as well as their accessibility for participants from a range of demographics. At each site, surveys were undertaken on the low shore, where algal communities were dominated by the canopy-forming macroalgae, Fucus serratus.

Field surveys

Data were collected as part of an intertidal ecology experiment exploring rates of recovery from manipulated disturbances. Experimental plots of 0.25m2 were surveyed seasonally from autumn 2016 to spring 2018 by observer units comprised of (1) a group of 2 to 3 trained citizen scientists, (2) a professional scientist (i.e., individuals with a formal qualification in marine science who had undergone Capturing our Coast staff training), or (3) a combination of a citizen scientist and a professional scientist working together. Firstly, a non-gridded quadrat was placed over the plot, and a top-down photograph of the quadrat was taken. Algal percentage cover (0–100%) was then assessed by placing a 0.25m2 gridded quadrat with 100 squares over the plot and estimating the number of squares covered by live, attached algae (including all canopy, turf, and encrusting species). To account for individual learning styles and approaches to visualisation (Gardner 1993), participants could estimate percentage cover using various methods explained during the one-day training session, including summing up partially covered squares to make whole squares, or considering covered areas as patches and visualising the number of squares they would cover if all the patches were moved together. The visualisation method utilised by different participants was not recorded, nor was the date of the most recent survey experience, or the approximate number of surveys they had undertaken previously. The simple protocol employed was selected over others to facilitate the collection of robust data while encouraging involvement from a broad range of participants (Deither et al. 1992; Parsons, Lukyaneko and Weisma 2011).

Differences between field estimates and digital baseline estimates generated using three different techniques

Quadrat photographs were analysed by one professional scientist using Coral Point Count with Excel extensions [v4.1] (Kohler and Gill 2006). The point-intercept method was used to generate digital baseline estimates of algal percentage cover that were then compared with field estimates. To do this, a digital grid was positioned over each quadrat image in three different ways (Figure 1), and the underlying substrate at each grid intercept was identified as one of three categories: (1) live attached algae (including all canopy, turf, and encrusting species); (2) other (i.e., bedrock, dead encrusting algae, invertebrates); or (3) the quadrat frame.

Visualisation of the three methods of digital grid placement used to generate digital estimates of total algal percentage cover
Figure 1 

Comparison of the three different methods of digital grid placement. Grey squares represent the quadrat frame and coloured squares represent the digital grid outline (not including the intercepts). (a) Method 1 (purple) had a 10x10 point grid, (b) Method 2 (green) had a 10x10 point grid, (c) Method 3 (yellow) had a 13x13 point grid.

The first method of digital grid placement involved placing a 10x10 point grid within the largest possible area of the plot while including as little of the quadrat frame as possible (Figure 1a). Algal percentage cover was then calculated as (Total algal intercepts/(Total number of intercepts – Total quadrat frame intercepts) * 100). The second method involved placing a 10x10 point grid within the largest possible area of the plot while excluding all of the quadrat frame (Figure 1b). Algal percentage cover was given as the total number of algal intercepts. The third method involved placing a 13x13 point grid across the entire area of the plot while including as little of the quadrat frame and beyond as possible (Figure 1c). A 13x13 point grid was used as it was revealed to be the minimum grid size that ensured at least 100 intercepts fell within the plot area. Intercepts identified as quadrat frame or beyond were removed, and of the remaining intercepts, 100 were randomly selected to give an estimate of algal percentage cover.

Differences between estimates of algal percentage cover generated by two randomly selected trained citizen scientist units in the field were compared with digital baseline estimates (generated by a professional scientist using the three different methods of grid placement) across 15 plots. Differences were determined by subtracting the digital baseline estimate from the field estimate.

The three methods of grid placement (and thus baseline estimate generation) were selected because they are less subjective, complex, and time consuming to implement compared with photograph manipulation to make the quadrats and digital grids align (which is not always possible due to the angle of some photographs) or with digital area analysis (which would be challenging given the patchy nature of certain algal species). The three methods were compared over a limited number of quadrats to ensure they generated analogous data and that the method selected for subsequent analyses represented a robust means of baseline estimation.

Differences between field estimates and digital baseline estimates generated using one technique across different observer units and levels of algal cover

Observer units that surveyed and photographed ≥ 8 plots per survey (i.e., per site, per season) were identified. This comprised 11 citizen scientist units (i.e., a group of 2 to 3 citizen scientists that had attended a one-day Capturing our Coast training session), 18 professional scientist units (i.e., an individual with a formal qualification in marine science that had undergone Capturing our Coast staff training), and 13 combined units (i.e., one citizen scientist and one professional scientist working together), giving a total of 42 observer units. From each observer unit, eight of the surveyed plots were randomly selected for digital analysis (n = 336 plots). The low quality of one photograph meant digital analysis was not possible, and because no substitute plot from the same observer unit was available, n = 7 was used for one citizen scientist observer unit.

Digital analysis was conducted by one professional scientist who generated a baseline estimate of algal percentage cover using the third method of grid placement (Figure 1c). The third method of digital grid placement was selected over other methods because it incorporated the greatest area of the plot and ensured 100 intercepts within each plot. Differences between field and digital baseline estimates of algal percentage cover were determined by subtracting the digital baseline estimate from the field estimate. A categorical variable for the level of algal cover was generated based on the field estimates of total algal percent cover, with 0% to 25% considered low, 26% to 50% considered low-mid, 51% to 75% considered mid-high, and 76% to 100% considered high.

Data analyses

To investigate differences between estimates of algal percentage cover generated by a subgroup of citizen scientists in the field and digital baseline estimates generated by a professional scientist using three different digital analysis techniques, a linear mixed effect model (LMER) was used. Method type (i.e., field, digital method 1, digital method 2, digital method 3) was set as a fixed factor, and quadrat ID and date were set as random factors. Site and season were not considered factors in this analysis because we were not investigating spatiotemporal variability in algal coverage, but rather variability across methods of estimating algal coverage. Data were converted to proportions (divided by 100), and logit transformed prior to analysis, with 0.025 added or subtracted from proportions equal to 0 or to 1 (Fox and Weisberg 2019; Warton and Hui 2011). Data were visualised as the difference between the field estimate and each of the three different digital baseline estimates.

To determine whether differences between estimates of algal percentage cover generated by trained citizen scientists, professional scientists, and combined units in the field and digital baseline estimates generated by one professional scientist using the third method of digital grid placement (Figure 1c) were significant, and whether differences were influenced by the level of algal cover, a LMER was used. Observer unit (i.e., citizen scientists, professional scientist, combined), algal cover (i.e., low, low-mid, mid-high, high) and the interaction between these factors were set as fixed factors, and quadrat ID and date were set as random factors. As above, site and season were not considered factors in the analysis because the aim was to understand variability in the capacity of different observer units to correctly estimate algal cover. Data were square root transformed prior to analysis. Data were visualised as the difference between the field estimate and the digital baseline estimate, with lower variation assumed to indicate greater accuracy of field estimates.

All analyses and plotting were undertaken in the statistical software R [v.4.1.0] (R Core Team 2021). Models were generated using the lme4 package (Bates et al. 2015), and model fits were determined through visual examination of the quantile-quantile (QQ) and residual versus fitted values plots. Type II sum of squares were calculated using the Anova function of the car package (Fox and Weisberg 2019). Post hoc Tukey-adjusted comparisons were generated for individual fixed effects using the glht function of the multcomp package (Hothon, Bretz, and Westfall 2008). Graphs were produced using the ggplot2 package (Wickham 2016).

Results

Differences between field estimates and digital baseline estimates generated using three different techniques

Differences between field and digital baseline estimates of algal cover were not consistent across quadrats, nor the three digital baseline estimation methods, although in general, field estimates were lower than digital baseline estimates (Figure 2). However, estimates of algal percentage cover were not significantly different across the three estimation methods (χ2 = 4.8105, df = 3, p = 0.1862).

Graph showing differences between field estimates of algal percentage cover generated by trained citizen scientists and digital baseline estimates generated by a professional scientist using the three different methods of grid placement
Figure 2 

Differences between field estimates of algal percentage cover generated by trained citizen scientists and digital baseline estimates generated by a professional scientist using the three different methods of grid placement across 15 plots. Coloured circles represent differences, calculated as a field estimate minus a digital baseline estimate, per digital baseline method (Figure 1). Black squares and error bars represent the mean difference across the three baseline estimation methods ±1 standard error.

Differences between field estimates and digital baseline estimates generated using one technique across different observer units and levels of algal cover

Differences between field and digital baseline estimates of algal percentage cover were, on average, greater for citizen scientist units (mean ± 1 SE; –3.68 ± 1.07) compared with professional and combined units (mean ± 1 SE; –0.74 ± 1.06 and –1.21 ± 1.29 respectively), although the range of difference values was greater for the latter two units (Figure 3a; Supplemental file 3: Appendix A). However, overall, differences did not significantly differ across the different observer units (χ2 = 6.1124, df = 2, p = 0.9453; Supplemental file 4: Appendix A), nor did different types of observers vary in their capacities to estimate algal percentage cover based on the field-estimated level of algal cover (χ2 = 6.1522, df = 6, p = 0.3070).

Two graphs showing differences between field and digital baseline estimates of algal percentage cover. Graph A shows differences across the different types of observer units (citizen scientists, professional scientists, and combined units). Graph B (at the bottom of the figure) shows differences across different levels of algal cover (low, low-mid, high-mid and high)
Figure 3 

Differences between field and digital baseline estimates of algal percentage cover across (a) three different types of field observer units and (b) different levels of algal cover. Yellow circles represent differences, calculated as field estimates minus digital baseline estimates (generated using the third method of digital grid placement; Figure 1c) per plot. Black squares and error bars represent the mean variation across all plots ±1 standard error. Values in turquoise represent the number of plots. Significant post hoc Tukey-adjusted comparisons for single fixed effects are indicated as *** = p < 0.05.

Differences between field and digital baseline estimates of algal percentage cover were, however, found to be related to the level of algal cover alone (χ2 = 58.5496, df = 3, p < 0.000). Average differences were greatest for plots with a medium level of algal cover (mean ± 1 SE; low-mid (26–50%) –6.40 ± 3.16 and mid-high (51–75%) –7.02 ± 2.21) compared with plots with high (76–100%) and low (0–25%) levels of cover (mean ± 1 SE; 0.78 ± 0.47 and –1.06 ± 1.69, respectively; Figure 3b). Pairwise post hoc analyses revealed that the most significant differences were between plots with a high (76–100%) level of algal cover compared with those with low-mid (26–50%) and mid-high (51–75%) levels of cover (Figure 3b; Supplemental file 4: Appendix A).

Discussion

Concerns regarding data quality remain a key barrier to the wider use of citizen science datasets (Burgess et al. 2017; Penrose and Call 1995). Therefore, data verification and comparisons with professionally generated data are critically important to determine data validity and to promote confidence in the conclusions. This study demonstrates a novel means of digitally assessing field data collected by citizen scientists, and places variation in the data collected by trained citizen scientists and professional scientists in the same context. We demonstrate that when using a simple, low-taxonomic-resolution field-monitoring protocol, trained citizen scientists can generate estimates of algal percentage cover comparable to those of professional scientists. We also show that irrespective of the type of observer, field estimates of algal cover are most variable in plots with medium (26–50% and 51–75%) levels of algal cover. Here we discuss these trends, identify areas for further research, and provide considerations for future citizen science programmes. We also highlight the strengths and limitations of digital verification and field approaches to citizen science.

Trends in estimates of algal percentage cover and considerations for future work

Trained citizen scientists within this study generated data that were comparable to those of professional scientists, which is consistent with findings of previous studies involving marine citizen scientists (Cox et al. 2012; Delaney et al. 2008; Edgar and Stuart-Smith 2009; Hidalgo-Ruz and Thiel 2013; Rosenthal et al. 2018). In this case, we believe the result is likely due, in part, to the use of a simple, low-taxonomic-resolution field-monitoring protocol (i.e., estimating total algal percentage cover as opposed to functional group or species-level cover). Estimating percentage cover is a simple yet effective means of gathering data from intertidal environments, which is more accurate (i.e., closer to digital baseline estimates) and repeatable (i.e., less intra- and interobserver variation), and is less time-consuming to implement (thus allowing for greater replication) compared with methods such as point quadrats (Dethier et al. 1993; Meese and Tomich 1992). The low taxonomic resolution of the protocol reduced the scope for error through misidentifications (e.g., confusion between morphologically similar species) (Gardiner et al. 2012; Koss et al. 2009; Parsons, Lukyanenko, and Wiersma 2011), as well as the likelihood of overestimations by participants who may be eager to report rare species (Galloway, Tudor, and Vander Haegen 2006). Participant training is also known to increase the quality of the data generated (Edgar and Stuart-Smith 2009), and coupled with the provision of ongoing participant support, is also likely to have contributed to the generation of comparable data within this study, although comparisons with untrained participants are required to support this.

Although not significant, trained citizen scientists tended to underestimate algal percentage cover compared with both digital baseline estimates and other field observer units (i.e., professional scientists and combined units), yet in the laboratory, they overestimated algal cover in to-scale photos compared with professional scientists (Grist et al. 2019 unpublished data in Vye et al. 2020). We suggest that these differences, plus the small range of difference values observed for citizen scientists, may be due to the perceived importance of real-life field data that results in extra care being taken to get the data right, as well as the ability to physically manipulate specimens in the field. However, the experience level (e.g., previous monitoring experience) of citizen scientists was not considered, and thus further investigations are required to see if and how this may influence the findings. Similarly, an individual’s interpretation of instructions, or their experience on the training day, could have influenced the data they generated (Gollan et al. 2012), but it was not possible to assess this in our analysis.

On average, differences in estimates generated by professional scientists were lower than differences in estimates generated by citizen scientists. This may be explained by the higher experience level of professional scientists who often undertake regular field monitoring as part of their employment, and therefore the data they generate is often considered to be more accurate and consistent (Gollan et al. 2012), although, the range of differences in estimates was slightly greater for professional scientists owing to a small number of outliers. While the cause of these outliers remains unclear, they could be attributed to survey fatigue, a survey-specific factor (e.g., becoming time/tide limited or being distracted by participants they are supervising) (Swanson et al. 2016) or errors in image processing, although the latter is unlikely. These outliers demonstrate, like others (e.g., Cox et al. 2012), that professionally generated data may also contain discrepancies.

Pairing citizen scientists with professional scientists (i.e., combined units) was shown to reduce some of the observed disparity between citizen and professionally generated field data. While we acknowledge that such pairings may not be a feasible and/or cost-effective option for all citizen science projects, where possible, they could be suggested as a beneficial means of supporting citizen scientists with their transition from training to independent surveying, and have been suggested as a means of increasing the appeal of citizen science programmes (Crall et al. 2011). Such pairings may also be beneficial at certain time intervals to re-align citizen scientists and professional scientists, allowing citizen scientists to continue their development and granting professional scientists a dedicated opportunity to reflect on their own surveying skills and potentially enhance their teaching practices.

Irrespective of observer unit, estimates of algal percentage cover were most variable in plots with medium (26–50% and 51–75%) levels of algal cover. This indicates that some observers may find these plots more challenging to survey, potentially because of the patchy nature of algae within these plots. The different visualisation methods participants may have used to determine algal cover in the field may also have contributed to some of the observed variation in plots with medium levels of algal cover. Investigations into whether different field estimation techniques influenced the data generated were not possible here but would be beneficial for future efforts to determine the most appropriate technique to estimate percentage cover. Additionally, investigations into other potential drivers of the variability in plots with medium coverage are required, for example, the number of recent surveys a participant has undertaken, as those with more recent survey experience may “have got their eye in” and therefore generate more accurate estimates. In the meantime, future citizen science project facilitators should consider placing greater emphasis on training all participants (i.e., professionals and citizen scientists) to monitor plots with medium levels of coverage, and increasing replication in ecological studies to account for this variability.

Accurate estimates of percentage cover generated using simple, low-taxonomic-resolution protocols, although limited in terms of resolution, can be beneficial to scientists addressing certain ecological questions, for example, by identifying areas where changes are occurring (e.g., large scale losses/recovery of canopy algae) so that scientists may target these areas for further investigation. In addition, the use of these protocols provides important field experience for citizen scientists, allowing them to become confident employing a monitoring protocol in a realistic setting. The complexity of the protocol can then be enhanced depending on the confidence of observers, the quality of the data generated, and the requirement of the area, although data verification techniques would need to be adapted, and further training provided.

Strengths and limitations of digital and field approaches to citizen science

Owing to recent improvements in photographic technology, the analysis of digital imagery has become a common method of monitoring marine environments (Durden et al. 2016; Swanson et al. 2016) that can generate reliable estimates of abundance for low-resolution taxonomic groups (e.g., coral cover) (Carleton and Done 1995). Here, we have shown that digital analysis of photographs also represents a beneficial means of validating certain field monitoring data and understanding observer variation. All three methods of digital analysis generated comparable estimates of algal percentage cover that correlated well to field estimates despite slight differences in the approach (i.e., visual estimation versus point-intercept).

Digital analysis approaches are expected to increase in the coming years due to technological advances such as mobile applications and machine learning (Garcia-Soto et al. 2021). Such techniques can be considered a cost-effective means of generating monitoring data that constitutes a permanent visual record (Carleton and Done 1995). Furthermore, they can remove observer variability as field quadrats are often monitored once per time point, and usually by different observers over time and across sites, whereas digital analysis of a photograph can be done multiple times and by multiple people to minimise observer error and/or biases. Digital analysis can also increase the scope of citizen science projects because it allows for participation by individuals who may otherwise be unable (e.g., due to their location or physical ability). For example, online platforms (e.g., Zooniverse; www.zooniverse.org) have allowed citizen scientists to participate remotely in research by identifying, classifying, and marking factors of interest in images, videos, and audios (Simpson, Page and De Roure 2014), in turn reducing the time-consuming nature of digital analysis for professional scientists. Further research is required to understand the variation in data generated digitally by citizen scientists compared with professional scientists (but see Swanson et al. 2016 for an example), particularly in terms of percentage cover.

A major limitation of digital analysis, and therefore a strength of field monitoring, is the capacity to increase taxonomic resolution. Although this was not a problem during this study, it could become an issue should a more complex field protocol be employed (i.e., examining functional groups or a greater taxonomic resolution) because the three-dimensional nature of algal canopies would result in sub-canopy individuals being obscured from the photograph and thus inaccurately quantified, while smaller, cryptic, and finely branched species may go undetected. Additionally, digital baseline estimates could have been influenced, to a degree, by difficulty in determining the status (i.e., dead or alive) of encrusting species such as Lithophyllum sp. on some photographs. While this was minimised by using high resolution photographs, and would likely have influenced digitally generated estimates across all observer units and levels of algal cover equally, future efforts could consider excluding encrusting algae from estimates of algal abundance to minimise this issue. Furthermore, there are several, often unquantified, benefits that can be derived from field citizen science approaches, meaning they cannot simply be replaced by digital approaches. For example, being outdoors can improve fitness and allow for social interaction among participants that can in turn reduce stress levels and enhance mental wellbeing (O’Brien, Townsend, and Ebden 2010). Field approaches also allow participants to witness the natural world and the threats it faces firsthand, which may lead to increased environmental stewardship, positive behavioural change, and greater acceptance of management strategies among participants compared with those involved only in digital approaches (Cerrano, Milanese, and Ponti 2017; Evans, Gebbels, and Stockill 2008).

Conclusion

Confidence in the data generated by citizen science programmes is critically important if citizen science is to be fully embraced by the scientific community and used to inform management initiatives. While there is natural variation among observers, digital analyses and placing data generated by citizen scientists in the same context as professionally generated data are effective means of quantifying and examining this variation and ensuring data quality. The low taxonomic resolution estimates of algal percentage cover generated by citizen scientists as part of the Capturing our Coast programme can be considered accurate, with reported changes over time more likely to represent actual changes in marine ecological communities as opposed to differences among observer units. The validity of the data generated was likely due to the use of a simple, low-taxonomic-resolution monitoring protocol alongside effective training of participants and ongoing support and resources that collectively reduced the scope for error. Current and future citizen science projects, including those in the terrestrial realm, would benefit from adopting similar approaches to ensure and evaluate data quality, to strengthen training and protocols, and to promote the wider application of citizen science data.

Data Accessibility Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Supplementary Files

The supplementary files for this article can be found as follows:

Supplemental File 1: Appendix A

Coordinates and exposure of sites where surveys of total algal percentage cover were undertaken. Exposure was determined based on wave fetch values extracted from Burrows (2020), with sheltered sites having wave fetch values of < 2, moderately exposed sites between 2 and 3.5, and exposed > 3.5 (Burrows 2012). DOI: https://doi.org/10.5334/cstp.483.s1

Supplemental File 2: Appendix A

Locations of the twelve moderately exposed and exposed shores across the United Kingdom where surveys of total algal percentage cover were undertaken. DOI: https://doi.org/10.5334/cstp.483.s2

Supplemental File 3: Appendix A

Differences between field and digital baseline estimates of algal percentage cover per individual field observer unit. Circles show individual algal percentage cover estimates with colours representing the different surveyor types. Black squares and error bars represent the mean ± 1 standard error. N quadrats per observer unit = 8 except for unit 7 where N = 7. DOI: https://doi.org/10.5334/cstp.483.s3

Supplemental File 4: Appendix A

Post hoc Tukey-adjusted comparisons generated for single fixed effects to investigate differences between field and digital baseline estimates of total algal percentage cover across (a) field observer units, and (b) the level of algal coverage. Significant values (p < 0.05) are indicated in bold. DOI: https://doi.org/10.5334/cstp.483.s4

Acknowledgements

We would like to thank the amazing Capturing our Coast citizen scientists who helped to collect the data used in this study as part of the Race to Recovery experiment. We would also like to thank the two anonymous reviewers for their constructive comments which contributed to the improvement of this manuscript.

Funding Information

Capturing our Coast was funded by the Heritage Lottery Fund.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

HSE, SRV, and SRJ conceived the idea. HSE, SRV, KB, MTB, JC, SD, CF, HG, PL, SL, ZM, JP, AS, GW, VW, and SRJ undertook the fieldwork. HSE conducted the analysis. HSE and HS drafted the manuscript. All authors took part in discussions, contributed to subsequent drafts, and gave approval for submission. JD acquired funding for the Capturing our Coast project.

References

  1. Balázs, B, Mooney, P, Nováková, E, Bastin, L and Arsanjani, JJ. 2021. The Science of Citizen Science. In: Vohland, K, Land-Zandstra, A, Ceccaroni, L, Lemmens, R, Perello, J, Ponti, M, Samon, R, and Wagenknecht, K (eds.) The Science of Citizen Science. Cham: Springer. Pp. 139–197. DOI: https://doi.org/10.1145/3022198.3022652 

  2. Bates, D, Mächler, M, Bolker, B and Walker, S. 2015. Fitting linear mixed-effects models using lme4, Journal of Statistical Software, 67(1): 1–48. DOI: https://doi.org/10.18637/jss.v067.i01 

  3. Bonney, R, Cooper, CB, Dickinson, J, Kelling, S, Phillips, T, Rosenberg, KV and Shirk, J. 2009. Citizen science: A developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11): 977–984. DOI: https://doi.org/10.1525/bio.2009.59.11.9 

  4. Bonney, R, Shirk, JL, Phillips, TB, Wiggins, A, Ballard, HL, Miller-Rushing, AJ and Parrish, JK. 2014. Citizen science: Next steps for citizen science. Science, 343(6178): 1436–1437. DOI: https://doi.org/10.1126/science.1251554 

  5. Branchini, S, Pensa, F, Neri, P., Tonucci, BM, Mattielli, L, Collavo, A, Sillingardi, ME, Piccinetti, C, Zaccanti, F and Goffredo, S. 2015. Using a citizen science program to monitor coral reef biodiversity through space and time. Biodiversity and Conservation, 24(2): 319–336. DOI: https://doi.org/10.1007/s10531-014-0810-7 

  6. Burgess, HK, Debey, LB, Froehlich, HE, Schmidt, N, Theobald, EJ, Ettinger, AK, Hillerislambers, J, Tewksbury, J and Parrish, JK. 2017. The science of citizen science: Exploring barriers to use as a primary research tool. Biological Conservation, 208: 113–120. DOI: https://doi.org/10.1016/j.biocon.2016.05.014 

  7. Burrows, MT. 2012. Influences of wave fetch, tidal flow and ocean colour on subtidal rocky communities. Marine Ecology Progress Series, 445: 193-U462. DOI: https://doi.org/10.3354/meps09422 

  8. Burrows, MT, 2020. Wave fetch GIS layers for Europe at 100m scale, Figshare. Dataset, DOI: https://doi.org/10.6084/m9.figshare.8668127. 

  9. Carleton, JH and Done, TJ. 1995. Quantitative video sampling of coral reef benthos: large-scale application. Coral Reefs, 14: 35–46. DOI: https://doi.org/10.1007/BF00304070 

  10. Cerrano, C, Milanese, M and Ponti, M. 2017. Diving for science – science for diving: volunteer scuba divers support science and conservation in the Mediterranean Sea. Aquatic Conservation: Marine and Freshwater Ecosystems. DOI: https://doi.org/10.1002/aqc.2663 

  11. Clare, JDJ, Townsend, PA, Anhalt-Depies, C, Locke, C, Stenglein, JL, Frett, S, Martin, KJ, Singh, A, Van Deelen, TR and Zuckerberg, B. 2019. Making inference with messy (citizen science) data: when are data accurate enough and how can they be improved?. Ecological Applications, e01849. DOI: https://doi.org/10.1002/eap.1849 

  12. Cox, TE, Philippoff, J, Baumgartner, E and Smith, CM. 2012. Expert variability provides perspective on the strengths and weaknesses of citizen-driven intertidal monitoring program. Ecological Applications, 22(4): 1201–1212. DOI: https://doi.org/10.1890/11-1614.1 

  13. Crall, AW, Newman, GJ, Stohlgren, TJ, Holfelder, KA, Graham, J and Waller, DM. 2011. Assessing citizen science data quality: An invasive species case study. Conservation Letters, 4(6): 433–442. DOI: https://doi.org/10.1111/j.1755-263X.2011.00196.x 

  14. Delaney, DG, Sperling, CD, Adams, CS and Leung, B. 2008. Marine invasive species: Validation of citizen science and implications for national monitoring networks. Biological Invasions, 10(1): 117–128. DOI: https://doi.org/10.1007/s10530-007-9114-0 

  15. Dethier, MN, Graham, ES, Cohen, S and Tear, LM. 1993. Visual versus random-point percent cover estimations: ‘objective’ is not always better. Marine Ecology Progress Series, 96(1): 93–100. DOI: https://doi.org/10.3354/meps096093 

  16. Dickinson, JL, Zuckerberg, B and Bonter, DN. 2010. Citizen science as an ecological research tool: challenges and benefits. Annual Review of Ecology, Evolution, and Systematics, 41(1): 149–172. DOI: https://doi.org/10.1146/annurev-ecolsys-102209-144636 

  17. Durden, JM, Bett, BJ, Schoening, T, Morris, KJ, Nattkemper, TW and Ruhl, HA. 2016. Comparison of image annotation data generated by multiple investigators for benthic ecology. Marine Ecology Progress Series, 552: 61–70. DOI: https://doi.org/10.3354/meps11775 

  18. Earp, HS and Liconti, A. 2020. Science for the future: The use of citizen science in marine research and conservation, In: Jungblut, S, Leibich, V, and Bode-Dalby, M (eds.). YOUMARES 9 – The cceans: Our research, our future. Cham: Springer. Pp 1–19. DOI: https://doi.org/10.1007/978-3-030-20389-4_1 

  19. Edgar, GJ and Stuart-Smith, RD. 2009. Ecological effects of marine protected areas on rocky reef communities – A continental-scale analysis. Marine Ecology Progress Series, 388(4): 51–62. DOI: https://doi.org/10.3354/meps08149 

  20. Evans, SM, Gebbels, S and Stockill, JM. 2008. ‘Our shared responsibility’: Participation in ecological projects as a means of empowering communities to contribute to coastal management processes. Marine Pollution Bulletin, 57(1–5): 3–7. DOI: https://doi.org/10.1016/j.marpolbul.2008.04.014 

  21. Finn, PG, Udy, NS, Baltais, SJ, Price, K and Coles, L. 2010. Assessing the quality of seagrass data collected by community volunteers in Moreton Bay Marine Park, Australia. Environmental Conservation, 37(1): 83–89. DOI: https://doi.org/10.1017/S0376892910000251 

  22. Fore, LS, Paulsen, K and O’Laughlin, K. 2001. Assessing the performance of volunteers in monitoring streams. Freshwater Biology, 46(1): 109–123. DOI: https://doi.org/10.1046/j.1365-2427.2001.00640.x 

  23. Forrester, G, Baily, P, Conetta, D, Forrester, L, Kintzing, E and Jarecki, L. 2015. Comparing monitoring data collected by volunteers and professionals shows that citizen scientists can detect long-term change on coral reefs. Journal for Nature Conservation, 24(C): 1–9. DOI: https://doi.org/10.1016/j.jnc.2015.01.002 

  24. Fox, J and Weisberg, S, 2019. An R Companion to Applied Regression (Third Edit). Thousand Oaks, CA: Sage. 

  25. Freitag, A, Meyer, R and Whiteman, L. 2016. Strategies employed by citizen science programs to increase the credibility of their data. Citizen Science: Theory and Practice, 1(1). DOI: https://doi.org/10.5334/cstp.6 

  26. Galloway, AWE, Tudor, MT and Vander Haegen, WM. 2006. The reliability of citizen science: A case study of oregon white oak stand surveys. Wildlife Society Bulletin, 34(5): 1425–1429. DOI: https://doi.org/10.2193/0091-7648(2006)34[1425:TROCSA]2.0.CO;2 

  27. Garcia-Soto, C, Seys, JJC., Zielinski, O, Busch, JA, Luna, SI, Baez, JC, Domegan, C, Dubsky, K, Kotynska-Zielinska, I, Loubat, P, Malfatti, F, Mannaerts, G, McHugh, P, Monestiez, P, van der Meeren, GI and Gorsky, G. 2021. Marine citizen science: Current state in Europe and new technological developments. Frontiers in Marine Science, 8: 1–13. DOI: https://doi.org/10.3389/fmars.2021.621472 

  28. Gardiner, M.M, Allee, LL, Brown, PMJ, Losey, JE, Roy, HE and Smyth, RR. 2012. Lessons from lady beetles: Accuracy of monitoring data from US and UK citizen science programs. Frontiers in Ecology and the Environment, 10(9): 471–476. DOI: https://doi.org/10.1890/110185 

  29. Gardner, H. 1993. Multiple intelligences: The theory in practice. New York: Basic Books. 

  30. Gillett, DJ, Pondella, DJ, Freiwald, J, Schiff, KC, Caselle, JE, Shuman, C and Weisberg, SB. 2012. Comparing volunteer and professionally collected monitoring data from the rocky subtidal reefs of southern California, USA. Environmental Monitoring and Assessment, 184(5): 3239–3257. DOI: https://doi.org/10.1007/s10661-011-2185-5 

  31. Gollan, J, De Bruyn, LL, Reid, N and Wilkie, L. 2012. Can volunteers collect data that are comparable to professional scientists? a study of variables used in monitoring the outcomes of ecosystem rehabilitation. Environmental Management, 50(5): 969–978. DOI: https://doi.org/10.1007/s00267-012-9924-4 

  32. Halpern, BS, Walbridge, S, Selkoe, KA, Kappel, CV, Micheli, F, D’Agrosa, C, Bruno, JF, Casey, KS, Ebert, C, Fox, HE, Fujita, R, Heinemann, D, Lenihan, HS, Madin, EMP, Perry, MT, Selig, ER., Spalding, M, Steneck, R and Watson, R. 2008. A global map of human impact on marine ecosystems. Science, 319(5865): 948–952. DOI: https://doi.org/10.1126/science.1149345 

  33. Hidalgo-Ruz, V and Thiel, M. 2013. Distribution and abundance of small plastic debris on beaches in the SE Pacific (Chile): A study supported by a citizen science project. Marine Environmental Research, 87–88: 12–18. DOI: https://doi.org/10.1016/j.marenvres.2013.02.015 

  34. Hothon, T, Bretz, F and Westfall, P. 2008. Simultaneous inference in general parametric models. Biometrical Journal, 50(3): 346–363. DOI: https://doi.org/10.1002/bimj.200810425 

  35. Kohler, KE and Gill, SM. 2006. Coral Point Count with Excel extensions (CPCe): A Visual Basic program for the determination of coral and substrate coverage using random point count methodology. Computers and Geosciences, 32(9): 1259–1269. DOI: https://doi.org/10.1016/j.cageo.2005.11.009 

  36. Kosmala, M, Wiggins, A, Swanson, A and Simmons, B. 2016. Assessing data quality in citizen science. Frontiers in Ecology and the Environment, 14(10): 551–560. DOI: https://doi.org/10.1002/fee.1436 

  37. Koss, RS, Miller, K, Wescott, G, Bellgrove, A, Boxshall, A, McBurnie, J, Bunce, A, Gilmour, P and Ierodiaconou, D. 2009. An evaluation of Sea Search as a citizen science programme in marine protected areas. Pacific Conservation Biology, 15(2): 116–127. DOI: https://doi.org/10.1071/PC090116 

  38. Magurran, AE, Baillie, SR, Buckland, ST, Dick, JMP, Elston, DA, Scott, EM, Smith, RI, Somerfield, PJ and Watt, AD. 2010. Long-term datasets in biodiversity research and monitoring: Assessing change in ecological communities through time. Trends in Ecology and Evolution, 25(10): 574–582. DOI: https://doi.org/10.1016/j.tree.2010.06.016 

  39. Marshall, NJ, Kleine, DA and Dean, AJ. 2012. CoralWatch: Education, monitoring, and sustainability through citizen science. Frontiers in Ecology and the Environment, 10(6): 332–334. DOI: https://doi.org/10.1890/110266 

  40. Meese, RJ and Tomich, PA. 1992. Dots on the rocks: a comparison of percent cover estimation methods. Journal of Experimental Marine Biology and Ecology, 165(1): 59–73. DOI: https://doi.org/10.1016/0022-0981(92)90289-M 

  41. Mieszkowska, N, Sugden, H, Firth, LB and Hawkins, SJ. 2014. The role of sustained observations in tracking impacts of environmental change on marine biodiversity and ecosystems. Philosophical Transactions of the Royal Society A, 372(20130339). DOI: https://doi.org/10.1098/rsta.2013.0339 

  42. Mumby, PJ, Harborne, AR, Raines, PS and Ridley, JM. 1995. A critical assessment of data derived from coral cay conservation volunteers. Bulletin of Marine Science, 56(3): 737–751 

  43. O’Brien, L, Townsend, M and Ebden, M. 2010. ‘Doing something positive’: volunteers’ experiences of the well-being benefits derived from practical conservation activities in nature. Voluntas, 21(4): 525–545. DOI: https://doi.org/10.1007/s11266-010-9149-1 

  44. Parsons, J, Lukyanenko, R and Wiersma, Y. 2011. Easier citizen science is better. Nature, 471(7336): 37–37. DOI: https://doi.org/10.1038/471037a 

  45. Pecl, GT, Araújo, MB, Bell, JD, Blanchard, J, Bonebrake, TC, Chen, IC, Clark, TD, Colwell, RK, Danielsen, F, Evengård, B, Falconi, L, Ferrier, S, Frusher, S, Garcia, RA, Griffis, RB, Hobday, AJ, Janion-Scheepers, C, Jarzyna, MA, Jennings, S, Lenoir, J, Linnetved, HI, Martin, VY, McCormack, PC, McDonald, J, Mitchell, NJ, Mustonen, T, Pandolfi, JM, Pettorelli, N, Popova, E, Robinson, SA, Scheffers, BR, Shaw, JD, Sorte, CJB, Strugnell, JM, Sunday, JM, Tuanmu, MN, Vergés, A, Villanueva, C, Wernberg, T, Wapstra, E and Williams, SE. 2017. Biodiversity redistribution under climate change: Impacts on ecosystems and human well-being. Science, 355(6332). DOI: https://doi.org/10.1126/science.aai9214 

  46. Penrose, D and Call, SM. 1995. Volunteer monitoring of benthic macroinvertebrates: regulatory biologists’ perspectives. Journal of the North American Benthological Society, 14(1): 203–209. DOI: https://doi.org/10.2307/1467735 

  47. Poloczanska, ES, Brown, CJ, Sydeman, WJ, Kiessling, W, Schoeman, DS, Moore, PJ, Brander, K, Bruno, JF, Buckley, LB, Burrows, MT, Duarte, CM, Halpern, BS, Holding, J, Kappel, CV, O’Connor, MI, Pandolfi, JM, Parmesan, C, Schwing, F, Thompson, SA and Richardson, AJ. 2013. Global imprint of climate change on marine life. Nature Climate Change, 3(10): 919–925. DOI: https://doi.org/10.1038/nclimate1958 

  48. R Core Team. 2021. R: A language and environment for statistical computing. R Foundation for Statistical Computing. 

  49. Rambonnet, L, Vink, SC, Land-Zandstra, AM and Bosker, T. 2019. Making citizen science count: Best practices and challenges of citizen science projects on plastics in aquatic environments. Marine Pollution Bulletin, 145: 271–277. DOI: https://doi.org/10.1016/j.marpolbul.2019.05.056 

  50. Rosenthal, IS, Byrnes, JE, Cavanaugh, KC, Bell, TW, Harder, B, Haupt, AJ, Rassweiler, ATW, Perez-Matus, A, Assis, J, Swanson, A, Boyer, A, McMaster, A and Trouille, L. 2018. Floating forests: Quantitative validation of citizen science data generated from consensus classifications. ArXiv Preprint, 1801.08522. 

  51. Sandahl, A and Tøttrup, AP. 2020. Marine citizen science: Recent developments and future recommendations. Citizen Science: Theory and Practice, 5(1): 1–11. DOI: https://doi.org/10.5334/CSTP.270 

  52. Simpson, R, Page, KR and De Roure, D. 2014. Zooniverse: Observing the world’s largest citizen science platform. In: Proceedings of the 23rd International Conference on World Wide Web, 2014, pp. 1049–1054. DOI: https://doi.org/10.1145/2567948.2579215 

  53. Swanson, A, Kosmala, M, Lintott, C and Packer, C. 2016. A generalized approach for producing, quantifying, and validating citizen science data from wildlife images. Conservation Biology, 30(3): 520–531. DOI: https://doi.org/10.1111/cobi.12695 

  54. Thiel, M, Penna-Díaz, M, Luna-Jorquera, G, Salas, S, Sellanes, J and Stotz, W. 2014. Citizen scientists and marine research: volunteer participants, their contributions, and projection for the future. Oceanography and Marine Biology: An Annual Review, 52: 257–314. DOI: https://doi.org/10.1201/b17143-6 

  55. van der Velde, T, Milton, DA, Lawson, TJ, Wilcox, C, Lansdell, M, Davis, G, Perkins, G and Hardesty, BD. 2017. Comparison of marine debris data collected by researchers and citizen scientists: Is citizen science data worth the effort?. Biological Conservation, 208: 127–138. DOI: https://doi.org/10.1016/j.biocon.2016.05.025 

  56. Vye, SR, Dickens, S, Adams, L, Bohn, K, Chenery, J, Dobson, N, Dunn, RE, Earp, HS, Evans, M, Foster, C, Grist, H, Holt, B, Hull, S, Jenkins, SR, Lamont, P, Long, S, Mieszkowska, N, Millard, J, Morrall, Z, Pack, K, Parry-Wilson, H, Pocklington, J, Pottas, J, Richardson, L, Scott, A, Sugden, H, Watson, G, West, V, Winton, D, Delany, J and Burrows, MT. 2020. Patterns of abundance across geographical ranges as a predictor for responses to climate change: Evidence from UK rocky shores. Diversity and Distributions, 26(10): 1357–1365. DOI: https://doi.org/10.1111/ddi.13118 

  57. Warton, DI and Hui, FKC. 2011. The arcsine is asinine: The analysis of proportions in ecology. Ecology, 92(1): 3–10. DOI: https://doi.org/10.1890/10-0340.1 

  58. Wickham, H. 2016. Elegant graphics for gata analysis. New York: Springer-Verlag. 

  59. Zettler, ER, Takada, H, Monteleone, B, Mallos, N, Eriksen, M and Amaral-Zettler, LA. 2017. Incorporating citizen science to study plastics in the environment. Analytical Methods, 9(9): 1392–1403. DOI: https://doi.org/10.1039/C6AY02716D