Introduction
Coral reefs are one of the world’s most highly valued natural treasures. As the “rainforests of the sea,” they sustain about 25% of the ocean’s fish; provide food, income, and protection for over one billion people; contribute tens of billions of dollars in net economic value; and have high cultural value to people globally (). However, coral reef ecosystems are under severe threat from increasing storm intensity and marine heatwaves induced by climate change, with 99% of the world’s coral reefs potentially suffering from frequent marine heatwaves in the future (). Coral reefs also face increasing anthropogenic pressures such as overfishing, pollution, sedimentation, and poor water quality. Management strategies are not able to cope with the acceleration of disturbance regimes, resulting in unprecedented decline of coral reef communities across the globe () and a slow-down in the implementation of the United Nations (UN) Sustainable Development Goals (SDGs) framework within these regions ().
The Great Barrier Reef (GBR), situated in Australia, is the largest coral reef system in the world. Designated as a World Heritage Area in 1981, the GBR comprises more than 3,000 individual reefs, is 2,300 km (1429 miles) long, and encompasses almost 35 million hectares (13,3000 square miles). The GBR has suffered greatly from the impact of cumulative disturbances including back-to-back mass bleaching events in 2016 and 2017 that killed corals, the primary reef-building organisms, across hundreds of kilometers (). The broad-scale impacts of climate-driven disturbances increase uncertainty about how coral reef communities respond to climate change and our capability to predict the future status of coral reefs and manage them in light of new environmental regimes (; ).
The critical importance of coral reefs, and the international concern about their long-term viability, is reflected by two SDGs specifically dedicated to their protection. Here, SDG 13, Climate Action (Take urgent action to combat climate change and its impacts), relates to the provision of cost-effective and timely information to reef managers. SDG 14, Life Below Water (Conserve and sustainably use the oceans, seas and marine resources for sustainable development), aims to sustain healthy coral reefs in the future. The progression toward these SDGs is informed by a range of coral reef monitoring programs across the globe. For example, the GBR Long Term Monitoring Program (LTMP), led by the Australian Institute for Marine Science, is based on biannual underwater surveys of coral cover and other key indicators of reef health. Since 1985, the program has surveyed hundreds of reefs within the GBR, providing the most comprehensive record of coral status in a reef ecosystem (). Despite this massive effort, the size and complexity of the GBR, the high cost of surveys, and the accelerated decline in reef health mean that the LTMP alone cannot provide all the information required to address the SDGs (). Other reefs around the world are similarly challenged by the need for more, and more timely, information, including the sea-countries in the Pacific Ocean. Indeed, many reef ecosystems have no formal monitoring program to inform these SDGs (; ). Therefore, the use of CS to monitor coral reefs is relevant to fill the information gaps and to enhance the capacity to provide rapid and timely information for decision-making ().
Virtual Reef Diver (VRD) is the premier online engagement platform for the GBR that enables citizen-science information to be combined with existing monitoring efforts to provide more comprehensive and up-to-date information on reef health (www.virtualreef.org.au). Established in July 2018, VRD was inspired by the following question: Could we use photography from recreational divers, as well as international public interest in coral reefs, to help estimate the health of the entire GBR? This inspired a second question: What if the scuba-divers could geo-tag their photos to an interactive web-based visualization of the GBR, and what if other people across the world could access these photos and classify them with respect to key indicators of reef health such as presence/absence of coral to produce scientific information? The VRD program thus empowers citizen scientists across multiple communities to contribute to monitoring the GBR in three ways: 1) Underwater divers and on-the-reef communities can upload geo-referenced reef photos taken from cameras or underwater drones, 2) citizens can help classify benthic communities on uploaded photos, and 3) citizens can receive education about GBR health and management efforts.
VRD was featured during Australia’s National Science Week 2021, challenging the online citizen community to classify as many images as possible during seven days. A total of 55,215 points were classified on the images by 195 participants. These data were then incorporated into spatiotemporal models of coral cover, accounting for people’s abilities to classify corals on images as well as the associated difficulty in classification of the various images (; Santos Fernandez et al. 2021a; Santos Fernandez et al. 2023). This paper focuses on the delivery and outcomes of the National Science Week Challenge and addresses the role of CS in supporting the development of global policies for coral reef conservation. In addition to monitoring the achievement of the SDGs 13 and 14, the VRD program also contributes to evaluation of the effectiveness of management actions as required by the Reef 2050 Long-Term Sustainability Plan of the Australian Federal and Queensland Governments. The CS program also supports the Global Coral Reef Monitoring Network (GCRMN), an official partner of the UN, that compiles global datasets on reef health to respond to the SDG 14 requirements.
Through this case study, we describe the dual roles that citizen scientists play in VRD. In particular, we discuss ways in which we have tried to address three common concerns about CS programs highlighted by Fritz et al. () and Fraisl et al. (): whether the CS data are scientifically trustworthy, how CS data can be formally integrated with other more accepted data sources, and whether the CS data quantifiably improves monitoring and predictions to help address the SDGs. We also discuss the current challenges in developing and integrating CS programs able to fulfill SDG monitoring needs (; ) within the context of coral reef conservation.
Methods
Overview of Virtual Reef Diver
Dozens of publicly and privately funded programs have been established for monitoring the GBR. These are run by research institutions, government agencies, reef-based industries, citizen science groups, and traditional owners. Virtual Reef Diver is a web-based, interactive platform that has three main features. First, it enables images of the reef to be uploaded to a specified location (latitude/longitude) on a digital image of the GBR. Second, it enables these images to be accessed and classified by citizen scientists. Specifically, an image is chosen randomly from the set of available images, and the participant is asked to classify 15 randomly generated points in the image as hard coral, soft coral, algae, sand, water, other, or unsure (Figure 1a). These classifications are automatically extracted from the interface, coded, and stored in a database. Third, as detailed in the following section, VRD combines these citizen science data (Figure 1b) with data from other monitoring programs to derive spatiotemporal estimates of coral cover across the GBR (Figure 1c and 1d).
Main features of Virtual Reef Diver platform with (a) the classification module, (b) the map of the number of images contributed by the public, (c) the total number of images used for the modeling and (d) the predicted coral cover for the Great Barrier Reef.
Coral cover was chosen as the key measure in VRD because it is one of the most common indicators of reef health. Coral cover represents the proportion of the benthic zone covered in hard corals, without accounting for overlap in the three-dimensional coral structure. Hard corals build critical reef habitat.
VRD has received more than 255,000 image classifications, representing 3.8 million elicitation points. More than 1,300 images have been contributed by recreational divers. In addition to internal logging of user activities, VRD uses Google Analytics () to capture detailed technical, demographic, and content data on a per-user and per-request basis. Since July 2018, 45,480 users (defined as a person who has visited the website) have visited the VRD platform with some substantive increase in online traffic associated with a number of public events in which the program was featured (Figure 2). These show the potential of the online platform to gather a large number of people for a specific purpose. The most visited page is the classification module, with 61,547 views. Almost 75% of the users are based in Australia, followed by the USA at 13%, and Asia and Europe at 6%.
Number of users (defined as a person who has visited the website) that visited the Virtual Reef Diver platform since its release in July 2018, with highlighted main events (Features – ).
The VRD architecture ingests new classification contributions from citizens into a database. This information is then automatically combined with existing data to update spatiotemporal models of coral cover across the whole of the GBR. Changes in the health of the GBR can be monitored using the most up-to-date information about changes in coral cover. For the purpose of this study, images contributed by citizens are excluded from data analyses. Indeed, we aim here to describe the VRD methods to estimate bias associated with image classification by non-experts (i.e., CS data) and integration of these data with other data sources.
The National Science Week Challenge 2021
The National Science Week citizen science Challenge invited participants to help classify images of the Great Barrier Reef via the Virtual Reef Diver platform (Figure 3).
Official outcomes of the 2021 National Science Week released to the media.
The Challenge was promoted widely by the media, with multiple news articles, radio interviews, television broadcasts, and social media posts on Twitter and Facebook. This attracted a diverse range of participants geographically across Australia and multiple demographics, including primary and secondary school students, teachers, and other interested members of the community. Different types of log-in were proposed on VRD to separate individual participants from groups. Also, some rules are currently in place to automatically remove information provided by non-human participants (spam bots) during data extraction from the database.
After log-in, participants were presented with an image, and used the online classification module of VRD (https://www.virtualreef.org.au/classify/) to classify points on the image into the seven categories. Participants were not trained or tested to classify images. However, they were referred to VRD online resources on how to use the classification module, documents showing the different types of hard corals and other reef communities, and common misclassification errors (https://www.virtualreef.org.au/resources/volunteers/).
The Challenge occurred during the week of 14–22 August 2021, in which participants were free to classify as many images as possible at any time of the day and night. As part of VRD objective, classifications were then coded into presence or absence of hard corals with 1 when a classification was “hard coral” and 0 for the six other categories. The winner of the Challenge was the participant with the higher ability to classify hard corals. The following sections explain how VRD estimates the abilities of participants.
Increasing scientific trust in citizen science data
Classifications provided by citizen scientists differ with respect to accuracy and hence utility, primarily because of varying levels of expertise among the respondents (e.g., marine scientists are naturally more expert) and the difficulty of the images that the respondents were asked to classify (e.g., it is easier to classify an image comprising only sand than an image with a variety of coral and algae, and some cameras take better pictures than others).
To increase trust in these data, the ability of the participant can be estimated, adjusting for the image difficulty. This is achieved via a Bayesian item-response model (). This type of model is widely used in the applied fields of psychology, political, social sciences, education, and computer sciences. It is based on a logistic model that estimates the probability of answering a question correctly (). In the case of VRD, this probability is associated with the classification of hard coral on an image from a specific type of camera by a participant (). The item-response model adds additional parameters to logistic regression to estimate unobserved variables associated with the difficulties of the task. Here, these difficulties are associated with an image as reef images are different in terms of resolution and reef biodiversity. The estimation of these parameters allows us to correct participant’s answers for these biases ().
Integrating citizen science data with other data sources
VRD employs a weighted likelihood method for combining data from disparate sources in order to obtain combined estimates of coral cover on the GBR. This approach, described by Peterson et al. (), involves two steps. In the first step, a set of mechanistic weights is determined for each input source, based on a pre-specified measure of their respective accuracy obtained using a gold standard dataset. In the second step, these weights are included in a spatiotemporal Bayesian model. These steps are described in more detail below.
In the mechanistic weighting approach, the weight wijs associated with the classification by participant, image, and data source is calculated as the product of four separate weights characterizing the physical extent of the image, the number of points used to elicit the coral cover in the image, the overall accuracy of the person classifying the image, and a factor used to up- or down-weight an image depending on the source. The weights were then used to calculate a spatially weighted mean estimate of coral cover at raster cell level for each source and year.
The model also considered the impacts of environmental disturbances, reef locations across the GBR defined as reef shelf position (), and reef management status (). Additional model parameters were introduced to account for the dependency of observations across space and time in the form of random effects (). These spatial random effects were key to predicting values in coral cover at unobserved locations (). Environmental disturbances were characterized by the presence/absence of cyclones (; ), anomalous sea surface temperature, and bleaching ().
Peterson et al. () specifically investigated the impact of including different data sources in their models, including data collected through the VRD program and professional monitoring data. In particular, they showed that data integration improved the ability of the model to predict coral cover by 43%, and also improved the precision of the predictions in areas both with and without professional monitoring data.
In a companion paper, Santos-Fernandez and Mengersen () proposed an alternative to the mechanistic weighting approach described above. The new method avoids fixed, pre-specified weights of the data sources and instead adjusts for the quality of these data sources as part of the modelling process using the participants performance measures: sensitivity (se, correct classification of a point on an image as coral) and specificity (sp, correct classification of a point as not coral). This approach provides more accurate estimation of each participant’s sensitivity and specificity, accuracy, and precision of the resultant predictions, and reduces bias in the regression coefficients of the model. The sensitivity and specificity parameters are informative in their own right, and can also be used to derive the average sensitivity and specificity, and the corresponding variance of these averages, for the citizen scientist cohort. This information can be used to cluster citizens into different groups based on their performance (e.g., beginners, advanced, experts). It also informs weighted consensus methods by deriving a weight for each participant proportional to their performance (). This is how VRD ensures data quality of citizen science data prior to the integration with other data sources.
Results
The National Science Week Challenge 2021
Overall, 195 participants registered for the National Science Week Challenge, either individually (n = 49) or as a team (n = 146). A total of 3,681 images and 55,215 points in the images were classified during the week of 14–22 August 2021 (Figure 3). More than 20 schools, 7 universities, and 7 community groups participated. Nine teams enrolled; eight of these registered between 2 and 11 members, and one team registered 102 participants and classified 1125 images. A composite team comprised 11 participants drawn from regional and remote areas of the State of New South Wales and classified 92 images.
During this week, a maximum of 220 users was recorded on Thursday, 21 August 2021 (Figure 4). Most of the website users came from Australia followed by USA and Europe. The most-visited page was the classification page, which showed that people were interested in the classification module on which the event was supported.
Overview of website activities during the National Science Week 14-08-21 to 22-08-21 ().
Using the Bayesian item-response proposed by Santos-Fernandez and Mergensen, (), the participants of the Challenge were clustered into four groups based on their abilities and image difficulties (Figure 5).
Ability scores of the Virtual Reef Diver participants to classify corals from images. The size of the dots corresponds to the number of classifications made by a participant and colors to the groups allocated based on their abilities.
A total of 48 participants were awarded the category of “experienced,” and 49 participants were awarded the category of “super citizen scientists (experts).” A similar number of participants were classified as beginners (49) and competent (49).
The winner was a high school student who classified 18,195 points, and she belongs to the group of super citizen scientists. The first prize was a trip to the GBR. The Reef Today film showcased the VRD’s Challenge, and the winner can be seen at: https://www.youtube.com/watch?v=8F019Rl1Gds.
Increasing scientific trust in citizen science data
We assessed the reliability of CS data and the effectiveness of the statistical model to correct for bias in CS datasets by focusing on a subset of 514 underwater images for which the true labels were considered known based on classifications by marine science experts (; ). This subset of images is a balanced representation of the benthic composition and image resolution (i.e., different types of cameras) in the full set of VRD images provided by professional monitoring programs (see for additional information on sampling design). These images (among others) were randomly presented to the CS participants as part of the Challenge.
Figure 6, extracted from , shows empirical densities of known proportion of coral cover based on the subset of images (in red), as well as the reported proportion of coral cover from participants’ answers (a total of 157 non-experts, in green), and the estimated proportion of coral cover after accounting for the abilities of the participants and image difficulty (in blue). It is obvious that the adjustments make a large difference to the accuracy of the CS data, thereby increasing trust in this data source.
Density plot of the true coral cover (red) in the images, the reported values from the participants (green) and estimated proportion from the model after accounting for the participants abilities (blue). Figure reproduced with permission from Santos-Fernandez et al. ().
Another insight that we can derive from the model is the estimation of sensitivity (se) and specificity (sp) of the participants. Table 1 shows a sample of results from 3 representative participants of the Challenge and their performances. The columns mean and sd represent the posterior distribution mean and standard deviation of the parameters, while the 2.5–97.5% gives the uncertainty around these estimates. For example, participant 2 has high performance (with both se and sp above 90%) compared with subject 3. Hence, information produced by this subject 2 receives more weight in the item response model than that from 3.
Table 1
Summary statistics of the posterior performance measures from a sample of 3 participants. Se and sp refer to the sensitivity and specificity measures of participant performances, sd stands for standard deviation, and 2.5%–97.5% correspond to the measurement of uncertainty estimated by the model.
| PARTICIPANT | VARIABLE | MEAN | SD | 2.5% | 97.5% |
|---|---|---|---|---|---|
| 1 | se | 0.777 | 0.140 | 0.519 | 0.991 |
| sp | 0.827 | 0.128 | 0.542 | 0.995 | |
| 2 | se | 0.935 | 0.117 | 0.572 | 1.000 |
| sp | 0.984 | 0.036 | 0.878 | 1.000 | |
| 3 | se | 0.512 | 0.010 | 0.501 | 0.537 |
| sp | 0.712 | 0.112 | 0.515 | 0.921 | |
Integrating citizen science data with other data sources
As shown above, the CS data obtained through the National Science Week Challenge, and indeed through all of the VRD classifications, can be used to improve predictions of coral cover across the GBR. This relative benefit is demonstrated clearly in Figure 7, reproduced from the paper by Peterson et al. (), which shows the cross-validated predictions after including all existing data with the survey data from the gold-standard LTMP survey data, compared with using the LTMP data alone.
Model performance when integrating all the existing data versus using the long-term monitoring data only (reproduced with permission from ).
The VRD statistical model can also be used to identify important features relevant to the SDGs. For example, the model was used to estimate the influences of various disturbances, management zoning, and reef locations on coral cover changes between 2002 and 2020. Coral bleaching was the unique variable that contributed to the decline of coral cover at the scale of the GBR (Figure 8). The other variables are considered as non-substantive because the 95% credible interval includes zero.
Effects of model covariates with (a) all data combined and (b) professional data only. The dots show the estimated mean and error-bars associated 95% credible intervals.
The final outputs of VRD are predictive maps showing changes in coral cover across the entire GBR (https://www.virtualreef.org.au/explore/). This represents one of the most spatially and temporally extensive datasets available for research and management. Indeed, predicted values of coral cover are accessible at 500 m2 across the 3,000 reefs of the GBR from 2002 to 2020. The maps provide up-to-date knowledge to understand the impacts of disturbances on the reef health and inform decision-making about large-scale changes and effectiveness of management interventions.
A total of 18 years of predictive values of coral cover along the entire GBR is freely accessible online. Uncertainty layers associated with these predictions are also available online. Predictions can be downloaded from the platform, and instructions to perform various manipulations with the data are also available at (https://www.virtualreef.org.au/resources/).
Discussion
This paper has focused on an exemplar event, the National Science Week in Australia in 2021, to highlight a variety of ways in which citizen science can help to meet the SDGs associated with conservation of coral reefs. The event employed the premier citizen science platform, Virtual Reef Diver, to facilitate wide public engagement in classifying images of the Great Barrier Reef. Overall, more than 55,000 new classifications were made, which contributed directly to improved spatial predictions of coral cover on the GBR. The high level of participation and interest in the Challenge demonstrated a broad interest by people of all ages, in particular young people, in contributing to citizen science programs in general, and to environmental conservation programs in particular.
One of the participants, an adult from an outback town in the desert area of New South Wales, responded after the event with the comment that although they had never seen the ocean, they now felt a strong connection with the GBR. This epitomized for us the power of citizen science and programs such as VRD to protect the largest World Heritage Area on Earth.
This paper has also focused attention on ways in which persistent concerns about citizen science data have been addressed in the VRD program. The first of these concerns is scientific trust in citizen science data (). The solution adopted in VRD is to employ a form of item-response modelling to statistically quantify the ability of each participant, taking into account the difficulty of the images that they classified, and grouping them by ability levels. Papers from Santos-Fernandez and Mengersen () and Santos-Fernandez et al. () provide detailed explanations of the method and its application for complicated image classification tasks. Importantly, this approach can be translated to other CS projects for which there is concern about the quality of CS data. An example is the large class of more general species distribution studies in which species of interest may be missed or mistaken by untrained participants (indeed, even experts), thereby creating bias in estimated probabilities of presence unless such adjustments are made ().
Citizen science data is often collected opportunistically without a sampling design or a professional survey, which can lead to geographical, spatial recording, and preferential sampling biases (; ; ). To mitigate these biases, several strategies have been proposed, including the use of occupancy models and accounting for observer effort and human population density. Another significant source of bias is imperfect detection or quantification, which is particularly relevant for citizen science data. Researchers have developed methods to minimize the misclassification rate, such as Bayesian occupancy models that account for detection bias (; ; ).
Moreover, the proposed approach can be extended to include other important factors that may influence the ability of the participant, and can be oriented to more targeted aims such as preferring high sensitivity at the expense of lower specificity or selecting a group of citizen scientists with highest (or lowest) ability. Fraisl et al. () describes different approaches adopted by popular CS programs to curate data provided by citizen scientists and ensure data quality. For complicated tasks, testing can be performed before and during the project using gamification and/or artificial intelligence methods (; ; ). To date, no CS program uses the item-response model to estimate participants’ abilities while accounting for image difficulties and other variables that can influence an answer (). Along with the model development, we have learnt about the capability of non-experts to classify hard corals, marine animals that they will likely never see in real life, from images. An unexpected result shows that participants get better at classifying hard corals after only a few images, showing that people naturally learn even with complicated tasks (). These findings can complement current research using established citizen science online platforms, including iNaturalist, eButterfly, and Zooniverse, that host hundreds of projects involving image classification.
The second concern is how to aggregate the CS data with other data sources. As described above, two solutions have been employed in the VRD project. Under the first approach, a mechanistic weight is determined for each data source prior to the data integration, and these weights are used to manually adjust the relative influence of the data sources in a Bayesian spatiotemporal regression model (). A subsequent approach that we have developed overcomes the concern of pre-defined mechanistic weights (see discussion in ) by estimating the weights as part of the analysis (). Through this process, estimates of sensitivity and specificity can be derived for each individual participant, providing much more nuanced information about their relative ability and facilitating control for data quality. While the VRD platform currently includes the second of these approaches, the third solution combining item-response model to weight participants’ answers based on their abilities to classify hard corals and image difficulty with the spatiotemporal model will be deployed in the next update of the VRD system.
The third concern is whether CS data do, indeed, improve estimates and predictions relevant to SDGs and associated indicators (; ). In this paper, we demonstrate that the adjusted data contributed by CS participants through the VRD platform meaningfully improve predictions of coral cover on the GBR. These predictions can be employed to create indices to measure health and resilience of coral reef systems, as well as assess the effectiveness of reef restoration interventions, in line with the goals of major conservation policy and management strategies such as Reef 2050 and GCRMN. The outcomes of these major strategies in turn directly inform SDGs 13 (Climate Action) and 14 (Life Below Water) and indicators regarding the number of countries using ecosystem-based approaches to manage marine environments (indicator 14.2.1), informing the establishment of marine protected areas (indicator 14.5.1), supporting the development of adaptive management strategies for sea countries (indicator 13.b.1), and providing education for sustainable development (indicator 13.3.1).
VRD is still evolving to better meet the needs of both its participant community and its user community. Future directions for the project include the following. The first is a focus on adding functionality into the platform, with improved feedback to close the loop for the CS participants. The second is a focus on data integration, using both citizen science and artificial intelligence for image classification. This is the intention behind ReefCloud, a large international collaboration on coral reef monitoring, led by the Australian Institute for Marine Science (AIMS). The third is to explore broader directions with respect to other ecosystems using the concepts of image classification, citizen science, data integration, statistical modelling, and informed decision-making. The combination of these concepts covers a broad aspect of the current research and brainstorming about the utility of CS data into existing management frameworks, and additional work is needed to fully capture strengths and limitations of our proposed CS program before the diversification of its application.
Importantly, the methods developed through VRD could be employed to facilitate more, and more trustworthy, use of CS data to succeed in the implementation of SDG 14, sustaining the use of the oceans and marine resources. By protecting the Great Barrier Reef, VRD indirectly contributes to achieve other SDGs that support the “blue economy” (). These SDGs include: 1 (No Poverty), 2 (Zero Hunger), 3 (Good Health and Well-being), 4 (Quality Education), 5 (Gender Equality) 8 (Decent Work and Economic Growth) and 10 (Reducing Inequality), and showcase the importance of protecting coral reefs for our socioeconomic systems.
The path toward an active contribution of CS data to SDG monitoring is not straightforward (; ; ). The SDGs provide a globally generic, rather than locally specific, framework. The heterogeneity of CS programs in supporting local communities and addressing specific environmental issues makes the integration of these programs into the generic SDG framework difficult (; ). Another barrier is the deployment of reef citizen science programs in sea countries to fill the data gap (). To date, most citizen science programs are implemented in developed countries () in conjunction with professional coral reef monitoring. Ballerini and Bergh () argued that without UN consideration for diverse socioeconomic systems, traditional knowledge, and values of non-Western countries (such as sea countries), the adoption of the CS-SDGs approach may be deterred. The use of VRD to overcome these challenges is beyond the scope of this study; however, ensuring the production of trustworthy, locally relevant data is a first step towards increasing the adoption of a citizen science approach for coral reef monitoring and reinforcing the link between CS and the SDGs.
Data Accessibility Statement
Data are available upon request to the first author.