The promise of citizen science (CS) holds that it can both educate its participants and provide useful scientific data (Bonney et al. 2009; Raddick et al. 2009; Bonney et al. 2016). Educators intuitively know this is a lofty goal for formal education. For centuries, societies have supported educational institutions where students must undertake many years of schooling to prepare themselves for careers in all kinds of disciplines, including science. Beyond the historical perspective, our own experiences as teachers affirms this challenge. Most often, science educators instruct students using well-thought-out exercises instead of doing authentic science (but see studies of class-based or course-based undergraduate research experiences such as Spell et al. 2017 and Flaherty et al. 2017). These carefully planned demonstrations and laboratory experiments are tested and revised so they “work.” In contrast, the process of doing science is not straight forward, involving missteps and failed experiments (Understanding Science 2021) that necessarily slow down learning science content and that can be challenging to undertake in classrooms (Spell et al. 2017, Harlin et al. 2018).
The fundamental dichotomy between learning and doing science makes it unsurprising that people question the validity of the data coming from CS programs (Cohn 2008; Dickinson, Zuckerberg, and Bonter 2010; Gura 2013). Indeed, CS programs are often challenged about the quality of the data in their projects (Aceves-Bueno et al. 2017).
There is no single approach CS programs use to engage citizens/students and ensure the quality of their data (Wiggins et al. 2011; Kosmala et al. 2016; Freitag, Meyer, and Whiteman 2016; Stevenson et al. 2021). CS projects that record biodiversity have a variety of approaches to ensuring data quality. The largest CS biodiversity program, eBird, is available to any user who signs up. It uses a multipronged system for data quality assurance. Elements of the eBird system include minimizing errors with data inputs, checking inputs in real time to prompt users about unusual observations, review by regional experts, and calibration of the sensing ability of individual observers (Yu et al. 2012; Kelling et al. 2012, 2015). The Coastal Observation and Seabird Survey Team (COASST) is geared toward local community members (Parrish et al. 2019). It has a strict selection process to recruit participants and a rigorous training program with extensive protocols to monitor dead seabirds and marine debris (Parrish et al. 2017). Seabirds carcasses are collected and shipped to experts for ID confirmation. Reef Environmental Education Foundation (REEF) engages the diving community to monitor marine habitats, especially those associated with fish in tropical waters around the globe. Recreational divers survey habitats using a rigorous monitoring protocol. Observations are reviewed by expert marine scientists. The program rates divers based on experience. Participants that achieve “expert” status can be invited to join in special regional monitoring and assessment diving expeditions (Schmitt, Wells, and Sullivan-Sealey 1998; C. Semmes, personal communication).
Overall, it is fair to say that strategies are tailored to the focus of the program and to the difficulty of the participants’ tasks. Program design may include open enrollment or careful selection of participants, minimum or extensive time commitments, and little or extensive training, etc. The most evolved process to ensure data quality to date in the United States (USA) has been developed over decades by the water quality monitoring community. These CS projects require extensive planning documents called Quality Assurance Project Plans (QAPP) that are reviewed by state governments and the Environmental Protection Agency (EPA). Parrish et al. (2018) have provided a hierarchy of design criteria for collecting high-quality data in CS projects.
The elemental components of data needed for biodiversity studies are species observed, place of observation, and the date and time of observation. Recording place and time in a standard fashion can be difficult (Chapman 2005) but the adoption of smartphone apps for recording has reduced those errors. Identification of species is more difficult and can take specialists years to master and may require specialized equipment, if not DNA sequences, depending on the taxon. Image identification using neural networks algorithms has become practical in the past few years and is helping the ID process in apps such as Merlin, iNaturalist, and Pl@nt.Net. The success of eBird and REEF depend on the community of naturalists who have become experts at identifying birds and fish in the wild. Other CS biodiversity projects, such as iNaturalist, iSpot, and Pl@nt.Net assist with identification using a two step process. First, an observer takes a picture or makes a sound recording of the species in the field, so there is a digital record of the observation. The observations are then shared on the web, allowing species identification to be crowdsourced by experts around the globe in the second step. This CS design intentionally separates the collection of the data and the identification of the species so people collecting the data need not necessarily be able to identify a species. iNaturalist has been adopted by state agencies, the National Park Service, and the National Geographic Society for bioblitzes, and more recently, it has served as the backbone for the City Nature Challenge with bioblitzes now organized in over 200 cities around the globe.
This paper is motivated by the questions of data quality generally and specifically data quality from first-time and novice users engaged in exploring nature as part of an educational retreat. The motive and context for this retreat are documented in a companion paper (Rokop et al. submitted). We undertook analysis of the data collected by students using iNaturalist to answer the question to what degree might students entering college with little or no experience observing nature make useful contributions to scientific efforts to characterize biodiversity in our own backyards and across the globe? We broke down this focal question into three components as follows:
There was no specific biodiversity question the students were asked to investigate beyond documenting biodiversity. On the basis of our analysis of the data, we have included a series of recommendations for program design to increase the value for biodiversity science.
Each autumn from 2017 to 2019, two or three groups of 80 to 100 entering Honors students at the University of Massachusetts Boston were invited to attend a 2-day retreat on Thompson Island (https://thompsonisland.org/), part of Boston Harbor Islands National and State Park (https://www.nps.gov/boha/index.htm, and for history about the island, see bostonharborislands.org) One activity during this retreat was a three-hour bioblitz using the iNaturalist platform. The educational goal of this exercise was to allow the students to observe nature and to participate in a CS project (Rokop et al. 2021, submitted).
Thompson Island is located about 1 km from UMass Boston and about 6 to 7 km from the center of Boston. It is managed by the Thompson Island Outward Bound Education Center, a nonprofit educational organization. The island is a drumlin, 69 hectares in area (170 acres) at high tide, roughly 2 km long and 400 meters across at its widest point, and 24 meters above sea level at its highest elevation. Thompson Island contains a diversity of habitats including open meadows, forests, salt marshes, and beaches.
Students were asked to download the iNaturalist app to their cell phones before coming to the island. On island, they were provided a brief (~30 minute) introduction to the project and use of the app by National Park Service Rangers. Subsequently, teams (including guides) spent 1.5 to 2 hours making observations. Roughly equal numbers of students were sent to different quadrants of the island and instructed to make as many observations of organisms as they could record with iNaturalist. An exception to this general set of instructions occurred in the first year when four teams were sent to each quadrant and asked to concentrate on searching for members of one of four groups of organisms (plants, insects, marine organisms, or fungi). After the first-year experience, instructors thought this was too restrictive for the students and abandoned the requirement in subsequent years.
iNaturalist evaluates observations with a three-category system of Casual, Needs ID, and Research Grade (See here for the official rating details). In addition to the iNaturalist evaluations, we tallied the number of photographs per observation and developed a rubric to score the quality of images as good, OK, or poor (See below for further explanation). We identified whether or not the observer tried to identify the species being observed, and we also evaluated the likelihood that an expert in a specific taxonomic group could ID the species using the images taken (and any recorded notes). We totaled the number of observations that were identified to the species and genus level by August 1st, 2020. Lastly, we evaluated the accuracy of GPS data, the assigned place names of the observations, and the time stamps of the observations.
Image quality of an observation was determined for the group of photographs representing a specific specimen taken in aggregate (a recorded observation) using criteria of Table 1. It was important that a single species was clearly identifiable, properly lighted, and focused. If one or more photos included adjacent organisms, then at least one photograph must focus specifically on the selected species, and if there is only one photograph focusing specifically on the selected species, it must focus on aspects generally useful for taxonomic identification (examples are found in Figure 1 and Supplemental File 1: Supplemental Figure 1). Identification from photographs can be challenging but can be made more reliable by appropriate views of in-focus photographs. Capturing an image of the entire organism from a distance provides context for more detailed photographs. Observations were rated good for animals if multiple views were captured (lateral, dorsal, ventral), including images of appendages as appropriate. Likewise, a photograph or a group of photographs including an entire plant (above-ground aspects), branching pattern, leafing pattern, bark, and flowers or fruits (if present), was rated as good. For some species, this could be achieved in a single photograph and thus the image quality might be rated good if the observation included a single, in-focus photograph in which the intended species existed alone or was irrefutably the intended species (Figure 1). Such irrefutable intention was at times, although not often, made possible by explanatory notes recorded with the observation. In some instances, a single photograph or a collection of photographs was sufficient for identification of a commonly recognized species, but the observation was still rated poor or OK because it alone, or the aggregate photographs included, did not satisfy the above criteria. Thus, quality rating of an observation and its usefulness for identification were not necessarily tightly coupled. An OK rating indicated that the observation partially satisfied the criteria, but could easily be improved by better or additional photographs. Inclusion in an image of an object of known size (approximate or precise) generally improved ratings of an observation.
|Image quality||Good||Sharp focus, shows different angles of whole organism and parts|
|Ok||In focus, shows the (intended) organism|
|Poor||Multiple organisms in image, subject uncertain, images out of focus, scale insufficient to see organism clearly|
|ID possible||Yes||Well-known taxon, image and comments provide definite information|
|By expert||Individuals with sufficient taxonomic expertise for this group or for local species are likely to be able to ID|
|No||Critical characters for the taxon not visible or taxon cannot be identified with macro photograph alone|
During the course of the rubric development and scoring, we discovered that some observations made by students during the orientation program were not included in the project by the iNaturalist algorithm, a possibility we had not originally considered. As a consequence, we gathered and analyzed these additional observations located on the basis of user name, time frame, and GPS location. After completing these extra steps, we were confident that we had included all of the observations made by all users who were part of the orientation program. For analyses we relied on iNaturalist tools, Excel, Google docs, R (R Core Team, 2020), RStudio (RStudio Team 2019), and the rinat and iNatTools libraries.
Over three years (2017–2019) of the orientation program, a total of 468 students collected more than 5,600 images that were distributed across more than 2,000 observations, > 600 of which achieved Research Grade on iNaturalist (Tables 2 and 3). One new local species record, for Dekay’s Brownsnake, was obtained.
|YEAR||OBSERVATIONS||OBSERVERS||OBSERVATION IDENTIFIERS||SPECIES||START DATE||END DATE|
|YEAR||STUDENTS||STUDENT USER ACCOUNTS*||STUDENT OBSERVATIONS||SPECIES ID’ED|
Roughly 700 observations were collected each year using approximately 85 first-time user accounts (Table 2) that documented about 100 species (Table 3). The numbers given in Table 2 include Park Service employees as well as students. Readers should be aware that we found that all the numbers in Table 2 changed over time because of data curation by project managers, individuals withdrawing their observations, and more identifications being made over time.
There were almost twice as many student participants the first year compared with the 2nd and 3rd years, but the number of user accounts varied little (Table 3). The number of student observations and species identified increased slightly in each subsequent year (Table 3). The groups in the second and third years were comparable in their productivity, with each making about 9 observations per account and identifying 1.5 species (Table 3).
In the first two years, most observations were of plants (about 50%) and animals (40%), with the remaining 10% divided between fungi and algae (Figure 2). For 2019, the number of observations of fungi and algae remained the same as in prior years, the number of plant and animal observations increased, and the not available (NA) category dropped substantially relative to the first two years (Figure 2). NA might indicate that identity of the observation was difficult or not possible, images were of humans or of objects not organisms, several organisms were included in the image with insufficient information to determine intention, etc.
A total of 202 unique species were identified across the three years from the student orientation programs with 19% of the species common to all 3 years (Figure 3). Taxonomic diversity was high and included more than 90 families and about 100 species each year (based on analysis of iNaturalist data). Approximately 34% of the observations were identified to species level and thus achieved Research Grade. There was much more success with identification of animals and plants than of fungi and chromista (algae).
Among animals, the most commonly identified species were marine mollusks, crustaceans, and insects (Figure 4). These included hermit crabs, green shore crabs, blue mussel, common periwinkles, acorn barnacles, and European flat oysters in the marine habitats; and butterflies, bumblebees, grasshoppers, and spiders in the terrestrial habitats. As with animals, the students recorded or observed the most obvious plants, such as colorful herbaceous flowers or shrubs that were flowering or fruiting in the late summer.
Of the more than 200 species observed by students on the island, four struck us as immediately noteworthy. First, a student caught and photographed the Dekay’s Brownsnake (Storeria dekay). A search of iNaturalist found this to be the first and only observation of this species on the island (R. Vincent, personal communication). Second, there is a thriving population of English Oak (Quercus robur) that have naturalized the island. Quercus robur occurs across the USA and Canada and throughout New England (GoBotany 2021, iNaturalist 2021), but the largest and densest number of individuals in Massachusetts appears to be on Thompson Island (iNaturalist 2021). Lastly, students documented a large number of two invasive crab species, the European green crab (Carcinus maenas) and the Asian shore crab (Hemigrapsus sanguineus). The green crab has been on our Atlantic coast for 200 hundred years, whereas the Asian shore crab arrived in New Jersey in the 1980s and is now extending its range and population density along our coast (Bailie and Grabowski 2018). These data are likely to contribute to our understanding of how these species are interacting over time and space.
Students were encouraged to photograph more than one view/character of the species they were observing, and indeed they did so. Sixty percent of the observations have 2 or 3 images, while the amount with just one image was between 10% and 20%.
Aspects of data quality useful for species identification improved over the three year period and especially between the first and the second year. Observers increased the number of observations they tried to identify from 2017 to 2019 (Figure 5a). The “Research” and “Needs ID” observation grades increased, and the NA grade decreased during the course of the study (Figure 5b), with about ⅓ of the observations achieving Research Grade.
We further asked, “Is it possible to identify this observation to species with the images provided?” In 55% to 70% of the cases we thought it was possible, meaning we think that with more attention by expert identifiers on the ID process, more observations can attain Research Grade (Table 1; Figure 6a). We found that the quality of the collection of images for each observation (Table 1) was mainly OK or good in 85% of the observations; 15 % were poor (Figure 6b).
Location data were accurate as judged by two criteria. When plotted, observations follow expectations for terrestrial species (Figure 7, top panel) found across the island, mostly along walking paths, while marine species were observed along the shoreline (Figure 7, bottom panel). A few of the marine observations appear in deeper water indicating larger location errors. Those at the lowest latitudes (< 42.31°N) may represent observations made at low tide. The mode and median positional accuracy were good at 5 and 6 meters respectively (Figure 8).
Three issues compromised the quality of the location data. First, as noted above, when reviewing project data by user ID, we discovered that some observation records were not recorded as being on Thompson Island when they should have been, owing to low GPS accuracy. These observations were recovered by searching for observation records within the appropriate time window from participating contributors. In 2017, 4.7% (n = 29) of the observations collected were not included. The corresponding values for 2018 and 2019 were 4.3% (n = 27) and 0.6% (n = 5). Second, occasionally there was large uncertainty about an observation, leading to a long tail to the distribution and a mean of 41 m (Figure 8). Third, there were more than 60 different place names given in the data set, making us initially question the location of an observation even though the vast major of GPS coordinates accurately located observations on Thompson Island.
We found no discernable errors in the date-time data. It was our confidence in the date-time date that allowed us to discover the observations that did not make it into the projects described in the previous paragraph.
The identification process of iNaturalist is key to its success. The process allows anyone willing to review images the opportunity to help identify organisms. In all, there were 6,665 identifications made by 713 different identifiers over three years. The number of unique reviewers by year was 269, 319, and 312. The identification process revealed that most identifiers helped refine identifications.
Our naive users collected more than 5,600 images, which resulted in the identification of 202 species and potentially more. A year-long biodiversity inventory in 2017 of the Boston Harbor Islands using the iNaturalist platform found 475 species. By comparison, the 202 species identified on Thompson Island represent the short, late-summer sampling period and the three-hour-per-year limit to the activity. This short field experience contributes to the relatively low (19%) proportion of species common to the three years (Figures 3 and 4). The students were predictably attracted to species that were easily photographed (e.g., did not move or were of the right size). Examples include herbs and shrubs that were flowering or fruiting, oysters, mussels, snail shells, and insects such as butterflies. Nonetheless these observations build a permanent (assuming they are not withdrawn) and accessible record that can contribute to future studies of biodiversity such as species distributions, invasive species, phenology, and population studies.
Such data could also be of value to future researchers with more focused interests.
As noted in the Introduction, a common data-quality issue in biodiversity records is taxon identification. Because of the diversity of the information in the records, differences in the identification and review process, and because the taxa vary widely across projects, a logical measure of data quality would be to compare the overall percentage of Research-quality observations in the Orientation Retreat project with the percentage found across the entire iNaturalist platform. We expected the percentage for this project to be lower because the participants are mostly first-time users and because we have more plant than animal observations, whereas in iNaturalist, animal observations are more common than plant observations. The percentage of Research Grade observations in iNaturalist is 40.8 % (based on 23.0 million observations that are of Research Grade in GBIF and 56.4 million observations overall on the iNaturalist website), whereas the identification process for the Orientation Retreat projects has converted just 34.0% (Figure 5) of the observations to Research Grade. It seems that the observations provided by these first time users were advanced to Research Grade at approximately 80% of those of the iNaturalist community as a whole.
Although there are many ways that images of a species can be improved, including image focus, documenting the whole organism and its parts, and capturing details essential for identification, most of these aspects depend on experience and knowledge of the species being observed.
One of the benefits of smartphone use is the recording of precise GPS location and the date and time of the observation. Date-time data were precise. We also expected that using cell phone location technology, at a study site within a few miles of a major metropolitan city such as Boston, would lead to precise and accurate species locations, and that was what we generally found (Figures 7 and 8). However, the location data depends on the cell phone signal provider and relative tower location. Location errors associated with an observation can be large, and consequently, 2.7% of observations averaged over three years that should have been included in the projects were initially not included, and the location uncertainty for observations was greater than 10 m for 30% of all observations (Figure 8). We also found that the location names in the data files for this coastal/marine environment caused confusion. It is important for people analyzing location data to keep these potential uncertainties in mind.
We undertook analysis of the data collected by students using iNaturalist to answer the questions “Can and under what conditions might students entering college with little or no experience observing nature make useful contributions to characterize biodiversity in our own backyards?” We found that first-time users can make useful contributions. We believe a combination of factors work synergistically for success. These include:
As in most endeavors, there are several aspects that can be improved by:
The iNaturalist Honors College retreat exercise was designed to get incoming students outside to observe nature and to engage with a CS project (Rokop et al. submitted). In addition to this educational goal, we investigated the scientific contribution of the bioblitzes. College students, with their years of experience using smartphones, can, with only brief instruction, collect and post observations to the iNaturalist platform. We conclude that these students using iNaturalist, with its crowd-sourced ID process, have produced data that are useful for a variety of biodiversity studies.
Data available through the iNaturalist project pages as listed in the paper and provided in Supplemental Files 2 and 3.
The supplementary files for this article can be found as follows:Supplemental File 1: Supplemental Figure 1Supplemental File 2: Data File 1
Excel file of iNaturalist records for all three years of the study containing only student observations. DOI: https://doi.org/10.5334/cstp.407.s2Supplemental File 3: Date File 2
Excel data of iNaturalist records for all three years of the study containing observations and the history of identification of species in the images in the crowdsourcing process. DOI: https://doi.org/10.5334/cstp.407.s3
The authors would like to acknowledge the assistance of Megan Rokop of UMass Boston Honor’s College and both Marc Albert and Rachel Vincent from the National Park Service. We thank the Reviewers, whose comments significantly improved the manuscript.
Publication costs were covered by National Science Foundation RCN-UBE award #1919928.
The authors have no competing interests to declare.
Dr. Stevenson was responsible for the conception of the study, and major aspects of the data analysis. Drs. Merrill and Burn analyzed and graded observations, and identified species. All three authors contributed to writing the paper.
Aceves-Bueno, E, Adeleye, AS, Feraud, M, Huang, Y, Tao, M, Yang, Y and Anderson, SE. 2017. The accuracy of citizen science data: a quantitative review. Bulletin of the Ecological Society of America, 98(4): 278–290. DOI: https://doi.org/10.1002/bes2.1336
Bailie, C and Grabowski, JH. 2018. Invasion dynamics: interactions between the European Green Crab Carcinus maenas and the Asian Shore Crab Hemigrapsus sanguineus. Biological Invasions, 21: 787–802. DOI: https://doi.org/10.1007/s10530-018-1858-1
Bonney, R, Cooper, CB, Dickinson, J, Kelling, S, Phillips, T, Rosenberg, KV and Shirk, J. 2009. Citizen science: a developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11): 977–984. DOI: https://doi.org/10.1525/bio.2009.59.11.9
Bonney, R, Phillips, TB, Ballard, HL and Enck, JW. 2016. Can citizen science enhance public understanding of science? Public Understanding of Science, 25(1): 2–16. DOI: https://doi.org/10.1177/0963662515607406
Chapman, AD. 2005. Principles of data quality. GBIF. DOI: https://doi.org/10.15468/doc.jrgg-a190
Cohn, JP. 2008. Citizen science: Can volunteers do real research? BioScience, 58(3): 192–197. DOI: https://doi.org/10.1641/B580303
Dickinson, JL, Zuckerberg, B and Bonter, DN. 2010. Citizen science as an ecological research tool: challenges and benefits. Annual review of ecology, evolution, and systematics, 41: 149–172. DOI: https://doi.org/10.1146/annurev-ecolsys-102209-144636
Flaherty, EA, Walker, SM, Forrester, JH and Ben-David, M. 2017. Effects of course-based undergraduate research experiences (CURE) on wildlife students. Wildlife Society Bulletin, 41(4): 701–711. DOI: https://doi.org/10.1002/wsb.810
Freitag, A, Meyer, R and Whiteman, L. 2016. Strategies employed by citizen science programs to increase the credibility of their data. Citizen Science: Theory and Practice, 1(1). DOI: https://doi.org/10.5334/cstp.91
GoBotany. 2021. Native Plant Trust. https://gobotany.nativeplanttrust.org/species/quercus/robur/ accessed January 8, 2021
Gura, T. 2013. Citizen science: amateur experts. Nature, 496(7444): 259–261. DOI: https://doi.org/10.1038/nj7444-259a
Harlin, J, Kloetzer, L, Patton, D, Leonhard, C and Leysin American School high school students. 2018. Chap 28 Turning students into citizen scientists. In Hecke, S , Hakla, M, Bowser, A, Makuch, Z, Vogel, J & Bonn, A (Eds.), Citizen Science: Innovation in Open Science, Society and Policy. UCL Press: 410–428 (19 pages). https://www.jstor.org/stable/j.ctv550cf2.35. DOI: https://doi.org/10.2307/j.ctv550cf2.35
iNaturalist. 2021. https://www.inaturalist.org/observations?place_id=any&taxon_id=56133. Accessed January 8, 2021.
Kelling, S, Fink, D, La Sorte, FA, Johnston, A, Bruns, NE and Hochachka, WM. 2015. Taking a ‘Big Data’ Approach to data quality in a citizen science project. Ambio, 44(4): 601–611. DOI: https://doi.org/10.1007/s13280-015-0710-4
Kelling, S, Gerbracht, J, Fink, D, Lagoze, C, Wong, WK, Yu, J, Damoulas, T and Gomes, C. 2012. A human/computer learning network to improve biodiversity conservation and research. AI magazine, 34(1): 10–20. DOI: https://doi.org/10.1609/aimag.v34i1.2431
Kosmala, M, Wiggins, A, Swanson, A and Simmons, B. 2016. Assessing data quality in citizen science. Frontiers in Ecology and the Environment, 14(10): 551–560. DOI: https://doi.org/10.1002/fee.1436
Parrish, JK, Burgess, H, Weltzin, JF, Fortson, L, Wiggins, A and Simmons, B. 2018. Exposing the in citizen science: fitness to purpose and intentional design. Integrative and comparative biology, 58(1): 150–160. DOI: https://doi.org/10.1093/icb/icy032
Parrish, JK, Jones, T, Burgess, HK, He, Y, Fortson, L and Cavalier, D. 2019. Hoping for optimality or designing for inclusion: Persistence, learning, and the social network of citizen science. Proceedings of the National Academy of Sciences, 116(6): 1894–1901. DOI: https://doi.org/10.1073/pnas.1807186115
Parrish, JK, Litle, K, Dolliver, J, Hass, T, Burgess, HK, Frost, E, Wright, CW and Jones, T. 2017. Defining the baseline and tracking change in seabird populations: the Coastal Observation and Seabird Survey Team (COASST). In Citizen science for coastal and marine conservation, 19–38. Routledge. DOI: https://doi.org/10.4324/9781315638966
R Core Team. 2020. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. URL https://www.R-project.org/.
Raddick, MJ, Bracey, G, Carney, K, Gyuk, G, Borne, K, Wallin, J, Jacoby, S and Planetarium, A. 2009. Citizen science: status and research directions for the coming decade. AGB Stars and Related Phenomenastro 2010: The Astronomy and Astrophysics Decadal Survey, 2010: 46P.
RStudio Team. 2019. RStudio: Integrated Development for R. Boston, MA: RStudio, Inc. URL http://www.rstudio.com/.
Schmitt, EF, Wells, DF and Sullivan-Sealey, KM. 1998. Surveying coral reef fishes: a manual for data collection, processing, and interpretation of fish survey information for the tropical northwest Atlantic. Reef Environmental Education Foundation. https://nsuworks.nova.edu/cnso_bio_facbooks/5.
Spell, RM, Guinan, JA, Miller, KR and Beck, CW. 2014. Redefining authentic research experiences in introductory biology laboratories and barriers to their implementation. CBE—Life Sciences Education, 13(1): 102–110. DOI: https://doi.org/10.1187/cbe.13-08-0169
Stevenson, RD, Suomela, T, Kim, H and He, Y. 2021. Seven Primary Data Types in Citizen Science Determine Data Quality Requirements and Methods. REVIEW article Frontiers in Climate Change 3, 2 June 2021. DOI: https://doi.org/10.3389/fclim.2021.645120
Understanding Science. 2021. University of California Museum of Paleontology. Accessed 3 January 2021 <http://www.understandingscience.org>.
Wiggins, A, Newman, G, Stevenson, RD and Crowston, K. 2011. December. Mechanisms for data quality and validation in citizen science. In 2011 IEEE Seventh International Conference on e-Science Workshops, 14–19. IEEE. DOI: https://doi.org/10.1109/eScienceW.2011.27
Yu, J, Kelling, S, Gerbracht, J and Wong, WK. 2012, October. Automated data verification in a large-scale citizen science project: a case study. In 2012 IEEE 8th International Conference on E-Science, 1-8. IEEE. DOI: https://doi.org/10.1109/eScience.2012.6404472