Evaluation of observations

iNaturalist evaluates observations with a three-category system of Casual, Needs ID, and Research Grade (See here for the official rating details). In addition to the iNaturalist evaluations, we tallied the number of photographs per observation and developed a rubric to score the quality of images as good, OK, or poor (See below for further explanation). We identified whether or not the observer tried to identify the species being observed, and we also evaluated the likelihood that an expert in a specific taxonomic group could ID the species using the images taken (and any recorded notes). We totaled the number of observations that were identified to the species and genus level by August 1st, 2020. Lastly, we evaluated the accuracy of GPS data, the assigned place names of the observations, and the time stamps of the observations.

Image quality of an observation was determined for the group of photographs representing a specific specimen taken in aggregate (a recorded observation) using criteria of Table 1. It was important that a single species was clearly identifiable, properly lighted, and focused. If one or more photos included adjacent organisms, then at least one photograph must focus specifically on the selected species, and if there is only one photograph focusing specifically on the selected species, it must focus on aspects generally useful for taxonomic identification (examples are found in Figure 1 and Supplemental File 1: Supplemental Figure 1). Identification from photographs can be challenging but can be made more reliable by appropriate views of in-focus photographs. Capturing an image of the entire organism from a distance provides context for more detailed photographs. Observations were rated good for animals if multiple views were captured (lateral, dorsal, ventral), including images of appendages as appropriate. Likewise, a photograph or a group of photographs including an entire plant (above-ground aspects), branching pattern, leafing pattern, bark, and flowers or fruits (if present), was rated as good. For some species, this could be achieved in a single photograph and thus the image quality might be rated good if the observation included a single, in-focus photograph in which the intended species existed alone or was irrefutably the intended species (Figure 1). Such irrefutable intention was at times, although not often, made possible by explanatory notes recorded with the observation. In some instances, a single photograph or a collection of photographs was sufficient for identification of a commonly recognized species, but the observation was still rated poor or OK because it alone, or the aggregate photographs included, did not satisfy the above criteria. Thus, quality rating of an observation and its usefulness for identification were not necessarily tightly coupled. An OK rating indicated that the observation partially satisfied the criteria, but could easily be improved by better or additional photographs. Inclusion in an image of an object of known size (approximate or precise) generally improved ratings of an observation.

During the course of the rubric development and scoring, we discovered that some observations made by students during the orientation program were not included in the project by the iNaturalist algorithm, a possibility we had not originally considered. As a consequence, we gathered and analyzed these additional observations located on the basis of user name, time frame, and GPS location. After completing these extra steps, we were confident that we had included all of the observations made by all users who were part of the orientation program. For analyses we relied on iNaturalist tools, Excel, Google docs, R (R Core Team, 2020), RStudio (RStudio Team 2019), and the rinat and iNatTools libraries.

Use of the iNaturalist platform

Roughly 700 observations were collected each year using approximately 85 first-time user accounts (Table 2) that documented about 100 species (Table 3). The numbers given in Table 2 include Park Service employees as well as students. Readers should be aware that we found that all the numbers in Table 2 changed over time because of data curation by project managers, individuals withdrawing their observations, and more identifications being made over time.

There were almost twice as many student participants the first year compared with the 2nd and 3rd years, but the number of user accounts varied little (Table 3). The number of student observations and species identified increased slightly in each subsequent year (Table 3). The groups in the second and third years were comparable in their productivity, with each making about 9 observations per account and identifying 1.5 species (Table 3).

Overview of biodiversity documented

In the first two years, most observations were of plants (about 50%) and animals (40%), with the remaining 10% divided between fungi and algae (Figure 2). For 2019, the number of observations of fungi and algae remained the same as in prior years, the number of plant and animal observations increased, and the not available (NA) category dropped substantially relative to the first two years (Figure 2). NA might indicate that identity of the observation was difficult or not possible, images were of humans or of objects not organisms, several organisms were included in the image with insufficient information to determine intention, etc.

A total of 202 unique species were identified across the three years from the student orientation programs with 19% of the species common to all 3 years (Figure 3). Taxonomic diversity was high and included more than 90 families and about 100 species each year (based on analysis of iNaturalist data). Approximately 34% of the observations were identified to species level and thus achieved Research Grade. There was much more success with identification of animals and plants than of fungi and chromista (algae).

Among animals, the most commonly identified species were marine mollusks, crustaceans, and insects (Figure 4). These included hermit crabs, green shore crabs, blue mussel, common periwinkles, acorn barnacles, and European flat oysters in the marine habitats; and butterflies, bumblebees, grasshoppers, and spiders in the terrestrial habitats. As with animals, the students recorded or observed the most obvious plants, such as colorful herbaceous flowers or shrubs that were flowering or fruiting in the late summer.

Notable biodiversity observations

Of the more than 200 species observed by students on the island, four struck us as immediately noteworthy. First, a student caught and photographed the Dekay’s Brownsnake (Storeria dekay). A search of iNaturalist found this to be the first and only observation of this species on the island (R. Vincent, personal communication). Second, there is a thriving population of English Oak (Quercus robur) that have naturalized the island. Quercus robur occurs across the USA and Canada and throughout New England (GoBotany 2021, iNaturalist 2021), but the largest and densest number of individuals in Massachusetts appears to be on Thompson Island (iNaturalist 2021). Lastly, students documented a large number of two invasive crab species, the European green crab (Carcinus maenas) and the Asian shore crab (Hemigrapsus sanguineus). The green crab has been on our Atlantic coast for 200 hundred years, whereas the Asian shore crab arrived in New Jersey in the 1980s and is now extending its range and population density along our coast (Bailie and Grabowski 2018). These data are likely to contribute to our understanding of how these species are interacting over time and space.

Data quality

Students were encouraged to photograph more than one view/character of the species they were observing, and indeed they did so. Sixty percent of the observations have 2 or 3 images, while the amount with just one image was between 10% and 20%.

Aspects of data quality useful for species identification improved over the three year period and especially between the first and the second year. Observers increased the number of observations they tried to identify from 2017 to 2019 (Figure 5a). The “Research” and “Needs ID” observation grades increased, and the NA grade decreased during the course of the study (Figure 5b), with about ⅓ of the observations achieving Research Grade.

We further asked, “Is it possible to identify this observation to species with the images provided?” In 55% to 70% of the cases we thought it was possible, meaning we think that with more attention by expert identifiers on the ID process, more observations can attain Research Grade (Table 1; Figure 6a). We found that the quality of the collection of images for each observation (Table 1) was mainly OK or good in 85% of the observations; 15 % were poor (Figure 6b).

Location data were accurate as judged by two criteria. When plotted, observations follow expectations for terrestrial species (Figure 7, top panel) found across the island, mostly along walking paths, while marine species were observed along the shoreline (Figure 7, bottom panel). A few of the marine observations appear in deeper water indicating larger location errors. Those at the lowest latitudes (< 42.31°N) may represent observations made at low tide. The mode and median positional accuracy were good at 5 and 6 meters respectively (Figure 8).

Three issues compromised the quality of the location data. First, as noted above, when reviewing project data by user ID, we discovered that some observation records were not recorded as being on Thompson Island when they should have been, owing to low GPS accuracy. These observations were recovered by searching for observation records within the appropriate time window from participating contributors. In 2017, 4.7% (n = 29) of the observations collected were not included. The corresponding values for 2018 and 2019 were 4.3% (n = 27) and 0.6% (n = 5). Second, occasionally there was large uncertainty about an observation, leading to a long tail to the distribution and a mean of 41 m (Figure 8). Third, there were more than 60 different place names given in the data set, making us initially question the location of an observation even though the vast major of GPS coordinates accurately located observations on Thompson Island.

We found no discernable errors in the date-time data. It was our confidence in the date-time date that allowed us to discover the observations that did not make it into the projects described in the previous paragraph.

Identification process

The identification process of iNaturalist is key to its success. The process allows anyone willing to review images the opportunity to help identify organisms. In all, there were 6,665 identifications made by 713 different identifiers over three years. The number of unique reviewers by year was 269, 319, and 312. The identification process revealed that most identifiers helped refine identifications.

Biodiversity findings

Our naive users collected more than 5,600 images, which resulted in the identification of 202 species and potentially more. A year-long biodiversity inventory in 2017 of the Boston Harbor Islands using the iNaturalist platform found 475 species. By comparison, the 202 species identified on Thompson Island represent the short, late-summer sampling period and the three-hour-per-year limit to the activity. This short field experience contributes to the relatively low (19%) proportion of species common to the three years (Figures 3 and 4). The students were predictably attracted to species that were easily photographed (e.g., did not move or were of the right size). Examples include herbs and shrubs that were flowering or fruiting, oysters, mussels, snail shells, and insects such as butterflies. Nonetheless these observations build a permanent (assuming they are not withdrawn) and accessible record that can contribute to future studies of biodiversity such as species distributions, invasive species, phenology, and population studies.

Such data could also be of value to future researchers with more focused interests.

What is the quality of the data collected?

As noted in the Introduction, a common data-quality issue in biodiversity records is taxon identification. Because of the diversity of the information in the records, differences in the identification and review process, and because the taxa vary widely across projects, a logical measure of data quality would be to compare the overall percentage of Research-quality observations in the Orientation Retreat project with the percentage found across the entire iNaturalist platform. We expected the percentage for this project to be lower because the participants are mostly first-time users and because we have more plant than animal observations, whereas in iNaturalist, animal observations are more common than plant observations. The percentage of Research Grade observations in iNaturalist is 40.8 % (based on 23.0 million observations that are of Research Grade in GBIF and 56.4 million observations overall on the iNaturalist website), whereas the identification process for the Orientation Retreat projects has converted just 34.0% (Figure 5) of the observations to Research Grade. It seems that the observations provided by these first time users were advanced to Research Grade at approximately 80% of those of the iNaturalist community as a whole.

Although there are many ways that images of a species can be improved, including image focus, documenting the whole organism and its parts, and capturing details essential for identification, most of these aspects depend on experience and knowledge of the species being observed.

One of the benefits of smartphone use is the recording of precise GPS location and the date and time of the observation. Date-time data were precise. We also expected that using cell phone location technology, at a study site within a few miles of a major metropolitan city such as Boston, would lead to precise and accurate species locations, and that was what we generally found (Figures 7 and 8). However, the location data depends on the cell phone signal provider and relative tower location. Location errors associated with an observation can be large, and consequently, 2.7% of observations averaged over three years that should have been included in the projects were initially not included, and the location uncertainty for observations was greater than 10 m for 30% of all observations (Figure 8). We also found that the location names in the data files for this coastal/marine environment caused confusion. It is important for people analyzing location data to keep these potential uncertainties in mind.

Recommendations for program design

We undertook analysis of the data collected by students using iNaturalist to answer the questions “Can and under what conditions might students entering college with little or no experience observing nature make useful contributions to characterize biodiversity in our own backyards?” We found that first-time users can make useful contributions. We believe a combination of factors work synergistically for success. These include:

1. the separation of the collection of data from the identification process on the iNaturalist platform,
2. the superbly designed interface of iNaturalist providing a minimum threshold for students to start to contribute,
3. the number of sensors (built in cameras and GPS) and computer power of the current generation of smartphones,
4. the years of experiences students have with using smartphones,
5. the species image–recognition software available in iNaturalist,
6. the initial training sessions and guidance during the session, and
7. project curation after collection within the iNaturalist database.

As in most endeavors, there are several aspects that can be improved by:

1. gaining better skills at taking close-up images,
2. paying more attention to scale and parts of different species, and
3. increasing the use of the improving capability of species-identification software in iNaturalist, which was clearly enhanced over the 3-year study.