Introduction and Background

One of the most inflammatory topics in animal welfare and wildlife conservation is free-roaming domestic cat (Felis catus) management (Loss and Marra 2018; Lynn et al. 2019). Free-roaming domestic cats, as we use the term here, cover a behavioral spectrum from fractious feral animals to sociable outdoor pets. There is a need to control their populations to protect threatened and endangered wildlife species (Loss and Marra 2018), safeguard public health from disease and injury (Kravetz and Federman 2002), and curtail nuisance behaviors such as noisy mating activities (Levy, Isaza, and Scott 2014), urine spraying, and fighting with pets (Gramza et al. 2016).

The health and well being of free-roaming cats themselves is an important issue to cat welfare proponents, as well as to the public, which generally supports non-lethal management of cats (Wald, Jacobson, and Levy 2013). Cat advocates, focused on reducing feline deaths, champion the use of spay/neuter (often called trap-neuter-return or TNR, and community cat programs) with the aim of shrinking free-roaming cat populations over time as sterilized adults do not replace themselves with new kittens (Boone et al. 2019). While evidence exists to support the arguments that spay/neuter efforts are linked to reductions in animal shelter intake and euthanasia (Spehar and Wolf 2019), no long-term field research has demonstrated that these programs are affecting the metric of concern to conservation stakeholders: the sheer number of free-roaming cats on the landscape. Two controlled studies of spay/neuter interventions in North America ran for only a year apiece and concluded that the treatment areas did not experience a population decline (Bissonnette et al. 2018; Kilgour et al. 2016). This research is dismissed by cat advocates as having failed to collect data for a sufficient amount of time.

There is a need for long-term monitoring of the potential effects of spay/neuter programs on free-roaming cat population sizes. To be acceptable to all stakeholders, this research must combine science-based population estimation methods and multi-year time spans. Such an initiative could answer questions such as whether spay/neuter programs can reduce population sizes of free-roaming cats, what proportion of cats in an area need to be sterilized to see a population reduction, and how long it takes to see that decline. While computer simulations have resulting in widely varying answers, these ideas have never been field tested. Further, these answers are likely to vary based on location-specific context, so there is a need to replicate the same style of cat population monitoring in multiple locations so program efficacy can be compared based on factors such as climate and level of urbanization.

The gold standard for quantifying animal populations is photographic mark-recapture modeling, which estimates a population size based on how often animals in an area are detected and photographed, often combined with spatial data. An animal is “captured” and “marked” when it is first photographed and identified, and “recaptured” when photographed again, creating sighting histories for analysis. While there are statistical methods being developed to perform this type of modeling with only partial identification of individuals (Augustine et al. 2018), the ability to accurately identify each individual is preferable.

Photo identification using an animal’s natural marks has been tested across many terrestrial and marine species (e.g., Gibbon, Bindemann, and Roberts 2015; Gilkinson et al. 2007; Gomez-Salazar, Trujillo, and Whitehead 2011; Zheng et al. 2016). These studies sometimes compare the accuracy of identifiers from two groups: those who are considered experts or professionals, and those presumed to be less experienced, such as citizen science volunteers (Chesser 2012). However, despite an abundance of free-roaming cat population research that relies on photos as data (e.g. Cove et al. 2017), and individual photo identification studies in other felids (Johansson et al. 2020; Park et al. 2019), no whole-body photo identification work has been published about domestic cats.

Researchers from the world of machine learning and computer vision have had success in testing algorithms for identifying domestic cats by their noses (Chen et al. 2016) and faces (Lin and Kuo 2018; Klein 2019), but these approaches rely on high-resolution, well-lit photos of selected areas of the body in cooperative subjects. Such images would not be obtainable by field biologists collecting free-roaming cat photos via camera traps, smartphones, or possibly even DSLR cameras with telephoto lenses. Artificial intelligence tools are not yet advanced to the point where they can be used for identifying individual domestic cats in real-world research applications.

Field data collection for photographic mark-recapture research commonly uses motion-activated camera traps placed over a study area to obtain photos for building capture histories. However, these methods are not without drawbacks and limitations. Maintaining a camera trap array consumes professional labor, cameras and security boxes require purchasing and maintenance, and time is spent sifting through massive photo collections where many images feature swaying plants or the movement of humans and other non-target species. Camera trap placement permissions, theft, and vandalism are also of concern, especially if used in urban and populated areas.

We propose an alternative to traditional camera trap arrays and data management: engaging volunteers as both smartphone-wielding human camera traps and as data processors in identifying cats and building capture histories for analysis. While one free-roaming cat research program solicited members of the public to host camera traps on their property as part of the DC Cat Count (Herrera et al. 2021) and another deploys volunteers in road-based cat counts for the Hayden Island Project in Portland, OR (Cats Safe at Home, no date) no one has yet attempted a holistic, multi-role citizen science approach for studying free-roaming cat populations. To that end, a nonprofit citizen science program, Kitizen Science, is being developed to monitor free-roaming cat population sizes in urbanized areas in collaboration with spay/neuter organizations.

A flourishing body of literature appeals to practitioners to ensure that tapping into crowdsourced free labor does not cause an unacceptable decrease in the rigor of the scientific process (Kosmala et al. 2016, Parrish et al. 2018). While there is always benefit to ensuring that research methods are subject to transparency, validation, and piloting measures, this is especially true in areas with high conflict between stakeholder groups and where results could affect policy decisions.

In response to these needs, the first validation study of Kitizen Science’s methods tested our intended approach’s most basic assumption: whether citizen science volunteers drawn from a pool of cat advocates can identify individual free-roaming cats in smartphone photos at a reasonable level of accuracy, thereby making them capable of building credible capture histories for mark-recapture analysis. One metric for defining a reasonable level of accuracy is to compare volunteers in a citizen science program’s target demographic against those in the demographic that is already trusted to perform this type of task. Here, we tested the success of cat advocate citizen science volunteers in identifying and matching cats taken with smartphone photos, and compared them with a reference group of biology and environmental science students—a labor pool commonly trusted to organize data from camera trap research. Further, our study sought to probe the question of cat identification from two perspectives: what makes a volunteer better at cat identification, and what makes a cat photo more identifiable?

Methods

Study website

We organized photo-matching tasks into trials of 120 pairwise photo comparisons in which our website presented a volunteer with 1 of 50 randomized photos of a different cat to match and a randomized photo of a potential match, and the volunteer was asked a simple yes/no question of whether the same cat appeared in both photos (Figure 1). Each trial contained 0 to 5 correct matches for each cat to match within the library of 120 images. This arrangement was chosen to be a simplified version of how our citizen science website workflow will function: volunteers will be presented with a cat to match and then look through potential matches in search of the same individual. Volunteers were required to make a binary “yes” or “no” decision on each pair of photos (Chesser 2012; Gibbon, Bindemann, and Roberts 2015), as previous research found that photo classification participants offered an uncertain option overused it (Swanson et al. 2016). We restricted volunteers to completing a maximum of 2 trials (240 photo pair comparisons) per 24-hour period to avoid observer fatigue (Chesser 2012). For analysis, we excluded incomplete matching trials (Gould, Clulow, and Clulow 2021) on the assumption a user was merely curious about the project and not putting in genuine effort (Van Horn et al. 2014).

Figure 1 

A screenshot of our study website that presented volunteers with two cat photos for comparison. Users could zoom/pan to explore detail.

Results

In total, 151 cat advocates and 17 life science university students participated in this study by completing a total of 37,800 pairwise photo comparisons using our online platform (See Supplemental Files 1–5 for data analyzed).

Our volunteer background and demographics survey revealed similarities and differences between groups. The mean age for cat advocates was 47.1 (range 19–76) and 23.9 (range 18–50) for students. In both volunteer groups, the majority were women (cat advocates 90.7%, students 82.3%) and selected white as their race/ethnicity (cat advocates 90.1%, students 70.6%). Many in both groups had a pet cat or cats (cat advocates 90.1%, students 58.8%), with 35.1% of cat advocates and no students reporting that they cared for feral/free-roaming cats. Only 7.9% of cat advocates had previously participated in an online citizen science project doing image identification or classification, compared with 23.5% of students. A majority (62.2%) of cat advocates held a bachelor’s degree or higher. Regarding experience volunteering with cats, 57.6% of cat advocates were currently involved, 17.9% previously volunteered, and 24.5% reported having never volunteered with cats. Among students, 17.6% had previously volunteered to do image identification or classification as part of research that was not online citizen science, such as viewing camera trap images for University of Washington wildlife researchers. This student group was comprised largely of undergraduates (82.3%), with a few master’s and doctoral students (5.9% and 11.8%, respectively). See volunteer demographics summarized in Table 1.

Cat advocates’ matching attempts (n = 34,080) were correct 98.1% of the time compared with students’ 97.5% (n = 3,720). Among cat advocates, there were 33,089 true negatives, 329 true positives, 143 false negatives, and 519 false positives. The students had 3,594 true negatives, 34 true positives, 10 false negatives, and 82 false positives. While both groups performed highly, students made 27.3% more errors than cat advocates. This difference between groups was statistically significant using a chi-squared test (χ2 = 4.831, df = 1, p-value = 0.0280) using an alpha of 0.05. In considering the two different error types, which add confusion to animal capture histories and impact population estimates, cat advocates were 3.6 times more likely to make false positive than false negative errors, compared with the students’ 8.2-fold increase.

Our best model (by AICc) to determine what influences a human’s ability to match cat photos accurately was the GLMM, which contained user as a random effect and whether one had a pet cat, whether one had volunteered with cats previously, and whether one held a bachelor’s degree or higher as fixed effects (Table 2). Reporting a pet cat increased a volunteer’s cat photo identification accuracy, and holding less than a bachelor’s degree or having ever volunteered with cats reduced accuracy. Model diagnostics supported the model as fitting the data. In performing an ANOVA test between our best model and an intercept-only GLM to determine the importance of including user as a random effect, our best model was significant (<0.05). A Chi-squared goodness-of-fit test indicated no lack of fit (p > 0.05). The model was underdispersed due to a high accuracy rate by most volunteers.

Our best model (by AICc) to determine what makes a cat photo more identifiable was the GLMM which contained photo as a random effect and whether the cat was black as a fixed effect (Table 3). If a cat was a color other than black, its ability to be correctly identified increased. Model diagnostics supported the model as fitting the data. In performing an ANOVA test between our best model and an intercept-only GLM to determine the importance of including photo as a random effect, our best model was significant (<0.05). A Chi-squared goodness-of-fit test indicated no lack of fit (p > 0.05). As with above, this model was underdispersed due to a high matching accuracy rate by most volunteers.

Discussion

Here, we demonstrate that our sample of cat advocate citizen science volunteers were not only adequate at identifying and matching individual cats in smartphone photos, but performed better at the task than our sample of life science university students. These findings support our desire to engage cat advocates in matching cats and building capture histories for use in photographic mark-recapture population estimation. Further, by casting a wide net through online citizen science, we attracted nearly nine times as many participants as responded to a well-advertised student volunteer position at a large university.

Our results reveal that in our program’s target demographic of cat advocates, participants who were best at identifying and matching cats in smartphone photos are those who reported having a pet cat, holding a bachelor’s degree or higher, and having not previously volunteered with cats. The finding that a user having volunteered with cats reduced their level of photo identification accuracy was unexpected. One potential explanation for this seemingly paradoxical result may be that participants with cat volunteering experience (one metric of expertise in cats) were over-confident in their abilities and less careful in determining matches. However, a different metric of expertise, having a pet cat, increased users’ cat identification accuracy. From an applied perspective, this suggests that volunteer recruitment from an audience of cat advocates with pet cats is important, but recruiting those who already have experience volunteering with cats is unnecessary or even counter-productive. This allows us to solicit future volunteers from a broader audience of people who like cats while not incentivizing us to poach volunteers from cat welfare organizations.

Defining (and self-defining) expertise in animal photo identification is a complex phenomenon that has been explored by previous researchers. In a study which asked attendants at a herpetofauna conference to sort images of newts, Austen et al. (2018) noted that their volunteer with the highest accuracy score self-rated their newt identification skills as worse than their peers, but the volunteer with the lowest accuracy score self-rated as possessing the same level of expertise as their peers. Overall, however, no statistically significant difference in accuracy was detected in comparing groups of volunteers based on how they self-assessed their abilities.

Other research has also asked the question of which human traits lead to better success in photo identification. Similarly to our finding that possessing a bachelor’s degree improved cat identification accuracy, Delaney et al. (2008) discovered that education was an important predictor of accuracy in field-based species-level identification among participants in a marine invasive species program, where volunteers with two years of university education exceeded 95% accuracy. Testing the effect of an online training program on species-level identification, researchers found that volunteers with a background in biology had a higher percentage correct than those without, although the latter’s accuracy rose to a similar level with the addition of training (Katrak-Adefowora, Blickley, and Zellmer 2020). In a validation study of the potential of using citizen science volunteers for identifying individual Andean bears, Van Horn et al. (2014) found no personal characteristics that had a meaningful effect on participant accuracy.

Both identification error types introduce bias into capture histories for photographic mark-recapture population estimation, with a false positive trend resulting in an underestimation of the true population size (Urian et al. 2015) through an increase in the capture probability (Johansson et al. 2020). In our study, cat advocates were 3.6 times more likely, and students 8.2 times more likely, to make false positive than false negative errors in cat identification. Looking at identification errors in camera trap images of giant pandas, Zheng et al. (2016) discovered a mean bias of 1.58 toward false positives. Conversely, using camera trap photos of snow leopards, researchers found that capture events were 3 times more likely to be false negatives than false positives (Johansson et al. 2020). Testing images of salamanders, Chesser (2012) found that observers were 185 times more likely to make false negative identification errors than false positives. An Andean bear photo identification study found a bias toward false negatives (Van Horn et al. 2014). Studying the feasibility of using natural markings to identify giant anteaters, 87.5% of errors were false negatives (Möcklinghoff, Schuchmann, and Marques 2018). In seeking to identify polar bears by whisker spots, researchers found that both experienced an inexperienced participants were more likely to make false negative than false positive errors (Prop, Staverløkk, and Moe 2020). These varied findings underscore the need for species-specific validation of photo identification that includes reporting not just general accuracy but also error-type trends. Chesser (2012) noted that false positives, as was the trend in our cat study, could be viewed as the preferable problem because they are less time-consuming than false negatives to identify and correct with intervention.

Unsurprisingly, many previous studies of photo identification in wildlife qualitatively report that “distinctive” animals in “high quality” photos are most readily identified. However, the effect of some metric of visual distinctiveness or photo quality are rarely explored at an analytical level (Theimer, Ray, and Bergman 2017). In our study, we found that whether a cat was black was the key covariate in determining its identifiability, outweighing other factors such as face visibility or the presence of a collar or ear tip. However, this result should not be construed as demonstrating that black cats as a whole are unidentifiable and that all photos containing black cats must be removed from photographic mark-recapture photo data collections. We would argue that thresholds for whether to include photos in building capture histories should not be made on coat color alone, but take into consideration that some black cats and some black cat photos will be sufficiently distinctive for identification, just as some nonblack cats will fall below a threshold for inclusion. (Figures 2 and 3 show our four least and four most identifiable cat photos.)

Species-level identification accuracy in citizen science has been well explored in the literature. In analyzing Snapshot Serengeti, researchers tested untrained citizen science volunteers in species identifications and compared their success with expert opinion (Swanson et al. 2016). There was a 97.9% overall agreement between these two groups of classifiers, although accuracy was lower in less-common species. Similarly, Clare et al. (2019) reported 93.4% accuracy in crowdsourced species identifications, also with higher error rates in less-common species. In comparing the skills of volunteers self-reporting no biology background, some biology background, and professional biologists, both with and without training, Katrak-Adefowora, Blickley, and Zellmer (2020) found that there was no statistically significant difference among five of the six groups, and only those without a biology background and without training displayed a low proportion of correct answers.

Apart from supporting our program’s intended approach and establishing that individual domestic cats are generally highly identifiable from smartphone photos, our results add nuance to the way that we delineate experts versus non-experts in animal identification. People with professional experience or studying for life science degrees may not necessarily be experts at the task of identifying individuals of a given species, and enthusiastic non-professionals may possess higher levels of competency in some instances. Researchers should explore whether a hobby or subculture exists around a species or study system of interest and consider working with individuals from such groups not just to save costs and engage the public in research, but to potentially improve the quality of their data. Regardless of the entity responsible for animal identification—citizen science volunteers, student assistants, professional researchers, or artificial intelligence algorithms—validation work and pre-testing of accuracy rates and error trends should accompany all research that relies on photo identification.

Conclusions

Discussions about the validity of citizen science approaches are often framed in terms of untrained members of the public versus professional scientists. There is a notable lack of acknowledgment that data collection and interpretation in traditional, university-based research may be performed by another type of nonprofessional: students. Student research assistants, with varying and unknown levels of skill, training, and dedication, complete a sizable amount of grunt work in some fields, which can include sifting through photos in camera trap–based population ecology. We argue that validation studies of citizen science approaches should compare the performance of their participants not necessarily against that of professional scientists, but with those who otherwise might be tasked with a specific responsibility.

Our study sought to demonstrate that a group of people from our program’s target demographic could identify individual cats at a similar level of success as students from a major research university. Our citizen science volunteers outperformed the students both in terms of quality and quantity of work. We do not argue that this would be the case in every discipline or situation, but we cannot make data quality comparisons without university-based researchers subjecting their processes to the same level of skepticism and scrutiny.

Data Accessibility Statements

Data analyzed during this study are included in this published article and its supplemental data files.

Supplementary Files

The supplementary files for this article can be found as follows: