Introduction

Citizen science (CS; people involved in science as contributors to research) is increasing globally as people become interested in understanding how science is done and what it is for, and become concerned about the conservation of nature (; ; ). CS outcomes are influenced and categorized by the degree of public participation in the research process and the quality of public participation as negotiated during project design (). The roles of citizen scientists vary from one extreme in formal, well-defined, contributions in which volunteers go to the field with scientists and are instructed in the appropriate research methods, to the less formal extreme in which they provide voluntary observations on their own time, such as with the many projects of Zooniverse (https://www.zooniverse.org/) and Cornell University Live Bird Cams (https://www.allaboutbirds.org/Cams/).

Contributory CS can provide important observations for datasets comprising large spatial and temporal scales, which may be important for conservation. Collaborative CS can include involvement of participants in local conservation, which may have direct consequences for species and habitat management, and indirect consequences through community education and capacity building (). In addition to simply providing additional information through numbers of participants, contributions of CS to conservation have become more important, where participants play active roles in conservation planning. With birds, CS may indicate those of public interest, discover overlooked species, and improve understanding of common species, contributing to conservation decision-making (; )

With rapidly changing technology, especially under financial constraints, CS is a potentially important ally for research and conservation. CS can increase the number of observers monitoring natural resources and observing species at risk. Benefits of CS include increasing environmental awareness, scientific literacy, social capital, citizen inclusion in local issues, and conservation (; ). Yet, CS is not without shortcomings. First, citizen science can suffer from organizational issues (volunteer interest, networking opportunities, funding, access to information). Next, scientists must trust data collected by citizen scientists, and be able to incorporate those data in their experimental designs and hypothesis tests. Finally, bureaucratic issues arise with data from CS being included in decision-making for conservation, and how to give credit when data are published (Conrad and Hilchey 2013; ; ).

Citizen scientists have helped scientists understand population trends in which analyses require long-term studies and many observations. These include behavior, bioacoustics, breeding and reproductive success, evolutionary fitness, migration, occupancy, response to habitat modification, and the spread of diseases (; ; ; ; ; ). Advantages can come from combining traditional science with CS data because more observers result in larger sample sizes, that are necessary for precise estimations for many topics of study (; ; ; ; ; ; ; ).

In Brazil, eBird (https://ebird.org/) and WikiAves (http://www.wikiaves.com.br/) are two important CS initiatives, in which citizen scientists provide observations that are added to ever-growing databases that scientists use in their own studies (). We used WikiAves, the most-used CS platform in Brazil, to compare data gathered through traditional science with that by citizen scientists, to examine similarities and differences in the numbers of bird species reported, which should converge and have similar species lists. Recognizing that the number of endemic or threatened species in any given area may be a useful guide for conservation efforts (), we examined that convergence with those groups of birds. Based on those comparisons, we offer recommendations for how traditional and CS data may better integrate to work together.

Material and Methods

Study area

The Atlantic Forest of South America is a hotspot for avian diversity, even though it has lost ~ 89% of its original area since the 1500s (; ; ; ; ). Among the largest remnants of Atlantic Forest is Iguaçu National Park (1,853 km2), at the confluence of the Paraná and Iguaçu Rivers where Argentina, Paraguay, and Brazil share borders. Much of the Atlantic Forest in this region was lost since the mid 20th century, when the state of Paraná was rapidly deforested for agriculture (, ); remaining forests reside in small (<50 ha), variably sized, and isolated fragments (). Despite deforestation and fragmentation, the Atlantic Forest still harbors more than 2,645 species of vertebrates, of which up to 7% are endemic, including approximately 215 species of birds (; ). Of those endemics, some 70 species are globally threatened ().

We focused on 50 municipalities in western Paraná (Figure 1). This region is mostly below 800 m above sea level, and with a subtropical humid mesothermal climate (Cfa; ), rainfall (average 1,650 mm) occurs throughout the year, from an average of 20 mm in August to 230 mm in October. Average annual temperature is 20.3ºC, with June (average 16ºC ± 3.2) and July (average low temperature 11.3ºC) being the coldest months and January the warmest (23.8ºC ± 3.4, average maximum 28ºC; ). Here the Atlantic Forest is predominantly seasonal semideciduous forest, with mixed forests dominated by the Paraná Pine Araucaria angustifolia at higher elevations ().

Figure 1 

Maps illustrating the studied region in southern Brazil, with (a) the state of Paraná indicated in southern Brazil, and (b) the 50 municipalities in the state of Paraná where bird records were considered (dark gray). The remaining vegetation is shown in gray, including Iguaçu National Park (darker gray, due to overlap), western Paraná.

Sources of Information

Bibliographic review

Using the Directory of Open Access Journals (DOAJ; https://doaj.org/), Google Scholar (https://scholar.google.com), JSTOR (https://www.jstor.org), Scielo (https://scielo.org), ScienceDirect (https://www.sciencedirect.com/), Scopus (https://www.scopus.com/), and Web of Science (https://login.webofknowledge.com/), we searched for scientific publications with the following keywords (and their Portuguese equivalents): birds, ornithology, threatened birds, western Paraná. We also used those search terms in Google (https://www.google.com/) and Ecosia (https://www.ecosia.org) to find additional gray literature (sensu). We included only literature found since 1980, when 17% of original forest in Paraná remained (10% remained in undisturbed forests, 2% was protected, and 5% was recovering second growth), similar to conditions found today ().

Citizen science

WikiAves (WA; https://www.wikiaves.com.br), which began in December 2008, allows users to contribute georeferenced photographs and recordings of Brazilian birds. As of 26 April 2021, WikiAves provided 1,892 species in 3,533,477 records by 37,788 users. In Paraná, 726 species were documented in a total of 185,396 photographs and sound recordings, provided by 2,385 users. We compiled all bird species documented in any of the 50 municipalities through 8 June 2020. Typically, each record is of a single species at a specific place and time, in contrast to the scientific literature, which usually provides a list and a time interval. Misidentification can occur, so we double-checked (comparing photographs or vocalizations to field and online guides) any dubious (questionable time or place or species) record and planned to exclude unresolvable records.

Approximately 558 species are expected to be found within these 50 municipalities of western Paraná (; ). All municipalities are within the Atlantic Forest domain, and so we suggest that all 558 species may be found within any individual municipality. Thus, fewer species found in any municipality should be indicative of observer effort, habitat modification, or both.

Analysis

We examined how closely traditional and CS data agreed using the Jaccard Similarity Index (): Ssc = all shared/(all shared + science + citizen), in which Ssc is the Jaccard Similarity Index between the two samples science and citizen-science, all shared is the number of species shared by both sources, science is the number of species found in our literature search only, and citizen is the number of species in CS only.

We produced species richness maps for the 50 municipalities using the statistical environment R (). Because of the goals of this study and the ambiguous nature of records, we simply map the municipalities by the number of species in each category of endemicity or conservation status. We tested for correlations between characteristics of the municipalities (total area, population size, forested area remaining, protected area) and total number of species (species richness), number of endemic species, and number of threatened species in the Atlantic Forest of western Paraná, for both traditional and CS data. Log10 transformation normalized species counts and areas for all analyses. Species accumulation curves were estimated using the package BiodiversityR, and similarity was compared using Principal Coordinates Analysis (PCoA) using the package vegan.

Atlantic Forest endemic species followed Vale et al. (), and our region includes 98 endemic species. We defined threat categories following state (), national (), and global () designations. For sample-size reasons, we created a category for comparison that includes species that are either data deficient (only at the state level; DD), or threatened at some level: vulnerable (VU), endangered (EN), or critically endangered (CR), including near threatened (NT), following the state, Brazilian, and IUCN sources. This category will be called threatened herein, for the sake of simplicity. Taxonomy and nomenclature follow the Classification of the Bird Species of South America ().

Results

Bibliographic review

A total of 15 primary and 7 gray literature sources were found in the nearly 40-year interval of studies in 15 of the 50 municipalities in western Paraná (Supplemental File 1: Table S1; Figure 2a). The primary literature listed 386 species in 68 families and 24 orders. The gray literature comprised 378 species in 71 families and 24 orders.

Figure 2 

Number of studies or records from western Paraná, in southern Brazil. (a) Number of studies in 15 of the 50 municipalities and (b) number of records by citizen scientists in 48 of the 50 municipalities. The largest remnant (grey areas) indicates Foz de Iguaçu National Park.

Literature included 466 species (84% of the expected 558) in 74 families and 25 orders (Figure 3a). The most species-rich municipalities contributed half or more of the total species count. Thus, there were 358 species in Foz do Iguaçu, 266 species in Santa Helena, 253 in Três Barras do Paraná, and 209 in Guaíra (Figure 3b).

Figure 3 

(a) Total number of species reported (summing literature and WikiAves records) in the 48 of the 50 municipalities in western Paraná, southern Brazil, (b) total number of species reported in the literature in the 15 municipalities in which avian research took place, and (c) total number of species reported by citizen scientists in Wiki Aves by municipality, in 48 of the 50 municipalities they visited. Gray areas indicate forest fragments.

A total of 83 endemic species were reported in 14 municipalities, with 73 (86% of endemics) in Foz do Iguaçu, 60 (70%) in Três Barras do Paraná, 33 (37%) in both Céu Azul and Serranópolis do Iguaçu, and 28 (34%) in Santa Helena (Supplemental File 2: Table S2; Supplemental File 9: Figure S1a). Nineteen endemic species were also included in our threatened category.

Our threatened category contained 59 species, of which 16 were DD (one was also NT by IUCN), 17 were VU (4 IUCN, 10 Brazil, 8 state), 17 were NT (16 IUCN, 1 state), 12 were EN (3 IUCN, 3 Brazil, 10 state), and four were CR (1 Brazil, 3 state; numbers do not sum to 59 because categories overlap). Threatened species were reported in 13 municipalities, most in Foz de Iguaçu, with 46 species, followed by 21 in Três Barras do Paraná and 14 in Guaíra (Supplemental File 2: Table S2; Figure 3b).

Citizen science

Citizen scientists accumulated ~ 4,000 valid records (Figure 2b), documenting 430 species (73 families, 27 orders) in 48 municipalities (Figure 3c). The most recorded species accounted for 30% (878) of all WikiAves counts, and 44 species were common and easily photographed birds. Fifty-eight species are endemic to the Atlantic Forest (13% of reported species, 54% of all Atlantic Forest endemics; Supplemental File 3: Table S3, Supplemental File 9: Figure S1c). Endemic species were a subset of those also found in the literature (only one endemic was unique to WA, the Auraucaria Tit-spinetail Leptasthenura setaria). Also, 37 were threatened, including 13 DD, 10 were VU (2 IUCN, 5 Brazil, 3 state), 10 were NT (10 IUCN, 2 state), seven were EN (1 IUCN, 2 Brazil, 6 state), and one was CR (Brazil only; Supplemental File 3: Table S3, Supplemental File 9: Figure S1d). Records from only two municipalities sum to 200 or more species, 345 species in Foz do Iguaçu and 285 in Cascavel. WA included at least one record of endemics in 31 municipalities, including 44 in Foz do Iguaçu, 34 in Cascavel, and 20 in Três Barras do Paraná (Supplemental File 3: Table S3, Supplemental File 9: Figure S1c), while threatened species were only listed in 12 municipalities, with 25 threatened species in Foz do Iguaçu (Supplemental File 3: Table S3, Supplemental File 9: Figure S1d). Two municipalities (Diamante do Sul and São José das Palmeiras) had no WikiAves records.

Combined sources

The 48 municipalities had a total of 512 reported species (two introduced, 76 families, 27 orders), of which 68 (13%) are threatened and 84 (16%) are Atlantic Forest endemics. Foz de Iguaçu had the most reported species (423, 83% of the total; Figure 3a), as well as the greatest number of records for literature and WA (Figure 3b,c). Of the endemics, 28 species are also threatened (Supplemental File 4: Table S4) and 63 are not (Supplemental file 5: Table S5). There were 52 threatened non-endemic species (Supplemental File 6: Table S6) and the remaining non-threatened, non-endemic species added to 377 (Supplemental file 7: Table S7). The literature and CS were 75% similar, with 384 shared species, but also different, with 82 species only reported in the literature and 46 species only in WA (PCoA, F1,62 = 6.02, p = 0.013; Supplemental File 8: Table S8; Figure 4). When the PCoA uses the municipality as the sample unit, species demonstrate that each method includes species not observed in the other method (Figure 4a). From the perspective of the species, municipalities also vary by method and by species (Figure 4b).

Figure 4 

(a) Bray dissimilarities between bird species compositions among municipalities based on source (scientific literature—L, WikiAves—WA), illustrating that despite similarities (0 is total similarity, 1 is total dissimilarity, each point is a municipality), the species compositions of the two sources tend to vary, as their centroids are statistically different (P = 0.013). (b) Similar figure, but here, Bray dissimilarities are between municipalities by bird species composition based on source. This indicates that many municipalities are similar in the species they have and that those similarities depend on the source.

Endemic species were reported in 33 municipalities and threatened species in 14. Forested area remaining (ha), municipality area (ha), and municipality population size were all positively correlated (all degrees of freedom are 45, Pearson correlations): forested area remaining by municipality area – r = 0.79, p < 0.001, municipality area by population size – r = 0.74, P < 0.001, forested area remaining by population size – r = 0.51, p < 0.001. Municipality population size was the variable most strongly correlated with any measure of species richness, and the correlation was due to WA. Thus, population size and total species richness were correlated in WA (rSpearman = 0.518, df = 45, p < 0.001) but not with the literature (rSpearman = 0.289, df = 15, p = 0.296, Supplemental File 10: Figure S2a). The number of endemic species was also correlated with population size in WA (rSpearman = 0.439, df = 31, p = 0.014), but not the literature (rSpearman = 0.137, df = 14, p = 0.641, Supplemental File 10: Figure S2b). The number of threatened species also increased with population size in WA (rSpearman = 0.739, df = 21, p < 0.001) but not the literature (rSpearman = 0.058, df = 13, p = 0.851, Supplemental File 10: Figure S2c). The number of all species apparently increases with municipality population size, which is correlated with municipality area and the forested area remaining in each municipality.

Overall, forested area remaining is correlated with the number of endemic species (rSpearman = 0.404, df = 24, p = 0.050). Protected area was correlated with the number of endemics in WA (rSpearman = 0.50, df = 32, p = 0.003). In the literature, protected area and the number of endemics were independent (rSpearman = 0.322, df = 12, p = 0.262). For threatened species, in WA species richness was weakly correlated with forested area remaining (rSpearman = 0.379, df = 19, p = 0.090). The number of threatened species was independent of forested area remaining in the literature (rSpearman = 0.080, df = 11, p = 0.795). There may also be a tendency for traditional science to occur more often in protected areas by municipality (10 of 15, or 66%) than WA (10 of 48, or 21%). In the literature, 460 species (99% of all species reported) were reported from protected areas, and 154 (33%) from non-protected areas. WA was more evenly divided, with 392 species (91%) reported from protected areas and 335 (78%) from non-protected areas.

Traditional science had data that permitted subdivision into samples; WA did not because each species was an independent record. Thus, by using municipalities as replicates, estimating accumulation curves was possible. Curves indicate that traditional science data reported more species more quickly than did WA (Figure 5). The Chao estimate of expected species richness was 563 species in traditional science, whereas 545 were expected in WA.

Figure 5 

Species accumulation curves comparing the data from the literature (solid line) with WikiAves (dotted line). Note that species accumulated more rapidly and with more species in the literature. In WikiAves, the predicted value (Chao, 545 species) was less than the known value (558), whereas the literature predicted more species (563).

Discussion

Traditional science studies tend to accumulate more species with less effort than CS. Citizen scientists tend to visit locations that have been neglected by traditional scientists, and so encounter species not reported by traditional scientists. CS is increasingly important for the support it provides traditional science, and that support is especially important in field studies, in which information includes species, times, locations, and other information useful for understanding migration and distribution patterns (; ). Traditional science usually requires additional information. In Brazil their contribution is already proving useful. Citizen scientists reported 430 species from western Paraná, which is approximately 90% of those expected in the municipalities, and 68% of the total of all birds (766 species) in the state (). While 48 of 50 municipalities supplied records, only 12 (25%) reported ≥ 100 species.

Determining species richness is challenging because of past and current environmental modification. Therefore, the true number of species is dynamic, including replacement of specialists by generalists occurring after habitat loss, or changing heterogeneity of anthropic habitats (; ). Reporting by citizen scientists may often provide timely and useful information to help understand those processes. We demonstrated that neither CS nor traditional science data reached the expected asymptote (Figure 5), and so both will continue to accumulate previously unreported species. Traditional science is expected to find them more quickly, but by mutually informing one another, efforts can be more effectively directed to accumulate a more complete list ().

Traditional scientists tend to focus on testing hypotheses, with methods to record more species in less time, such as capturing birds using mist-nets, or point counts to determine local species assemblages. These methods provide estimates of effort and lists of species observed. Citizen scientists, however, may go where they please to photograph target species closer to their homes (). WA tends to provide individual observations that do not permit measures of effort (eBird is an exception and does provide lists and time intervals).

Species records in this study indicated that traditional science data predicted 563 species (466 observed) with less effort, whereas CS predicted 545 (430 observed) with greater effort. Traditional science consistently includes a greater number of species that are undetected by citizen scientists (imperfect detection is a partial explanation; ), but citizen scientists visited more places than did traditional scientists. Habitat modification changes the composition of forest bird communities (; ; ) and protected areas have better, more stable, habitat. Thus, endemic and threatened species should be found there. Endemic and threatened species are also more likely to become locally extinct after habitat loss (), from which they may not recover if stepping stones or corridors are unavailable (). We found that more endemic and threatened species were noted by traditional scientists but missed by citizen scientists, even though Atlantic Forest endemics are relatively easily detectable (). Correlations between population size and species richness suggest that citizen scientists tend towards convenience and take pictures close to home rather than travel distances to search for species (). Therefore, the difference in the number of species reported between traditional and CS data is more likely to be due to sampling effort than it is to habitat alteration.

An example of this bias is Foz do Iguaçu, where more species were noted by both sources, and where more studies and WA observations took place. Iguaçu National Park includes 14 municipalities, but few species were reported from them, except for Foz do Iguaçu. Because the number of species found in the municipalities that share the national park should be similar, the low numbers are due to poor sampling because of the lack of infrastructure that permits park use in those places. In eBird (in Brazil since 2015), as of 8 June 2020, birds (439 species) were reported from only eight municipalities in western Paraná, almost all of which were reported from Foz do Iguaçu. Although eBird allows including the list of birds seen during an outing, it remains biased towards birding destinations.

Indeed, because population dynamics and local extinctions often depend on forest fragment size and quality (; ), we expect that species richness in all these municipalities should be associated with the quality and size of the forested area remaining. However, because of sampling biases, it remains impossible to determine how much those factors, and sampling bias, contribute to the observed number of species. One of our goals is to suggest changes that will help resolve this issue.

The literature includes older studies and studies that specifically looked for rare or endemic species (; ). Some of those studies include difficult-to-find species, and species that are now absent, lost due to habitat modification in the intervening period. Citizen scientists are less likely to spend time and effort, and so their observations are more informal and comparatively unplanned. Thus, endemic and threatened species are underrepresented in citizen scientist records, even in relatively small fragments ().

Of the 83 endemics in the literature, 28 were not reported by citizen scientists. Several of these are commonly found in mixed and rain forests and probably found in semideciduous forests, such as the Helmeted Woodpecker Celeus galeatus, the Vinaceous-breasted Parrot Amazona vinacea, and the Bare-throated Bellbird Procnias nudicollis. For whatever reason, these conspicuous birds are not commonly seen (). Thus, greater and directed effort to find them would be required, and knowing whether the birds are absent or not detected cannot now be determined using WA.

Almost half of the 78 threatened species were also absent from CS data. Two of the conspicuous macaw species (Ara ararauna and A. chloropterus) are historical records and were reported only in the literature. Also, a woodcreeper, the Black-billed Scythebill Campylorhamphus falcularius, widespread in the Atlantic Forest, but uncommon in western Paraná, was reported only in traditional science in two municipalities, but never in WA (; ). Again, being absent or unseen presents a conservation challenge.

Recommendations

Cooperation between traditional and citizen sciences can improve data quality that will better inform conservation of the Atlantic Forest of western Paraná (; ). Recognizing that simple observations (as in photographs or recordings) are insufficient to estimate species richness is a beginning. Our first recommendation is to add rigor to WA by including additional information in their observations. This information begins with submitting lists of species observed at the time photographs were taken. Next, we recommend that the species list include an approximate time in which each was seen, and a start-stop time of the observation period. WA already provides location, but that could be more precise and include coordinates, unless justified to avoid problems identifying where threatened species may be found. Finally, WA might improve their forum to include topics in which traditional scientists can go to ask for the help of citizen scientists in finding particular species in particular places, and where citizen scientists might go to ask for help in understanding where and when to search for any of those desired species. With this additional information, WA has the potential to provide important information for science, conservation, and natural resource management (). For traditional scientists, we recommend that they also post their observations in WA and explain in their posts what and where they are studying.

By posting friendly challenges (such as Great Backyard Bird Count, Big Days, City Nature Challenge) to find and photograph rare, endemic, or threatened species, and find them in new places, WA might increase awareness and interest that stimulates citizen scientists to make special effort to accept those challenges. As lists grow and improve in each municipality, we may discover scientifically interesting species or novel distribution patterns (; ) and generate a positive feedback loop. Improved information will result in more realistic and justifiable conservation practices, such as where a corridor might be most effective.

Our recommendations are not time consuming, but with existing statistical methods, will permit bringing together these disparate sources of data in the analyses, making comparisons much more informative and predictions much more precise, both regionally and locally (; ; ; ). WA has the potential to become important as a role model to be followed in South America for the promotion of conservation of nature and of endangered species, similar to the Breeding Bird Survey (https://www.usgs.gov/centers/pwrc/science/north-american-breeding-bird-survey?qt-science_center_objects=0#qt-science_center_objects) or the Christmas Bird Count (https://www.audubon.org/conservation/science/christmas-bird-count).

Conclusions

Contributions by citizen scientists can grow in importance for traditional science and conservation. Thus, we recommend that contributory citizen scientists, and the platforms they use, include a small amount of additional information as we have described. We also recommend that traditional scientists communicate better with citizen scientists and encourage them to visit particular places, and to search for target species. The additional information provided will allow scientists to reliably estimate local and regional biodiversity and population trends, thereby alerting the conservation community more quickly and efficiently to problems.

Data Accessibility Statements

Data are available through the online sources described herein.

Supplementary Files

The supplementary files for this article can be found as follows:

Supplemental File 1: Table S1

Sources used in this study collected from the literature search, in chronological order by source type, with municipalities included in the study, in western Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s1

Supplemental File 2: Table S2

Number of species by municipality for published (including all sources) studies including 15 municipalities of western Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s2

Supplemental File 3: Table S3

Number of species by municipality for records from WikiAves in the 48 municipalities of western Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s3

Supplemental File 4: Table S4

Atlantic Forest bird species that are both endemic and threatened species reported in at least some of the 48 municipalities in western Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s4

Supplemental File 5: Table S5

Atlantic Forest endemic species reported from 48 municipalities in western Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s5

Supplemental File 6: Table S6

Threatened and Atlantic Forest non-endemic species in western Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s6

Supplemental File 7: Table S7

Non threatened and non-endemic species in western Paraná, southern Brazil. Literature (L), WikiAves (WA). DOI: https://doi.org/10.5334/cstp.349.s7

Supplemental File 8: Table S8

Bird species exclusively recorded only in one source, either literature (L) or WikiAves (WA), in the state of Paraná, southern Brazil. DOI: https://doi.org/10.5334/cstp.349.s8

Supplemental File 9: Figure S1

Endemic and potentially threatened species in western Paraná, southern Brazil. (a) Endemic species and (b) threatened (as we defined them here) species from the literature in bird species mentioned in the literature in 15 of the 50 municipalities and Atlantic Forest. (c) Endemic and (d) threatened bird species recorded by citizen scientists in 48 of the 50 municipalities. Gray areas indicate forest fragments. DOI: https://doi.org/10.5334/cstp.349.s9

Supplemental File 10: Figure S2

(a) The number total of species reported, (b) the number of endemic species and (c) the number of threatened species (as we defined them here), by municipality population size. Lines are intended to show trends, not significance. DOI: https://doi.org/10.5334/cstp.349.s10

Supplemental File 11

Supplementary References. DOI: https://doi.org/10.5334/cstp.349.s11