What does citizen science stand for?

Although there seems to be general agreement that citizen science refers to the participation of the general public in scientific processes, this participation manifests itself in highly varied ways (Bonney 1996; Irwin 1995). Other terms currently in use for similar activities include community science, participatory science, and participatory action research. These terms share distinctive but often tacit ideals of good scientific practice, and particularly ideals of participation. Schäfer and Kieslinger (2016) underline the heterogeneity of citizen science projects, which range from science-driven contributory projects to innovative participatory approaches and citizen-driven action projects. A helpful analysis of the terminology of citizen science and related concepts has been provided by Eitzel et al. (2017) who suggest using the term as broadly as possible, and including a whole set of related terms referring to science that involves the public. Strasser et al. (2019), however, express concern that citizen science is currently being used as a fashionable term that does not delineate itself clearly enough from other scientific practices and thus runs the risk of becoming an obscure label that could be applied to almost anything.

In an effort to develop a set of minimum criteria for good practice in citizen science, the ECSA has identified ten core principles (Robinson et al. 2018), which have been adopted by the United States and Australian Citizen Science associations as well. Although the ten principles provide a sound basis for good practice in citizen science, we as a platform needed assessable and explicit criteria to decide whether to list a project or not. Therefore, in 2017, we established a working group on quality criteria for citizen science projects listed on Österreich forscht.

Developing procedures to promote a joint understanding

Between March 2017 and March 2018, a working group within the CSNA, consisting of representatives from 17 institutions (see Heigl et al. 2018a for the detailed list of contributing institutions), developed criteria for the transparent evaluation of projects in the application phase for listing on Österreich forscht. The starting point of this collaborative process was based on existing classifications of citizen science projects (Cohn 2008; Haklay 2013; Sanz et al. 2014; Wiggins and Crowston 2011), the ECSA Ten Principles of Citizen Science (Robinson et al. 2018), and the Vienna Principles on Scholarly Communication (Kraker et al. 2016). From the outset, the intention was to make this process as open and transparent as possible (Figure 1).

All project leaders listed on Österreich forscht were given the option of joining the working group at any stage in the process, either by attending working group meetings in person or through online communication tools. In this multi-stage process, the conveners collected different forms of knowledge (e.g., scientific, experiential) from working group members as well as the feedback repeatedly provided by external experts from science and technology studies. During each stage of the process, detailed minutes were sent to all project leaders listed on Österreich forscht to get feedback on both the meeting minutes and the most recent version of the criteria. Additionally, throughout September 2017, the general public was invited to comment on the draft criteria through an online consultation process similar to a public opinion survey (Rowe and Frewer 2000) (Figure 1). The platform coordinators issued invitations in a press release, in social media advertisements, and on the Österreich forscht website in a dedicated public consultation section called “Diskutieren Sie mit!” (Join the discussion!). From previous media analysis of our channels, we know that the audience reached through these channels are citizens with a genuine interest in science. In addition, project leaders were invited via personal e-mails to give feedback. The online consultation resulted in 57 comments by 15 anonymous users on version 0.2 of the quality criteria catalogue and 6 e-mails from project leaders.

The combined approach of open working group meetings, their minutes, feedback loops involving project leaders, and the public consultation allowed us to merge different perspectives and research traditions. In particular, we discussed the nature and shape of public participation in scientific processes across disciplines as well as its value for practitioners (such as public authorities or conservation associations). The heterogeneous composition of the working group also challenged members to critically reflect on their traditions, convictions, and experiences as scientists and practitioners. For example, the notion of scientific rigor was the subject of several challenging discussions, and turned out to be understood differently in the different disciplines. In fact, the group was able to formulate the respective criteria only after reaching a common understanding of the research process in different academic fields (natural sciences, humanities, social sciences, and art sciences).

Version 1.0 of the quality criteria catalogue was published on Zenodo (https://zenodo.org/) in German and in English (Heigl et al. 2018a) and was presented at the 4th Austrian Citizen Science Conference in February 2018. The German citizen science platform hosted a workshop at the same conference to discuss the criteria from various perspectives. The results of the conference workshop were incorporated into version 1.1 of the catalogue and published in the Open Science Framework (Heigl et al. 2018b). Version 1.1 is the current version of the quality criteria catalogue. Since 1 February 2018, new projects wishing to be listed on Österreich forscht have been required to meet the criteria at the time of listing. Projects already listed on the platform were asked to adapt to meet the criteria within 16 months.

To ease the implementation of the new criteria, each criterion was converted into a question, i.e., a questionnaire was generated (Heigl et al. 2019). Guidelines, frequently asked questions (FAQs), and links (e.g., to Data Management Plan (DMP)–templates) were provided as additional support (Heigl et al. 2019). Following submission of the completed questionnaire, the platform coordinators consult with the working group, and in case of ambiguities, contact the project leaders for clarification and to provide any support needed. The aim of this process is to ensure a shared understanding of the characteristics of citizen science projects, and work jointly towards maintaining and improving this. An open dialogue and respectful interaction between all actors involved are prerequisites for this process.

Description and scientific background of the criteria catalogue

The current version (Version 1.1) of the catalogue consists of 20 criteria covering seven areas: (1) what is not citizen science; (2) scientific standards; (3) collaboration; (4) open science; (5) communication; (6) ethics; and (7) data management (Heigl et al. 2018b). While these criteria represent the outcome of the transdisciplinary co-creation process described above, current citizen science literature was also incorporated.

In the following section, we discuss each criterion in relation to existing experiences and literature (Table 1).

The first set of criteria (A–D) is aimed primarily at framing what the core of a citizen science project is (see Cohn 2008; Haklay 2013; Sanz et al. 2014; Wiggins and Crowston 2011). The working group opted for a negative list for this set (i.e., projects that are not citizen science), to retain as much openness as possible to a range of different concepts and disciplines. As a consequence, any project that is not excluded by these four criteria is considered potentially eligible for inclusion on Österreich forscht. Because this first set of criteria relates to the very core of a citizen science project, they are numbered separately (i.e., A–D versus 1–20 for the remaining parts of the catalogue).

The second set of criteria (1–3) refers to scientific standards as they apply to the different characteristics of the research conducted: How is scientific rigor defined and what can scientific rigor mean in different disciplines? Who defines the nature of scientific standards and what implications do these standards have for the participatory nature of citizen science, particularly beyond the natural sciences? These questions were discussed extensively in group meetings, initially producing a diverse range of notions of scientific standards across disciplines. Broadly informed by a Science and Technology Studies perspective (Felt et al. 2016), the working group members realized the contingent nature of standards in their own disciplines, and the performative nature of standards in allowing for some forms of knowledge to be produced but not others. This does not mean that anything goes, but indicates the need for a decentralized approach when it comes to scientific standards: Criteria 1 and 2 thus remain more flexible than the first set (A–D), allowing for hypotheses, research questions, or scientific goals to drive the research design of a project including field-specific methods. It is essential that new knowledge is generated by citizen science projects (criterion 3) even if the purpose is to confirm previous findings.

Collaboration between professional scientists and non-professional scientists (i.e., citizen scientists) is essential in most citizen science projects. As a consequence, the way in which this collaboration is organized is one key area in our criteria catalogue. To collaborate on a research project, people need to see the benefit of this collaboration (criterion 4) (Robinson et al. 2018; Tweddle et al. 2012).

Citizen science is frequently criticized as a way of gathering large amounts of data without financial compensation for participants (e.g., Lave 2012). However, citizen science may also generate insights that could never be achieved without the participation of citizen scientists. To address this aspect, we included criterion 5, which states that the project’s goals are not realistically achievable without the participation of citizen scientists. This ensures that the collaboration between scientists and citizens is an integral part of the project, if professional scientists play the lead role (as this could be otherwise).

Collaboration may take place at different stages in the research process. Citizen scientists may contribute to, or indeed lead, the formulation of research questions, methods design, the collection or interpretation of data, and the communication of results, or they may manage the whole project (Shirk et al. 2012). In criterion 6, we address the specific elements of the research process that allow for this participation. In fact, the goals and objectives of the project must be clearly understood by all parties (e.g., Kraker et al. 2016; Tulloch et al. 2013), allowing potential participants to make an informed decision about whether to participate or not. The clear formulation of the goals and objectives of the project is therefore addressed in criterion 7.

For collaboration to function smoothly, all those involved should be aware of their roles in the project (Newman et al. 2010). Criterion 8 therefore addresses the need to ensure that tasks are communicated clearly and in accessible language.

Because the data and results obtained through citizen science rely on voluntary collaboration, data and results should be openly available, except in cases where publication would lead to legal or ethical issues. This request is consistent with the fundamental demands of the open science movement (e.g., FAIR principles, Budapest Open Access Initiative, Berlin Declaration), funding requirements at the EU level (e.g., European Research Council 2017), the ECSA principles on citizen science (Robinson et al. 2018), and the Vienna Principles (Kraker et al. 2016). Three criteria (9, 10, and 11) therefore relate to open data, open access publication, and the transparent and accessible communication of results.

Communication tasks in citizen science projects do not exclusively address participating citizen scientists. We also request transparent communication to the general public. Any interested person should be able to contact a project delegate and find information about the project in an easily accessible and comprehensible manner (criteria 13 and 15; (Bonney et al. 2009; Newman et al. 2010). Participating citizen scientists need bi-directional communication; e.g., they need to receive feedback on the process and results, and they need to be able to interact with others in the project team (criterion 14) (Mackechnie et al. 2011). Finally, good communication in citizen science projects is also defined by targeted communication strategies and methods (Bonney et al. 2009; Pace et al. 2010). Project leaders need to address specific target groups (e.g., school children or senior citizens) in an appropriate manner (criterion 12).

Basic research ethics have become standard and are part of the majority of research funding programs today. In quality criteria 16, 17, 18, and 19, the ethical requirements are rooted in the Charter of Fundamental Rights of the European Union (European Parliament 2000) and in Responsible Research and Innovation (RRI) literature (e.g., Kupper et al. 2015) and practice (e.g., RRI Tools 2019). In addition, the importance of diversity and inclusion was a key finding in the co-creation process of the quality criteria. Because diversity and inclusion are not yet considered in standard scientific practice, these criteria help to raise awareness of an inclusive community and of barriers to participation.

Many research funders now require a data management plan (DMP) that describes how data will be handled during the research project—e.g., the Austrian Science Fund [FWF] (Rieck 2019) and the European Research Council (2017). The working group agreed that citizen science projects would benefit from a DMP, which has manifested in criterion 20.

Current status of adoption, implementation, and reactions

When the criteria were presented in February 2018, 64 projects were listed on Österreich forscht. When the criteria went into effect in June 2019, 15 projects were finalized and archived and 12 projects were newly listed, resulting in 61 active projects. Of these 61 projects, 39 managed to adapt to the criteria within the given timeframe; 13 were still in the process of adapting to the criteria; and 9 did not send a complete questionnaire for their projects. Those projects were therefore removed from the platform (Figure 2).

Figure 2 indicates that the criteria in general are applicable regardless of whether the project is led by a university, by an association, by a citizen, or by another entity. Some projects have still not been able to complete the implementation process, often because of a lack of time, as personal communication with project leaders revealed. Initial feedback from those who have applied the quality criteria, however, confirms that the process was feasible, although not always easy, to complete. Admittedly, the obligation to make a case for inclusion on the website can prove onerous for some and may create or further exacerbate inequalities of representation between different disciplines and practices. We want to emphasize that despite this hurdle, we have observed an increase in social science and humanities projects (as newer and more experimental forms of citizen science).

At the national level, the quality criteria have been critically reviewed by the working group since they were first established. Based on the experiences of the project leaders who were responsible for implementing the criteria, additional support resources have been put in place. These include explanatory guidelines, templates, and targeted workshops on how to implement the criteria, requirements, and recommendations. Furthermore, the process is being evaluated externally by a consultancy company that is not part of the CSNA to enable changes to the criteria and to the process on an objective basis. The evaluation process should also provide insights into why some projects did not implement the criteria.

Reflections on the process

We have herein described the collaborative development of a set of quality criteria for citizen science projects within the Austrian citizen science platform Österreich forscht. The quality criteria developed have been presented in the contexts of the relevant literature, of previously formulated principles of citizen science and open science, and of discussions within the working group. We now critically reflect on those criteria in light of their applications and of their impacts on perceptions of citizen science in the CSNA.

The contradiction between the very notion of citizen science, which is open to a range of different stakeholders and approaches, and the exclusionary nature of establishing quality criteria was a constant subject of discussion before and during the development process. Defining what constitutes citizen science by establishing a set of criteria may therefore seem a counterintuitive step. The application of such criteria could stifle new developments, especially if this is done in a top-down manner and without any regulatory process that can take new developments into account (Ottinger 2010). In the case of the citizen-led documentation of environmental degradation, for instance, bottom-up initiatives serve an important function in democratizing institutionalized science, and standardization may hamper such efforts to innovate (Ottinger 2010). Moreover, this could lead to a situation in which the criteria are mainly addressed by projects led by universities, and bottom-up initiatives may feel excluded by the complexity of the criteria catalogue. We found that this drawback can be overcome by including bottom-up initiatives in the process of developing such criteria. All stakeholders involved in the process should be treated as equals, regardless of their background. Everyone involved in our process of formulating the criteria for Österreich forscht shared the concerns described above and contributed to making the process as open as possible. We sought to actively engage non-university stakeholders in the development of the criteria, which led to the direct involvement of 12 non-university members (compared with five university members) in the working group, as well as other interested citizens who provided feedback during the online consultation. In hindsight, the anonymous collection of feedback from the general public could have been improved by collecting some demographic data, since this would have allowed us to better understand the comments made in context and to learn more about those who took an interest.

The development of quality criteria for citizen science projects and the possible exclusion of projects from Österreich forscht does, of course, pose risks: Excluded projects may not have access to the resources of the platform (e.g., an active community, public relations activities, or recruitment support). Moreover, the platform itself could suffer reputational costs if the quality criteria are so difficult to achieve that many projects are excluded. Indeed, developing quality criteria for citizen science projects implies a trade-off between openness and minimum standards (Ottinger 2010). For example, one of the goals of the criteria was to demarcate citizen science projects from purely educational or non-scientific participatory projects, which are equally important to society but usually have no research goal or question (Kasperowski et al. 2017). In our view, the distinction between citizen science and other forms of participatory projects can only be made provisionally, should always be made pragmatically, and be driven by transdisciplinary input in a bottom-up manner. To address these concerns, the set of quality criteria that we have presented is reviewed continuously based on the experience of applying them to new projects, and on feedback from project leaders, engaged citizens, and new initiatives in the field of citizen science. The working group will therefore evaluate and jointly reflect on the criteria on a regular basis to identify and correct any shortcomings.

As previously stated, citizen science as a whole has a threefold responsibility to citizen scientists, to the scientific community, and to decision makers (Hecker et al. 2018; Kasperowski et al. 2017): (1) citizen scientists must be able to trust that the time and effort they invest in a project serve a greater goal (Land-Zandstra et al. 2016), that their contributions are recognized (Rotman et al. 2012) and that their privacy is not misused in any way (Bowser et al. 2014); (2) the scientific community must have confidence that the results generated through citizen science projects are valid and reliable (Elliott and Rosenberg 2019; Guerrini et al. 2018; Hecker et al. 2018; Newman et al. 2012); and (3) policy-makers have to be sure that citizen science projects are both scientifically and ethically sound (Haklay 2014, 2020; Hecker et al. 2018; Kasperowski et al. 2017). These expectations can be met by applying quality criteria and continuously evaluating citizen science and the criteria applied.

When seeking to establish quality criteria for citizen science that are equally valid across all disciplines, interdisciplinarity itself is another obstacle. Any attempt to impose quality criteria would most likely favor certain disciplines at the expense of others. For example, conceptions of what constitutes a scientific approach in the natural sciences, humanities, social sciences, and art sciences will not necessarily be consistent (Bauer 1990; Lélé and Norgaard 2005). Whereas natural scientists strive for the reproducibility of results (the extent to which consistent results are obtained when an experiment is repeated), humanities scholars emphasize the singularity of cases and the inherent perspectivism of qualitative research (e.g., in cultural anthropology). Taking the obstacle of interdisciplinarity into account, the process of formulating quality criteria for Österreich forscht was designed to be open to all citizen science actors. The composition of the group enabled a respectful and wide-ranging discussion of matters such as different conceptions of scientific approaches, and this ultimately led to a consensus on the definition of minimum scientific standards.