Every public participation program results in a promise to the public regarding the level of their potential influence on the outcome of the project and what they can expect … [program developers should] make this promise clear and explicit so as to create common expectations among all stakeholders (EPA 2018).
Citizen science is a powerful approach for creating new scientific knowledge (Kullenberg and Kasperowski 2016). At the same time, it can be leveraged for science education, social engagement with science and the subject of study (Bonney et al. 2009), and raising awareness about environmental problems (Johnson et al. 2014). Also, if strategically designed and conducted, citizen science might empower participants to improve the stewardship of their communities’ natural resources (Vitos et al. 2013) and engage in policy processes for local improvements (Overdevest et al. 2004). However, the academic research community has expressed concerns about data quality, integrity, and confidentiality; recognition of intellectual property and authorship for volunteers; and doubts about conflicts of interest in citizen science (Nature Editorial 2015; Resnik et al. 2015).
As a democratizing approach in research, citizen science inevitably will encounter some resistance for recognition: The status quo of science governance is being challenged, while science processes and methods are adapted in innovative ways (Newman et al. 2012). However, identifying and addressing weaknesses in citizen science is important to ensure that its full potential can be realized.
Traditional academic research is not without its flaws, similarly facing challenges with integrity, conflicts of interest, and a crisis of trust (Saltelli and Funtowicz 2017). Citizen science provides research institutions with opportunities to rethink the ways they do science and connect with society to improve their positive social and ecological impact and increase trust in science.
The International Association for Public Participation (IAP2) provides a set of core values and a code of ethics to enhance any public participation process, notably, “We will carefully consider and accurately portray the public’s role in the decision-making process” (IAP2 n.d.a). During the design of a citizen science project, the degree and quality of public participation should be established in relation to desired outcomes and impacts. In some cases, involving the public in the early stages of design might be appropriate (Shirk et al. 2012).
Many classifications of citizen science projects exist to orient the type of design and the public’s role in the project, e.g., the spectrum of public participation by IAP2 (n.d.b), a ladder of citizen participation (Arnstein 1969), Haklay’s typology of participation (Haklay 2013), and models for public participation in scientific research (Shirk et al. 2012). These different sources coincide in some levels of the spectrum, for example, they identify: (1) contributory or distributed intelligence projects, where citizens are involved in data collection or analysis tasks; (2) participatory research projects, where citizens partner or collaborate with scientists in a wider range of the research process; and (3) co-creation projects, or “extreme” citizen science, where citizens are empowered, initiate, and control the project.
Taking into account the design of a project and the role of the public within it, project leaders can establish the promise made to the public. At the least, project leaders should commit to providing feedback on how project outcomes were influenced by the volunteers’ input (IAP2 n.d.a). If a project is of a more participatory nature, project leaders should commit to incorporating advice and recommendations from participants into the decisions, including designing how they participate, and should be transparent in recognizing and communicating the needs and interests of all participants (IAP2 n.d.a).
Based on our experiences with the projects CITI-SENSE, Mapping for Change, Making Sense, SalusCoop, and others—as well as literature about environmental and biomedical citizen science—we detected various challenges to appropriately developing and committing to a formal promise to the public. We also envisioned promising solutions. In particular, this article examines the challenges and proposes solutions regarding:
We limit the scope of the promise to the public to levels of participation of volunteers and their decision-making power. Other facets of the promise, such as overstating scientific outcomes (e.g., promises about finding a cure or advancing science), or overstating policy outcomes (e.g., forcing the government to take measures to improve air quality), deserve their own article.
Reflecting on the experience of project CITI-SENSE (https://social.citi-sense.eu/), which involved volunteers evaluating air quality with low-cost sensors, a facilitator of the project expressed that:
[Citizens] wanted to receive the air quality data directly. However, only the researchers were receiving the sensor data. The format of the data was only understandable for specialized technicians. On top of that, the data were not reliable nor comparable with the official values. For these reasons, we could not provide air quality information to volunteers and, later, we had to manage their high expectations (Personal communication in Spanish, 15th of February, 2018).
The role of the public in CITI-SENSE should have been adequately defined in relation to planned outcomes (Shirk et al. 2012). In this case, albeit in good faith on behalf of the project leaders, the outcomes were not clear because they changed during the project’s life. This situation is not uncommon in scientific projects and that is why Shirk et al. (2012) propose to conduct a continuous evaluation of evolving interests that might not be known or recognized at the onset of the project. For example, as CITI-SENSE progressed, researchers’ interests changed from wanting to gather data for generating a detailed map of air quality in a city to needing to gather data for testing the actual utility of the sensor pods. This change in outcomes and interests was not adequately transmitted to the volunteers; researchers should have explained that they were evaluating the sensor pods before embarking upon actual air quality evaluation.
Alternatively, they could have invited public participation at the early stages of the project for designing the tools for air quality evaluation, as the Technical Advisory Group noted: “Not enough citizen focus in the development of tools, it’s coming late.” (See supplemental material for notes from CITI-SENSE annual consortium meeting, March 2015).
Shirk et al. (2012) proposed a framework to design citizen science projects in which one of the first questions asked is “whose interests are being served?” They add that the hopes, desires, goals, and expectations of the public and the scientific community should be taken into account and “balanced.” Wiggins and Crowston (2012) conducted an analysis of the goals identified in citizen science projects to develop a typology; i.e., a scientific goal, education and outreach, action, monitoring, and conservation are some of the goals they included. However, the critical question of how to balance diverse interests and goals can be addressed only after evaluating the expectations of stakeholders and their needs (supplemental materials 2015: 7).
For example, in project CITI-SENSE we found a mix of goals and concerns. Project facilitators and teachers involving school students had an education and outreach goal, but the schools were concerned about their own image if they would find a pollution problem (supplemental materials 2015: 3). Meanwhile, other research institutions involved in the project had a public health goal. This example serves to illustrate the need for establishing an engagement plan before the onset of the project, involving stakeholders in the evaluation of their needs and expectations and in the design of the project, and creating a two-way feedback channel between project leaders and participants that can help in the continuous evaluation of dynamic interests (Serrano Sanz et al. 2015; Shirk et al. 2012).
Shirk et al. reflect on the “the rhetoric of participation”: “simply invoking the language and ideals of participation is insufficient.” Instead, they call for “careful, intentional, and transparent employment of participation strategies to achieve targeted outcomes” (Shirk et al. 2012: 3), which they then classify in outcomes for science, for individual participants, and for social-ecological systems. In the case of participatory and co-creation projects, leaders and professional scientists should go beyond the idea of volunteer participation and establish a collaboration or partnership relationship with citizen scientists, balancing power relations. The role of the facilitator is critical in establishing, balancing, and maintaining this collaboration relationship.
Additionally, the institutions involved should have a commitment to their citizen collaborators; provide them with timely, frequent, actionable feedback; and communicate how their input affected decisions and outcomes (IAP2 n.d.a). Here, the facilitator can act as a two-way feedback channel between professionals in research institutions and citizen scientists.
Furthermore, the institutional economic and logistical support to the facilitator, and providing decision-making power to the facilitator, are fundamental to ensure long-term trust relationships, motivation, and engagement (Serrano Sanz et al. 2015).
A good facilitation model is Mapping for Change (http://mappingforchange.org.uk/) in London. This social enterprise, partly owned by University College London, was created in 2010 with the objective of providing a suite of mapping tools to help communities engage with local issues such as noise or air pollution (UCL News 2010). Since then, Mapping for Change has facilitated many citizen science projects, helping local communities to partner with UCL researchers and providing tools and training (City of London 2017).
The example of Mapping for Change becoming a social enterprise contrasts with funded academic research projects, like CITI-SENSE, where the facilitator’s contract is terminated once funding has finished. In such cases, the research institution loses a “community champion” (Balestrini et al. 2017; Balestrini et al. 2015) who is perceived as trustworthy by the local community and citizen collaborators, as well as the social capital that this person has generated. The latter is a lost opportunity for having a positive social impact.
In addition to Solutions 1 and 2, we also propose to make the promise to the public explicit and in writing, and to disseminate it for transparency. Transparency is the first step for informed participation, which we examine below.
In particular, this promise should include the expected outcomes whether or not they come from a participatory process. We also recommend making the expectations of different stakeholders explicit. In this manner, project leaders will be transparent about potential conflicts of interest. Furthermore, an explicit promise could help in settling conflicting interests.
Very importantly, the role of the public in the project should be clearly stated, as should a commitment to inform participants about how their input influenced outcomes. Additionally, the timeline of the project and the period of the institutional commitment should be clearly communicated, to avoid deception when a facilitator disengages and communications stop. Ideally, a sustainability plan should seek to prevent this situation (Taylor et al. 2013).
Finally, the stakeholders should be invited to evaluate the project’s commitment to the promise for accountability. The stakeholders include professional researchers, participants, and organizations involved in the citizen science project and directly affected by it.
Our proposed steps for designing a citizen science project with ethics at its core, preparing the promise to the public, and ensuring commitment to the promise, are summarized in Figure 1.
Citizen science projects often require the collection of personal data or data that could lead to re-identification. For instance, geolocalized data are important for environmental monitoring and epidemiology. Even when direct identifiers such as names and addresses are avoided, geolocalized data or metadata contained in pictures and other digital pieces of information could pose data protection and privacy risks.
The project Making Sense (http://making-sense.eu/) empowers communities with methods and tools for environmental monitoring. It enables citizens to capture data, make sense of those data, and work towards positive change. The pilot carried out in Barcelona involved a group of neighbors asked to measure how high noise levels are and what can be done to improve the situation in a specific area (a public square). The Smart Citizen Kit was used by neighbors to collect data about the noise levels coming from the square, which were publicly shown as live feeds in the online platform, smartcitizen.me. The platform showed the level of noise of each individual sensor, some of which were positioned inside the houses of participants, in their bedrooms or living rooms. This led some participants to raise concerns about privacy issues, reporting that they were not always comfortable about sharing their data and would rather disconnect their sensor at times. To address this concern, co-creation workshops were held with participants and designers to discuss data governance tensions and envision new features that would empower them to have better control over their data.
This example illustrates how addressing privacy concerns is crucial, not only to ensure legality and data protection compliance, but also to ensure the willingness of participants to contribute data to citizen science projects, and to trust their proponents and results.
Galdon Clavell assessed the privacy and data protection aspects of three different citizen science projects. In a conference presentation cited by Wiggins (2016), Galdon Clavell identified a series of problems and provided a set of recommendations for promoters and developers of citizen science projects, shown in Table 1.
|Privacy policies do not always reflect data practices.||Create transparency, accountability, and audit mechanisms, allowing others to verify that the stated policies are a clear reflection of actual data policies.|
|Privacy risks of metadata and third party sharing are not taken into account.||Determine what data can be released and under which conditions (anonymization).|
|Basic privacy precautions such as data minimization are not implemented consistently.||Require only minimal personal information about project participants, give sufficient notice of privacy options, provide users the option to hide some of their data, and allow citizen users the possibility to modify and delete their data.|
To incorporate privacy and data protection precautions in citizen science projects, developers of technological innovations should not only consider co-designing privacy with citizen collaborators, but also should consider Privacy by Design (PbD) principles and configure them based on the Privacy by Default parameters. PbD puts the privacy of users first, promoting its integration into the design of technologies (in a broad sense: Hardware, software, network designs). It implies proposing by default the most restrictive privacy options in a way that reinforces the active choice of the user when it comes to sharing data.
The 7 foundational principles of PbD (Cavoukian 2009) are useful to achieve this in practical terms:
Cheung (2017) describes the main challenges of informed consent in biomedical citizen science: The vast amounts of data collected, aggregated, or repurposed imply a degree of uncertainty about the outcomes, which could evolve over time. In addition, the one-to-one model of informed consent does not fit the networked structure of citizen science collaborations. New models of informed consent have been proposed to overcome these challenges: Open consent, which openly excludes privacy (Angrist 2009; Cheung 2017); dynamic consent, where each participant selects through an online platform which data she wants to donate and under what conditions (see next section); portable legal consent, which is not linked to a particular institution but to the participant and her data; and meta-consent, where participants can choose from a set of types of research and types of data to share (Cheung 2017).
However, all these new models of informed consent require a high level of information literacy (Cheung 2017). The process of understanding this complex information and concomitant privacy risks could be addressed with innovative educational materials, the use of new media, and giving participants time for their own personal research on the Internet (Henry et al. 2009). Also, Welch et al. (2016) propose the use of “teleconsent,” where the researcher uses remote video conferencing for informing participants and answering their questions. Very importantly, a risk visualization tool might facilitate the comprehension of risks for privacy or unforeseen consequences (Arthur et al. 2015). Finally, Bowser and Wiggins (2015) propose to design citizen science data platforms that support evolving choices of participants that might change with context or location. We will examine further the idea of dynamic consent with the lens of data governance below.
For projects that involve fewer participants, it is viable to use focus groups for education about the topic and for building a trusting and collaborative relationship between participants and researchers (Skinner et al. 2015). However, in co-creation or “extreme” citizen science projects, the boundaries between professional researchers and citizen collaborators become blurry (Banks et al. 2013). In such cases, establishing a research collaboration agreement might be more appropriate than an informed consent process.
New initiatives are pushing the frontiers of participation in citizen science forward by opening up a new dimension of participation that relates to the governance of research data. This is particularly relevant for the biomedical sector, where a large amount of personal data is needed to make advances in research. This need has fostered the creation of a wide range of initiatives from private, public, and academic sectors which aim to encourage citizens to make their health data available by praising their involvement in research for the public good. However, the way in which citizen participation is conceptualized varies significantly, as do the roles that citizens could play in the value chain of research (e.g., data collection and sharing, or instigation of the research agenda).
On one hand, many initiatives promoted by private companies focus on providing citizens with products or services that generate new data. Such data are being used in research studies aimed at creating new patents, products, or drugs from which the company derives a financial profit. This occurs, for instance, on patient platforms such as PatientsLikeMe, or genomic analysis services such as 23andMe, where user-generated data are shared with pharmaceutical companies to conduct research studies. Although these initiatives provide services of value for patients that have proactively consented to donate their data for research, the lack of transparency about the researchers’ and company’s intentions to commercialize and patent the results achieved through users’ participation have stirred up controversies (Sterckx et al. 2012).
On the other hand, governmental research programs (e.g., Care.data in UK, Visc+ in Catalonia) tend to leverage the civic duty of citizens to participate in public research (Woolley et al. 2016). The lack of transparency (e.g., who can gain access to citizens’ data) and the fact that these programs often are based on opt-out models of participation, have led to a cessation of such programs. Strong concerns were raised about the lack of compliance with the conditions of social license for research (i.e., voluntary participation, governance by values of reciprocity, non-exploitation, and service to the public good) (Carter et al. 2015).
Whether by leveraging a rhetoric of altruism or the appeal of benefits offered through services or products, citizen participation in medical research initiatives often is limited to data donation. The whole value chain remains under the control of professional scientists, and no opportunities are given to citizens to set research priorities and agendas or to have a voice about how research outcomes are exploited. Citizen control of such matters is especially crucial in the health domain, where both personal risks and collective benefits are high, and where competing interests must be aligned.
To tackle such ethical issues, new initiatives are looking at data governance as a means to enable citizen participation in the governance of science. The aim is to flip the traditional partnership relationship of citizen science from looking at citizens as collaborators in research projects (mainly written and run by professional researchers) to looking at them as the real instigators of the studies, as they supply the capital assets (in the form of data) to support research that they endorse (Evans 2016).
In the era of big data, where huge datasets are processed, groups of citizens can have greater control over their data than individuals acting alone. That is why collective governance models are strongly advocated and proposed as a way to overcome the pitfalls of individual solutions and the one-to-one model of informed consent. An example of collective governance is the EPPI model (Buyx et al. 2017). Conceived for the epidemiology field (and not yet implemented), EPPI citizens are envisioned as being part of the governing bodies of research, and they collectively exercise control over the data, as well as propose and deliberate on research questions and concerns. Other initiatives have chosen the cooperative model as a legal form to enact such collective governance. Healthbank, MiData.coop (Hafen 2014), and SalusCoop (SalusCoop 2016) are examples of data cooperatives. Their aim is to aggregate health data from cooperative members and to put these data at the service of research projects on which they have deliberated. Moreover, members of HealthBank can exercise the right to an economic reward from researchers who access their data.
Such models of citizen direct governance are still in their infancy, and have not yet been fully deployed. There is need to understand better how a model of this type could be enacted practically and at scale, and what challenges it might face.
To contribute towards filling this gap, and to support the implementation of SalusCoop in Catalonia (Spain), we have conducted desk research and interviews with key actors in the healthcare sector in Barcelona. A snowball sampling approach was followed to determine the sample group of interviewees. The selected sample was composed of 24 people, including the public; members of patients’ associations; professionals from private and public hospitals and research institutions; bioethics researchers; and companies working in the field of open data and bioinformatics. The study has been published as a public report (SalusCoop 2016). Here we summarize the main results as an example of potential solutions for the identified challenge of citizen data governance.
The results of that study highlighted four fundamental governance principles that we propose as driving values for the implementation of “citizen governance models for health data science”:
We argue that any citizen science project should start with a formal and well-disseminated promise to the public to enhance the public participation process and ensure transparency, accountability, and trust. We identify the challenges and propose solutions to appropriately develop and commit to this promise. First, we recommend evaluating and balancing stakeholders’ expectations while defining the outcomes of the project. Second, the project leaders should plan a continuous evaluation process to inform participants about evolving interests and adapt participation strategies to it. Third, the promise to the public should clarify the role of the public and stakeholders in the project. Fourth, to ensure accountability, we strongly recommend communicating how participant input affects outcomes and decisions. Fifth, we suggest that the promise to the public also could address privacy concerns and include a data governance model.
Citizen science projects often require the collection of personal data or data that could lead to re-identification. To safeguard participants’ privacy, we recommend requiring only minimal personal information about project participants, determining what data can be released and under which conditions (anonymization), and creating transparency, accountability, and audit mechanisms. In line with these recommendations, we propose following the principles of Privacy by Design.
The process of understanding complex biomedical information and risks for privacy constitutes a challenge for informed consent. To overcome this challenge, we suggest some novel approaches and tools to educate citizens and explain the risks of participation. Also, we explain the concept of dynamic consent, both with the lens of privacy and with the lens of data governance.
Finally, we propose solutions for data governance in citizen science, such as conditional donation of data and flexible and transparent collective governance. In future work, we would like to examine the obstacles that citizen science projects encounter when trying to implement these recommendations, for example, small staff devoted to citizen science projects, technical challenges, and unforeseen conflicts.
Citizen science practitioners have the opportunity to diminish the ethical doubts of the academic research community by setting an example with their projects. Most importantly, to fulfill the promise of citizen science empowering people and gaining trust in science, we need to design citizen science projects with ethics at their core. We hope that the ideas we propose in this article will help in this endeavor.
The authors would like to express their gratitude to all the citizens who have contributed to the lessons presented in this paper by taking part in the projects CITI-SENSE, Mapping for Change, Making Sense, and SalusCoop. Also, the authors would like to thank the community champions, technicians, and researchers who have shared their experiences, enriching the lessons presented in this paper with their true stories. Finally, the authors would like to thank the editors and reviewers for their help in improving this article with their comments and revisions. This article benefited from the EU Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant (award 656439), and from the Spanish Ministry of Economy and Competitiveness, under Grant PTQ 16-08682.
The authors have no competing interests to declare.
Angrist, M. 2009. Eyes Wide Open: The Personal Genome Project, Citizen Science and Veracity in Informed Consent. Personalized Medicine, 6(6): 691–699. DOI: https://doi.org/10.2217/pme.09.48
Arnstein, SR. 1969. A Ladder of Citizen Participation. Journal of the American Planning Association, 35(4): 216–224. DOI: https://doi.org/10.1080/01944366908977225
Arthur, T, Phillips, A, Franks, Z and Pearce, S. 2015. Redesigning Informed Consent: Enhancing Risk Communication Through a Novel Risk Visualization Tool. Journal of Current Surgery, 5(2–3): 160–164. DOI: https://doi.org/10.14740/jcs275w
Balestrini, M, Diez, T, Marshall, P, Gluhak, A and Rogers, Y. 2015. IoT Community Technologies: Leaving Users to Their Own Devices or Orchestration of Engagement? European Alliance for Innovation Endorsed Transactions on Internet of Things, 1(1): e7. DOI: https://doi.org/10.4108/eai.26-10-2015.150601
Balestrini, M, Rogers, Y, Hassan, C, Creus, J, King, M and Marshall, P. 2017. A City in Common: A Framework to Orchestrate Large-Scale Citizen Engagement Around Urban Issues. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2282–2294. Association for Computing Machinery, New York. DOI: https://doi.org/10.1145/3025453.3025915
Banks, S, Armstrong, A, Carter, K, Graham, H, Hayward, P, Henry, A, Holland, T, Holmes, C, Lee, A, McNulty, A, Moore, N, Nayling, N, Stokoe, A and Strachan, A. 2013. Everyday Ethics in Community-Based Participatory Research. Contemporary Social Science, 8(3): 263–277. DOI: https://doi.org/10.1080/21582041.2013.769618
Bonney, R, Cooper, CB, Dickinson, J, Kelling, S, Phillips, T, Rosenberg, KV and Shirk, J. 2009. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience, 59(11): 977–984. DOI: https://doi.org/10.1525/bio.2009.59.11.9
Bowser, A and Wiggins, A. 2015. Privacy in Participatory Research: Advancing Policy to Support Human Computation. Human Computation, 2(1): 19–44. DOI: https://doi.org/10.15346/hc.v2i1.3
Buyx, A, Del Savio, L, Prainsack, B and Voelzke, H. 2017. Every Participant is a PI: Citizen Science and Participatory Governance in Population Studies. International Journal of Epidemiology, 46(2): 377–384. DOI: https://doi.org/10.1093/ije/dyw204
Carter, P, Laurie, GT and Dixon-Woods, M. 2015. The Social Licence for Research: Why Care.data Ran into Trouble. Journal of Medical Ethics, 41(5): 404–409. DOI: https://doi.org/10.1136/medethics-2014-102374
Cavoukian, A. 2009. Privacy by Design: The 7 Foundational Principles, August 2009. Available at: https://www.ipc.on.ca/wp-content/uploads/Resources/7foundationalprinciples.pdf [accessed 15 August 2018].
Cheung, ASY. 2017. Moving Beyond Consent for Citizen Science in Big Data Health Research. Research Paper no. 2017/006, University of Hong Kong Faculty of Law. DOI: https://doi.org/10.2139/ssrn.2943185
CITI-SENSE project website. Available at: https://social.citi-sense.eu/ [accessed 15 August 2018].
City of London. 2017. Science in the City. London. First published on 13 October 2013, last modified on 29 September 2017. Available at: https://www.cityoflondon.gov.uk/business/environmental-health/environmental-protection/air-quality/Pages/Air-Quality-Champions.aspx [accessed 1 May 2018].
Evans, BJ. 2016. Barbarians at the Gate: Consumer-Driven Health Data Commons and the Transformation of Citizen Science. American Journal of Law & Medicine, 42(4): 651–685. DOI: https://doi.org/10.1177/0098858817700245
Hafen, E, Kossmann, D and Brand, A. 2014. Health Data Cooperatives—Citizen Empowerment. Methods of Information in Medicine, 53(2): 82–86. DOI: https://doi.org/10.3414/ME13-02-0051
Haklay, M. 2013. Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation. In: Sui, D, Elwood, S and Goodchild, M (eds.), Crowdsourcing Geographic Knowledge, 105–122. Dordrecht: Springer. DOI: https://doi.org/10.1007/978-94-007-4587-2_7
Henry, J, Palmer, BW, Palinkas, L, Glorioso, DK, Caligiuri, MP and Jeste, DV. 2009. Reformed Consent: Adapting to New Media and Research Participant Preferences. IRB: Ethics and Human Research, 3(2): 1–8. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2743865/ [accessed 1 May 2018].
International Association for Public Participation. n.d.a. The IAP2 Federation–The P2 Pillars. Available at: https://c.ymcdn.com/sites/iap2.site-ym.com/resource/resmgr/files/IAP2_Federation_-_P2_Pillars.pdf [accessed 1 May 2018].
International Association for Public Participation. n.d.b. IAP2’s Spectrum of Public Participation. Available at: http://c.ymcdn.com/sites/www.iap2.org/resource/resmgr/foundations_course/IAP2_P2_Spectrum_FINAL.pdf [accessed 1 May 2018].
Johnson, MF, Hannah, C, Acton, L, Popovici, R, Karanth, KK and Weinthal, E. 2014. Network Environmentalism: Citizen Scientists as Agents for Environmental Advocacy. Global Environmental Change, 29: 235–245. DOI: https://doi.org/10.1016/j.gloenvcha.2014.10.006
Kullenberg, C and Kasperowski, D. 2016. What Is Citizen Science?–A Scientometric Meta-Analysis. PLoS ONE, 11(1): e0147152. DOI: https://doi.org/10.1371/journal.pone.0147152
Making Sense Project. Available at: http://making-sense.eu/ [accessed 24 August 2018].
Mapping for Change. Available at: http://mappingforchange.org.uk/ [accessed 15 August 2018].
Nature Editorial. 2015. Rise of the Citizen Scientist. Nature, 524(7565): 265. August 2015. DOI: https://doi.org/10.1038/524265a
Newman, G, Wiggins, A, Crall, A, Graham, E, Newman, S and Crowston, K. 2012. The Future of Citizens Science: Emerging Technologies and Shifting Paradigms. Frontiers in Ecology and the Environment, 10(6): 298–304. DOI: https://doi.org/10.1890/110294
Overdevest, C, Huyck Orr, C and Stepenuck, K. 2004. Volunteer Stream Monitoring and Local Participation in Natural Resources Issues. Human Ecology Review, 1(2): 177–185. Available at: http://www.jstor.org/stable/24707677 [accessed 1 May 2018].
Resnik, DB, Elliott, KC and Miller, AK. 2015. A Framework for Addressing Ethical Issues in Citizen Science. Environmental Science & Policy, 54: 475–481. DOI: https://doi.org/10.1016/j.envsci.2015.05.008
Saltelli, A and Funtowicz, S. 2017. What is Science’s Crisis Really About? Futures, 91: 5–11. DOI: https://doi.org/10.1016/j.futures.2017.05.010
SalusCoop. 2016. Towards Citizen Governance and Management of Health Data. Barcelona. Available at: http://saluscoop.org [accessed 1 May 2018].
Serrano Sanz, F, Holocher-Ertl, T, Kieslinger, B, Sanz García, F and Silva, CG. 2015. White Paper on Citizen Science. Socientize project. Available at: http://www.socientize.eu/sites/default/files/white-paper_0.pdf [accessed 1 May 2018].
Shirk, JL, Ballard, HL, Wilderman, CC, Phillips, T, Wiggins, A, Jordan, R, McCallie, E, Minarchek, M, Lewenstein, BV, Krasny, ME and Bonney, R. 2012. Public Participation in Scientific Research: A Framework for Deliberate Design. Ecology and Society, 17(2): art. 29. DOI: https://doi.org/10.5751/ES-04705-170229
Skinner, HG, Calancie, L, Vu, MB, Garcia, B, DeMarco, M, Patterson, C, Ammerman, A and Schisler, JC. 2015. Using Community-Based Participatory Research Principles to Develop More Understandable Recruitment and Informed Consent Documents in Genomic Research. PLoS ONE, 10(5): e0125466. DOI: https://doi.org/10.1371/journal.pone.0125466
Smart Citizen Project. Available at: https://smartcitizen.me/ [accessed 24 August 2018].
Sterckx, S, Cockbain, J, Howard, H, Huys, I and Borry, P. 2012. Trust Is Not Something You Can Reclaim Easily: Patenting in the Field of Direct-to-Consumer Genetic Testing. Genetics in Medicine, 15(5): 382–387. DOI: https://doi.org/10.1038/gim.2012.143
Taylor, N, Cheverst, K, Wright, P and Olivier, P. 2013. Leaving the Wild: Lessons from Community Technology Handovers. Proceedings of the 2013 CHI Conference on Human Factors in Computing Systems, 1549–1558. Association for Computing Machinery, New York. DOI: https://doi.org/10.1145/2470654.2466206
UCL News. 2010. Mapping for Change: New Social Enterprise to Support Sustainable Communities. University College London News, 1 July 2010. Available at: http://www.ucl.ac.uk/news/news-articles/1007/10070101 [accessed 1 May 2018].
US Environmental Protection Agency. 2018. Public Participation Guide: Selecting the Right Level of Public Participation. 22 February 2018. Available at: https://www.epa.gov/international-cooperation/public-participation-guide-selecting-right-level-public-participation [accessed 1 May 2018].
Vitos, M, Stevens, M, Lewis, J and Haklay, M. 2013. Making Local Knowledge Matter: Supporting Non-Literate People to Monitor Poaching in Congo. Proceedings of the Third Annual Symposium on Computing for Development, Association for Computing Machinery, Bangalore, India. DOI: https://doi.org/10.1145/2442882.2442884
Welch, BM, Marshall, E, Qanungo, S, Aziz, A, Laken, M, Lenert, L and Obeid, J. 2016. Teleconsent: A Novel Approach to Obtain Informed Consent for Research. Contemporary Clinical Trials Communications, 3: 74–79. DOI: https://doi.org/10.1016/j.conctc.2016.03.002
Wiggins, A. 2016. Citizen Science: Beyond the Laboratory @ 4S/EASST 2016, blog post, 3 September 2016. Available at: http://andreawiggins.com/citizen-science-beyond-the-laboratory/ [accessed 1 May 2018].
Wiggins, A and Crowston, K. 2012. Goals and Tasks: Two Typologies of Citizen Science Projects. Proceedings of the 2012 45th Hawaii International Conference on System Sciences (HICSS), 3426–3435. IEEE, Maui. DOI: https://doi.org/10.1109/HICSS.2012.295
Woolley, JP, McGowan, ML, Teare, HJA, Coathup, V, Fishman, JR, Settersten, RA, Sterckx, S, Kaye, J and Juengst, ET. 2016. Citizen Science or Scientific Citizenship? Disentangling the Uses of Public Engagement Rhetoric in National Research Initiatives. BMC Medical Ethics, 1(1): art. 17. DOI: https://doi.org/10.1186/s12910-016-0117-1