The Open Innovation in Science Research Conference and Citizen Science as an Open Innovation in Science Practice
The annual Open Innovation in Science (OIS) Research Conference is co-organized by a group of international scholars. Past conferences were held in 2019 (Vienna; in-person), 2021 (online), 2022 (CERN Geneva; hybrid), and 2023 (Vienna; hybrid). The OIS concept covers a broad range of open and collaborative practices such as open data sharing and reusing, industry-academia collaborations, as well as the involvement of citizens and crowds (). The conference fosters discussions across disciplinary boundaries (e.g., social sciences and natural sciences) and brings together scholars as well as OIS practitioners.
Citizen science is an important specific OIS practice and features prominently at the conference because it entails two types of openness (). First, it opens up the boundaries of scientific institutions by enabling non-professionals to participate actively in research activities (; ; ). Second, many citizen science projects strive to make their data as well as intermediate and final results openly accessible (; ). The OIS conference has developed into a vibrant place to discuss the benefits as well as challenges resulting from this high degree of openness, and to derive implications for future research and practice.
The diversity of conference participants, however, sometimes results in confusion around terminology: While participants coming from fields of the natural sciences or science and technology studies usually refer to “citizen science,” scholars from the fields of management and economics often use “crowd science.” Interestingly, the projects studied by these two communities are largely the same, including cases such as eBird, Zooniverse, iNaturalist, and Foldit. As such, these communities do not primarily cover different phenomena but see the same phenomenon through different lenses (): On the one hand, scholars in the citizen science tradition focus on the fact that many project participants are not professional scientists, which has important implications for the knowledge participants bring to bear, the motivations that drive their efforts, or the organizational mechanisms required to coordinate the collaboration between professional scientists and citizens (; ; ).
Scholars using the crowd science lens, on the other hand, focus on the fact that projects typically recruit participants via open calls for participation. This open recruiting has implications for the breadth and diversity of contributions but also for the organizational mechanisms needed to broadcast calls for contributions, to coordinate distributed work, and to integrate individuals’ contributions (; ). Scholars using the crowd science lens often draw on the well-established literature on crowdsourcing within the field of management (; ,). Note that the term crowdsourcing has also been used in the citizen science literature to describe projects involving micro tasks (); the management literature studies crowdsourcing as a much more general mechanism that can also help solve complex challenges such as developing new drug compounds or methods to clean up oceans after an oil spill ().
Here, we report on four session formats that covered citizen science and crowd science at the 2023 OIS Research Conference and that highlighted the increasing diversity of projects. Moreover, we present a conceptual framework of crowd paradigms and an associated tool that were developed in response to discussions (and confusions) at the 2023 and prior conferences. This tool leverages insights from crowdsourcing and related literatures to better understand how citizen involvement in specific projects can advance science. We illustrate the tool using cases presented at the 2023 conference and include a template readers can use to analyze other existing or planned citizen science projects. Given our focus on how insights from the crowd science literature can inform research and practice on citizen science, we primarily use the term citizen science when discussing projects. We acknowledge important discussions regarding the suitability of the term citizen science (); we use it in an inclusive sense regardless of participants’ legal citizenship.
Four Session Formats Covering Citizen Science at the 2023 Conference
The 2023 conference had 179 participants (66 in person; 113 online) and included four session formats covering citizen science: a panel debate, case sessions showcasing leading-edge applications of citizen science, an OIS experiment, and a session with research paper presentations. A guiding question for 2023 was whether OIS approaches such as citizen involvement can foster both scientific and societal impact, or whether achieving greater societal impact may compromise rigorous research. Next, we outline the different sessions and the citizen science cases discussed in each. We then describe a confusion that arose as participants reflected on the increasing diversity of projects.
Panel debate
Every OIS Research Conference includes a panel debate. The 2023 debate challenged participants to think about whether and how various OIS practices may create —or resolve—tensions between scientific impact and broader societal benefits of research. Four panelists discussed a range of practices including open data and citizen science.
One panelist described an initiative where citizen science beach litter data have been integrated into the official statistics of Ghana, enabling the tracking of the related Sustainable Development Goals (SDGs) indicator on plastic debris density (). According to the panelist, these data are useful for both research and policy makers, illustrating that citizen science can simultaneously foster both scientific and societal impact. Yet, other panelists and audience members raised concerns about conflicting standards of data quality required for scientific publications versus practical applications. The lively discussion showed that more research is needed on how to design projects to generate synergies between different goals, and on integrating citizen science practices and data into the broader scientific system, official statistics, and policymaking.
Case application sessions
The conference includes parallel sessions in which citizen science practitioners showcase OIS applications. These cases make abstract concepts tangible, challenge scholars to apply theories in practice, and allow practitioners to help shape future research. The cases presented in 2023 included four citizen science projects. First, leaders of the Tiny Forests project discussed how enhanced organizational infrastructure expanded data collection to more than 150 Miyawaki forests across the UK. As a result, the project can generate better data on ecosystem service provisions in different regions while also creating learning effects for citizens. The second case, Co-creation with Vulnerable Groups, engaged the elderly and people with disabilities in the development of eHealth solutions for aging in place. This process enabled the solution to address needs overlooked by prior research and to increase the subsequent uptake of the solutions. Third, organizers of the Care for ME/CFS project discussed how they engaged patients suffering from Chronic Fatigue Syndrome (CFS) to develop an online questionnaire-based tool that enables a faster diagnosis and better medical care. This project also increased public awareness of CFS and generated data for future scientific research. Finally, the Picture Pile Platform addresses societal challenges through a combination of citizen science and Earth Observation approaches. The platform allows citizens to classify images from, among others, satellites and drones to generate data related to issues such as deforestation, post-disaster damage assessment, and poverty mapping ().
These examples helped conference participants discover new opportunities and challenges related to citizen involvement. Perhaps more importantly, these cases illustrate the growing diversity of the citizen science landscape with respect to participants’ contributions and to the scientific and societal impacts projects seek to generate.
Open Innovation in Science experiment
Every year, the conference organizers challenge participants to “walk the talk” by experimenting with different OIS practices. In 2023, the organizers selected abstracts of pairs of published research articles that covered the same topic but were produced either with or without the use of OIS practices. The organizers then assigned these articles to different groups of in-person participants, including both OIS scholars and practitioners. These groups of roughly eight participants each were asked to do background research on the scientific and societal impact of these articles and to discuss potential linkages between OIS practices and different types of impact.
Two pairs of articles were related to citizen science. One focused on RNA design, whereby one study was performed by a traditional academic lab and the other by a large-scale citizen science project (EteRNA). The second pair of articles studied children’s health. One article used a traditional cross-sectional survey design to understand the relationship between screen time, behavior, and health-related quality of life in children. The other article studied the relationship between screen use and children’s mental health symptoms in a longitudinal study design that was co-created by the TARGetKids! Parent and Clinician Team (PACT). This team engaged patients and families throughout the entire research process, from setting research priorities to the interpretation and use of findings.
In a subsequent plenary discussion, the groups studying these articles reconvened and presented their conclusions. Most groups saw greater benefits of citizen involvement for societal rather than scientific impact, although there was an engaging debate on defining and measuring such impacts. Participants also concluded that the impacts depend on the breadth and depth of citizen involvement (e.g., only in data collection versus also in method design) and that such involvement, in turn, hinges on the initial goals set by project organizers.
Scholarly session
The final format was a scholarly session with presentations of four research papers that the organizers had selected through a formal review process. The following summaries highlight the diversity of citizen science practices studied by these papers.
Paper 1
Crowdfunding platforms such as Experiment.com enable citizens to shape the direction of research by funding projects that they find important (; ). A paper by Hesselbein and Franzoni () highlights that crowdfunding platforms also allow citizens to propose new research studies, including projects that may differ in both topic and methods from those typically proposed by professional scientists. The authors analyze proposals submitted to Experiment.com to better understand what they call “grassroots science,” finding that projects can be very novel—but also at the fringe of what professionals would consider legitimate science.
Paper 2
Most discussions of Zooniverse projects focus on micro-tasks performed by crowd members who help classify images and other digital objects (; ). However, some participants also engage in discussion forums to make sense of unexpected data or to support each other. A paper by Sun () uses text analysis techniques to study forum discussions in more than 100 Zooniverse projects. The author identifies several different clusters such as questions, facts, conflicting situations, and new ideas brought up by participants, and also finds interesting differences in forum discussions across fields. This paper highlights that discussion boards can enable Zooniverse participants to interact and make scientific contributions beyond image classification.
Paper 3
There is a growing literature on crowd involvement in solving scientific and technological problems (; ). The conventional wisdom is that organizers need a clearly defined problem that can then be solved by individual crowd members working independently. A paper by Vrolijk and Szajnfarber () challenges this wisdom by describing a new project design implemented by NASA. Rather than relying on the traditional contest setup, NASA organized a forum that engaged citizens to both define and solve an open-ended problem. This design generated information flows among participants that enabled better problem identification and the development of creative solutions.
Paper 4
While most citizen science projects focus on data collection, data analysis, and problem solving, there are also efforts to involve citizens in evaluating research proposals (; ). Beck et al. () argue that such contributions matter little if professional scientists—who are often the final gatekeepers—do not listen to crowd evaluations. As such, these authors study whether and when professional scientists incorporate crowd opinions into their funding decisions. Preliminary results suggest that scientists’ openness to crowd opinions depends on both the type of input provided by the crowd (substantive knowledge versus preference judgments) and the way in which crowd opinions are collected (representative versus self-selected sample of citizens).
Confusion arising from project diversity
Although participants in the scholarly session appreciated hearing about the four research studies, the subsequent discussion surfaced confusion that was already in the air during the earlier case presentations and the OIS experiment: The citizen science projects discussed at the conference all involve citizens in one way or another, but they seem to do so for very different reasons and using different organizational mechanisms. How are they even comparable? Can we identify clusters of projects that help us understand why and how exactly citizen involvement helped projects do better science?
Triggered by this confusion, participants discussed during and after this session a number of existing project taxonomies, including distinctions by the field of science (e.g., biology versus astronomy), the stage of the research process (e.g., problem definition versus data collection), and the overall project goals (e.g., environmental monitoring versus education) (; ; ; ). Although useful from a descriptive perspective, none of these taxonomies seemed to get at the heart of the issue: What is the underlying logic for why involving citizens helps projects produce more or better research?
A potential answer emerged when a participant pointed out five paradigms initially proposed at the 2019 OIS conference and then further developed by Beck et al. (). These paradigms synthesize different rationales of involving crowds that have been suggested by management scholars and that also became apparent in discussions of projects presented at the OIS conference. Building on Beck et al. (), we now briefly outline the five paradigms and use case examples from the conference to illustrate how the paradigms can help us to better understand the nature and benefits of citizen involvement. Given that the paradigms draw heavily on ideas from crowdsourcing and related literatures, we follow Beck et al. in using the term crowds.
Five Paradigms of Crowd Involvement Applied to Selected Citizen Science Projects Presented at the Conference
Describing the paradigms and linking them to example projects
Crowd volume
One rationale for crowd involvement is that large numbers of participants can provide a high volume of effort that can be distributed in space and time (; ). In projects focused on crowd volume, tasks tend to be standardized and require only common cognitive skills, which enables broad participation ().
The crowd volume paradigm helps us understand several of the projects discussed at the conference. It is central in Zooniverse projects (discussed at the scholarly session), among others, where participants contribute thousands of hours of time and make millions of classifications (). Crowd volume is also relevant for the marine litter project discussed at the OIS debate: The participation of many citizens enabled this project to collect large amounts of litter and data across various locations in Ghana. As these examples illustrate, crowd volume seems particularly relevant for projects involving crowds in data collection and processing.
Broadcast search
Other projects do not need a large volume of inputs per se, but rather highly specialized or rare knowledge, resources, or pre-existing solutions to a problem. These projects broadcast (i.e., make known) the problem or need to a large and diverse crowd such that individuals who have what is needed can self-select and offer their unique contributions (; ). The broadcast search paradigm is useful when thinking about crowd involvement in problem solving, where a few individuals often have particularly creative or valuable ideas ().
The broadcast search paradigm is very relevant to the project EteRNA (which was presented in the experiment session). The primary goal of this project is not to generate a large volume of RNA structures, but to find particularly novel and valuable outlier solutions that significantly improve upon existing structures (). Even though this project has many participants that contribute many structures, the goal is to find a few particularly valuable needles in the haystack. Broadcast search also helps us understand the value of allowing crowd members to propose new projects (rather than just funding projects proposed by professional scientists) on the platform Experiment.com (introduced in the scholarly session): Most people visiting Experiment.com have no project ideas, or their ideas likely have little potential. Yet, some people may have project ideas that are particularly creative and may push the frontier in particular research areas.
User crowd
This paradigm highlights the distinction between the scientific knowledge relevant to a particular problem that is typically held by professional researchers (e.g., scientists who understand the neurological basis of Alzheimer’s) and the experiential knowledge held by potential users who have direct experience in the problem domain (patients who live with Alzheimer’s). Experiential knowledge may enable users to identify novel problems that may not be salient to professional researchers, to better understand potential applications of existing scientific findings, or to recombine experiential knowledge with scientific knowledge to generate novel solutions (; ).
There are several potentially relevant groups of users in science. They include end-users such as patients who can benefit from treatments based on medical research, farmers who implement insights from agricultural research, and companies whose operations are informed by research in the social sciences. But users may include scientists in other fields (e.g., biologists who use or need tools developed by statisticians) as well as professionals who apply research results to help others (e.g., doctors treating patients).
The user crowd paradigm helps us understand several cases presented at the conference. For example, accessing the experiential knowledge of the elderly and people with disabilities allowed the project Co-creation with Vulnerable Groups to develop eHealth solutions that better meet the needs of the target groups (case session). Similarly, access to patient knowledge enabled the Care for ME/CFS project to develop an effective questionnaire-based diagnostic tool (case session).
Community production
The crowd volume, broadcast search, and user crowd paradigms focus on the contributions of individual crowd members. The community production paradigm highlights the value of interactions and collaboration among contributors. Exchange and collaboration can generate superior joint outcomes when relevant knowledge components are held by different individuals and when problems are complex (; ). Moreover, discussions among participants can help clarify preferences and trade-offs that are sometimes required when making decisions about which problems to investigate, what methods to use, or how to evaluate potential negative consequences of solutions (; ).
Several of the projects discussed at the conference rely on the community production paradigm. This includes, for example, the NASA project that engaged citizens to both define and solve an open-ended problem (scholarly session), the TARGet Kids! PACT (case sessions), and also EteRNA, where participants often interact to come up with better solutions (experiment session).
Crowd wisdom
A fifth paradigm focuses on the advantages that crowds have in making predictions or estimating values. For example, the average guess of people regarding things such as the weight of an ox can be surprisingly accurate (; ). The reason is that if judgments are at least somewhat independent, then individuals’ biases and errors tend to cancel out in larger crowds. Going beyond benefits from averaging estimates, this paradigm also captures more broadly that organizers reach out to many individuals to obtain statistical measures of crowd opinions. In the project Tell Us! about Mental Health, for example, clustering crowd-generated research questions by general problem area gave organizers a good idea of which areas of research citizens found particularly important to address.
The crowd wisdom paradigm helps us understand several of the projects and studies discussed at the OIS conference. In the scholarly session, for example, Beck et al. () examined when professional scientists follow crowd recommendations in project selection. The authors’ prediction that scientists are more likely to follow preference inputs provided by a random and representative sample of individuals builds directly on the crowd paradigm, which suggests that aggregating independent judgements from a random sample of individuals helps overcome individual errors and biases. Crowd wisdom also comes into play in Zooniverse projects, which typically ask multiple participants to classify the same image and then aggregate classifications to reduce the influence of individual errors ().
Visualizing which crowd paradigms are relevant in a project
Although the mechanisms captured by the five paradigms are conceptually distinct, many projects draw on multiple paradigms at the same time. The project Care for ME/CFS, for example, heavily relies on the user crowd and community production paradigms but it also needs citizens to invest a considerable amount of effort (crowd volume paradigm). Zooniverse projects primarily benefit from crowd volume and crowd wisdom (), but the conference presentation by Sun () shows that participants also interact in forums to integrate their knowledge (community production paradigm). EteRNA relies on both broadcast search and community production to generate novel RNA designs but also needs many participants to invest significant time to come up with solutions (crowd volume paradigm). Finally, the marine litter project in Ghana () relies on many participants to collect beach litter and data from various locations (crowd volume paradigm) but also benefits from the fact that user participants have a deep understanding of the local context, allowing them to categorize litter more accurately considering the local production and use (e.g., the same type of litter may contain plastic in one context but not another) (user crowd paradigm) ().
Building on Beck et al. () as well as on related discussions at multiple OIS conferences, co-authors of this report developed the “crowd paradigm diamond” as a simple tool to visualize how important each paradigm is in a particular project. The more important a particular paradigm is in explaining how citizen involvement allows a project to advance science, the darker the shade in the respective area of the diamond. Figure 1 illustrates this tool by comparing four of the projects discussed at the 2023 OIS conference (selected by the authors to illustrate diversity in relevant paradigms). Assessments of the relevance of each paradigm for a given project were first made by one co-author and then discussed and agreed upon by the whole author team. In doing so, we considered several guiding questions that helped us to analyze whether and how a particular paradigm is relevant. Figure 2 embeds these questions into the diamond shape and may serve as a useful template for researchers and practitioners who want to better understand whether and how citizen involvement can help a (current or planned) project accomplish its scientific goals. The following section discusses why recognizing relevant paradigms also has important implications for project design.
Crowd Paradigm Diamond showing the relevance of each of the five crowd paradigms for four selected cases discussed at the 2023 Open Innovation in Science Research Conference.
Crowd Paradigm Diamond template with expected benefits and guiding questions to assess the relevance of each of the five crowd paradigms.
Implications of crowd paradigms for citizen science project design
Understanding which crowd paradigms are relevant can help organizers design projects to better achieve their scientific goals while also considering implications for other important outcomes such as skill development or community building. In this section, we illustrate this for two specific decisions that are often particularly important and challenging. Readers are encouraged to also consult other existing resources for more comprehensive guidance on project design (e.g., ; ; ).
First, organizers should consider the paradigms when deciding which kind of crowd they want to engage, factoring in crowd size, skill levels, location, and demographic diversity. Zooniverse projects, for example, can attract a high volume of contributions (crowd volume paradigm) by enabling participation at varying commitment levels and without special skill requirements. Projects that use crowd volume to enable biodiversity monitoring need to recruit many participants while also paying attention to diversity in crowd members’ geographic locations. Projects that rely on the community production and user crowd paradigms (e.g., Care for ME/CFS) will typically work with fewer participants who have access to specific knowledge and resources and are willing to commit substantial time to interact with others. Organizers who seek to harness crowd wisdom to make better decisions in the selection of proposals or problem solutions should focus especially on diversity in preferences and other aspects such as participant demographics (). Column 3 in Table 1 summarizes additional connections between crowd paradigms and relevant crowd characteristics.
Table 1
Five crowd paradigms: key differences and implications for crowd characteristics and organizational mechanisms.
| PARADIGM | PRIMARY RATIONALE FOR INVOLVING THE CROWD | CROWD CHARACTERISTICS THAT FIT THE PARADIGM | ORGANIZATIONAL MECHANISMS THAT FIT THE PARADIGM |
|---|---|---|---|
| Crowd volume | A large crowd of contributors can supply high volume of effort or other inputs, potentially across different locations (e.g., for biodiversity monitoring). |
|
|
| Broadcast search | Broadcasting a problem or call for inputs to the crowd helps identify individual outlier solutions or other rare inputs. |
|
|
| User crowd | Users have experiential knowledge or other use-related inputs that professional scientists lack. |
|
|
| Community production | Interactions among crowd members allow recombination of complementary knowledge, preferences, and other inputs to address complex problems. |
|
|
| Crowd wisdom | Aggregating independent knowledge and preference inputs from many crowd members can mitigate individual-level errors and biases. |
|
|
Second, the paradigms also have implications for project organization, for example, with respect to the coordination of crowd members, the exchange of information, and the integration of contributions. For example, most projects relying on crowd volume or crowd wisdom will require infrastructure that collects independent contributions from participants, often integrating contributions afterwards through pooling (e.g., of observational data) or statistical transformations (e.g., clustering of topics, averaging of opinions). Projects seeking to generate benefits from the interaction between different participants (i.e., community production paradigm) need to provide spaces or communication tools to enable such interactions. Some projects accomplish this using digital infrastructure (e.g., discussion forums in Zooniverse projects or EteRNA), whereas others may opt for more traditional formats such as face-to-face meetings. Column 4 in Table 1 summarizes additional connections between crowd paradigms and relevant organizational mechanisms.
Concluding Reflections on the Conference and Conference Results
The annual OIS Research Conference provides a unique place to discuss citizen and crowd science. Different session formats stimulate different types of discussions and build an interdisciplinary community of scholars and practitioners. These discussions often continue beyond the event itself, as evidenced by this meeting report, which is co-authored by conference organizers and participants with different disciplinary backgrounds and citizen science experiences.
One of the challenges—but also success factors—of the annual conference is to enable productive discussions among diverse participants, for example, those using the citizen science versus crowd science lens. The five paradigms and the diamond template presented here can serve as a useful tool for such discussions by clarifying different mechanisms through which involvement of citizens and crowds can help advance science. The paradigms also point towards best practices in areas such as recruiting of participants or the organization of projects. In addition, the paradigms may stimulate future research. For instance, it would be interesting to map the relevance of each paradigm in a broader representative population of citizen science projects and to study what project characteristics (e.g., nature of the task; scientific field) predict which paradigm is most relevant. Similarly, research is needed to further explore the linkages between different organizational designs and project performance, and how this depends on which paradigm organizers seek to leverage (see Table 1).
An important limitation is that the crowd paradigms focus on the benefits of crowd involvement for improving scientific and practical outcomes, for example, through better data for scientific articles or problem solutions that better address the needs of users such as patients. The paradigms seem less relevant for understanding projects that focus on important nonscientific outcomes such as skill development, community building, or advocacy. We look forward to discussing at future OIS conferences whether and how the paradigms can be extended or combined with other tools to account for an even broader range of project outcomes.
Another potential challenge is that it can be difficult to judge exactly how important a paradigm is to a given project. In our experience, however, even just thinking about the relevance of each paradigm can generate important insights, and it is typically easy to identify one or two paradigms that are most relevant in a particular project.
The OIS conference organizers plan to introduce the diamond tool in the upcoming OIS Research Conference 2024 to stimulate deeper analyses of citizen science projects that are presented in the various session formats and to help practitioners explore additional opportunities to benefit from citizen involvement. We will also offer this tool for use in other events such as the European Citizen Science Association (ECSA) or the Association for Advancing Participatory Sciences (AAPS) conferences through our networks with those organizations.
Openness and collaboration can make science more productive and are central values of the OIS conference and the community that has developed around it. In that spirit, we invite scholars and practitioners to share feedback on the ideas laid out in this report, and to join upcoming conferences (see https://ois-research-conference.org/).