The National Institutes of Health (NIH) is the primary agency for conducting and supporting research in medical sciences in the United States, with the goal of advancing and applying fundamental research in biological sciences to translational and clinical research to enhance health, lengthen life, and reduce illness and disability (NIH 2017). The NIH is organized into twenty-seven different institutes and centers, each focusing on a specific research agenda and programming that supports biological, biomedical, and behavioral research and information dissemination to the public (NIH 2021). Through peer-reviewed, primarily investigator-initiated grant mechanisms, the NIH supports projects involving experts that include biomedical researchers, clinicians, engineers, data scientists, patient advocates, and the public. Teams tackle research that includes elucidating the basic biological mechanisms underlying health and disease, disease treatment and prevention, and the impact of the environment and other factors on health, behavioral and population science, and implementation science. Crowdsourcing, citizen science, community-based and community-engaged research methods are used in a wide variety of research projects spanning the biomedical research spectrum.
Building on success in community-based participatory research and new methods arising in fields such as data science, NIH staff recognized the potential for citizen science, crowdsourcing, and related methods to bring new expertise, new ideas, creativity, innovation, and a broader range of perspectives into the biomedical research enterprise. In 2012, a discussion group formed around biomedical citizen science. This group, which evolved to become the NIH Citizen Science Working Group in 2017, shares information, methods, and strategies to promote and support citizen science across the biomedical research spectrum. The working group comprises more than 100 members who are federal employees serving in a variety of roles across the NIH, from program officers to communications experts. They also have expertise in a range of scientific disciplines and disease areas across basic, translational, clinical, and behavioral research. The Working Group investigates the utility of citizen science methodologies in health research, and works to incorporate public participation in the research enterprise in a manner that maintains the high levels of scientific and ethical standards of the NIH (Mintz and Couch 2021). The authors of this review, Drs. Mintz and Couch, currently coordinate and lead the NIH Citizen Science Working Group, respectively.
Biological, biomedical, and behavioral research is diverse and rapidly changing. Biomedical citizen science leverages advances in methods, strategies, and approaches from other fields and disciplines. This type of research may involve working directly with the public to answer basic biological and health-related questions, or partnering with the public in ways that use personal health data or information. The NIH supports research through a wide variety of mechanisms, initiatives, and programs. Because of this, the NIH Citizen Science Working Group considers methods, approaches, and strategies rather than rigid plans for the incorporation of citizen science and crowdsourcing in health research, which complement rather than replace existing successful research methods. To enable creative new approaches that engage the public, the Working Group developed three guiding principles when considering citizen science:
In this review, we explore examples of biomedical citizen science projects and programs supported all or in part by NIH funding. To do so, we describe the results of a portfolio analysis conducted by the Group in late 2021 and early 2022 to better understand NIH investment in citizen science. We highlight specific projects as well as examples of community engagement programs at the agency related to biomedical citizen science. The final section focuses on the expansion of citizen science principles into related disciplines including crowdsourcing and games for science. We conclude with a discussion of the projects described and their impact on the field of biomedical citizen science.
In late 2021 and early 2022, the NIH Citizen Science Working Group conducted a preliminary portfolio analysis to better understand the NIH investment in citizen science since 2008. While citizen science, community engagement, community-based participatory research, and open innovation methods like challenges and prize competitions are widely integrated into NIH-supported research, a complete listing of citizen science and crowdsourcing activities has proved challenging to assemble and maintain due in large part to the breadth of NIH research across institutional boundaries, program structures, and health science disciplines. Because funding opportunities do not incorporate standardized language that describes citizen science and instead encourage researchers to apply creative methodologies to answer pressing health questions, most grants that fit under the umbrella of citizen science do not contain the term “citizen science” or other associated identifiers.
To perform the portfolio analysis, an internal database was used to identify new, awarded NIH grants across all fiscal years. The search term citizen science resulted in records that contained the full term in any aspect of the grant including but not limited to the specific aims, proposal and summary language, cited literature, researcher biosketches, funding announcement language, and agency descriptors. Manual review further discerned if the grant described citizen science using the criteria below. NIH RePORTER, a publicly accessible system, was used as a secondary source to confirm the results of the analysis. Project websites and peer-reviewed publications were referenced to collect additional information about the grant. Grants qualified as citizen science practice if they met any of the following criteria: (a) participants had the opportunity to create and/or build upon existing resources or materials; (b) participants had the opportunity to direct their own learning, research efforts, or propose and answer their own research questions; (c) there is a bidirectional flow of information directly used to address research questions between researchers and the public; (d) it involved the development of tools and technologies to enable the use of citizen science in biomedical research; and (e) they had partnerships with the public that included working with communities to identify and define research questions directly affected them or their environment.
The portfolio analysis distinguished between the practice of citizen science and the development and advancement of citizen science theory. Citizen science theory included grants that involved promoting or defining citizen science principles and included conferences; investigations into the ethical, legal, and social issues around the use of citizen science; and the development of guidance for using citizen science. The analysis identified 71 grants focusing on the practice of citizen science, and 25 grants focused on the theory of citizen science. However, additional activities were identified by the working group that were not captured by the portfolio analysis search criteria. For example, older NIH grants involving the practice or theory of citizen science may not have appeared in the portfolio owing to the relatively recent adoption of the term citizen science to describe public-partnered research (Cooper and Lewenstein 2016; Haklay et al. 2021) compared with the lifespan of NIH support of biomedical research.
Here, we describe examples of NIH-supported projects and programs related to biomedical citizen science. Several of the citizen science projects sit within larger programs that encourage the use of citizen science methods; the program and example projects are featured as well. The examples were chosen based on the success that they have had in advancing biomedical citizen science, whether it be long-term success measured by an expansion of the project to answer broader research questions; multiple peer-reviewed publications or publicly available interfaces that describe the impact of the project; a unique application of citizen science methodologies to answer unmet community needs; or the creation of tools or infrastructure to enable future citizen science activities. The examples discussed throughout the rest of the paper were identified by a combination of portfolio analysis and input from domain experts across the NIH to capture activities that use alternate terminology to describe citizen science methods.
The Our Voice initiative uses a citizen science approach to engage community residents and organizations in the entire span of the research process. With the overarching goal of promoting health equity, Our Voice projects rely on the direct input of local citizen scientists to identify conditions, behaviors, and environmental characteristics that promote or detract from community health (King et al. 2021b). Data is entered into mobile app, the Stanford Healthy Neighborhood Discovery Tool™, by the participants (Buman et al. 2013). This data is summarized and returned to the community, where trained local community members act as facilitators to lead discussion of the collective findings and determine realistic actions to address the identified issue. These multi-level discussions occur in multiple arenas to drive appropriate changes in the local environment. Our Voice has developed projects in more than 20 countries on six continents targeting citizen scientists of all ages (King et al. 2021b; King et al. 2020). In addition to driving locality-based changes to promote health and wellbeing, the associated Steps for Change study is underway to assess the impact of participation in the Our Voice citizen science model on person-level physical activity (King et al. 2021a).
Foldit is an online game that relies on crowdsourcing to solve complex protein design and folding puzzles. Foldit players compete in challenges such as protein structure prediction and de novo protein design (Koepnick et al. 2019), contributing to real scientific knowledge (Eiben et al. 2012; Khatib et al. 2011). In addition to solving biochemical problems, the game also emphases the creativity of human reasoning over computational models as well as the benefit of engaging citizen scientists unconstrained by traditional research paradigms (Koepnick et al. 2019). Foldit has also served as a model for understanding the motivation and mechanisms driving group collaboration and public participation in serious games (Miller et al. 2019a; Miller et al. 2019b; Bauer and Popovic 2017; Cooper et al. 2010).
Eyewire was launched in December 2012 and is an online community and game where citizen neuroscientists mapped retinal neurons by tracing and coloring single cells through two-dimensional electron microscope images of the mouse retina. Players can track their progress and the shape of the neuron by viewing the colored sections in a reconstructed three-dimensional cube. Players received a score for accuracy, which is based on coloring agreement between multiple players. EyeWire has recruited more than 100,000 players from more than 130 countries, and has resulted in the discovery of wiring specificity from bipolar cells to starburst amacrine cells and a new model for neuron direction selectivity (Kim et al. 2014). In addition to the technical and scientific challenges of creating a gaming platform to answer biological questions, EyeWire has successfully navigated the challenges that accompany growing and maintaining an online community and empowering non-scientists to engage with a neuroscience game (Robinson n.d.).
The National Library of Medicine (NLM) has emphasized citizen science. The NLM Strategic Plan for 2017–2027 recognizes the need for support for libraries and librarians to enable participation in citizen science to assure open data science and open science proficiency (RNLM 2017). The NLM FY2020 Budget Congressional Justification commits to supporting citizen science through events such as hackathons, and encourages citizens to interact with NLM products and services (NLM 2020). Within NLM, the Network of the National Library of Medicine (NNLM) hosts the biannual #CiteNLM Edit-a-thon during which participants add sources to Wikipedia articles focused on a pre-selected topic. These efforts increase the volume of credible, evidence-based health information and drive public engagement with scientific literature. NNLM has developed resources for other organizations to host their own edit-a-thons as well (NNLM 2022). In partnership with SciStarter, NNLM has also developed a collection of health-related projects and resources for Global Citizen Science Month, occurring annually in April (SciStarter n.d.).
The Science Education Partnership Award (SEPA) program funds STEM and science educational projects, resources, and activities for students and educators. Based out of the National Institute of General Medical Sciences (NIGMS), SEPA provides increased opportunities for students to develop an interest in science and scientific career fields, and provides professional development to teachers focused on STEM topics. SEPA projects emphasize building community health literacy through education, and materials are available for public use. The goals of the SEPA are aligned with the use of citizen science methods as a mechanism for participant engagement, and two examples of SEPA projects are discussed in further detail below.
The Cold Spring Harbor Laboratory DNA Learning Center (DNALC) launched the Barcode Long Island program to engage high school students and teachers in exploring and documenting biodiversity in southeastern New York. Students and educators are provided with the tools to design their own research question and test hypotheses about observations in their local environment using molecular biology techniques that contribute to biodiversity research while empowering students to act as scientists and community experts (DNALC 2022a). Results not only offer insight for local communities but contribute to the greater scientific knowledge base (Marizzi et al. 2018). A nationwide citizen science project, Barcoding US Ants, expanded the reach of the DNALC barcoding initiative, and challenged citizen scientists to collect and identify ants in their environment. The data collected contributes to ant range maps and allows for the identification of different species and subpopulations in neighborhoods across the United States (DNALC 2022b). The newly funded Citizen DNA Barcode Network aims to further build on the use of insects as indicators of environmental change, and relies on citizen scientists and community partners to use DNA barcoding to map the ranges of ants, mosquitos, and beetles.
The presence of arsenic and metal contamination in private well water is a major public health concern in New England states, including Maine and New Hampshire. Prolonged exposure to these contaminants, even at a concentration deemed safe for drinking, can cause poorly understood negative physical and behavioral effects. This project relied on student citizen scientists to collect well water samples from their homes or communities, which were then used to test behavioral toxicity in a zebrafish model to understand the effects of water composition (Babich et al. 2021). Their teachers received training to facilitate water sample collection. Data was uploaded to Anecdata.org, which was augmented with features to ensure data and user privacy specifically for use for this project. These features are now broadly available to other projects using the site (Bailey et al. 2021). By recruiting citizen science to obtain water samples, the research team was able to collect an increased number of water samples spanning a multi-state geographical area, while also spreading awareness of the importance of clean drinking water (Babich et al. 2021).
The Partnerships for Environmental Public Health (PEPH) program is a network that coordinates efforts between communities, scientists, public health officials, and policy-makers to advance environmental public-health research (PEPH 2022). PEPH encourages the use of citizen science in the greater context of community engagement to address environmental concerns through scientific research and to translate findings into direct action benefitting local communities. PEPH also supports the development of resources to advance community-engaged environmental health research by convening conferences, sharing information, and sharing best practices and other guidance (NIEHS 2012). Two examples of NIEHS projects that rely on citizen science are described further.
Following the addition of the Iron King Mine Superfund site to the National Priorities List in 2008, members of the local community expressed concern about the safety of vegetables grown in home gardens. A resulting community-academic partnership created the Gardenroots citizen science project, in which scientists and community members collaborated to measure arsenic in garden vegetables grown in Dewey-Humboldt, Arizona, conduct an exposure assessment, and measure the potential risk of consuming these vegetables. As a result of this partnership, findings were translated into modified gardening practices to reduce arsenic exposure, and community members demonstrated increased curiosity and understanding of the scientific process, the impact of environmental conditions on community health, and meaningful actions to mitigate environmental hazards (Ramirez-Andreotta et al. 2015).
This study was a partnership between a local community organization (Local Environmental Action Demanded [L.E.A.D.] Agency), the University of Oklahoma Health Sciences Center, and researchers at the Harvard T.H. Chan School of Public Health. To address community concerns about mercury exposure from contaminated fish, Grand Lake area resident citizen scientists provided hair samples at multiple time points to analyze for mercury, and tracked their fish consumption over the course of the study. Residents also assisted in collecting fish from the Grand Lake watershed. Although the majority of participants and fish species tested had low levels of mercury, study outcomes included suggestions for geographical areas and species to avoid for consumption, water conditions and seasonal patterns that lead to increased mercury levels in fish, and potential changes in fish consumption to reduce mercury exposure (Dong et al. 2015; Dong et al. 2016; Jim et al. 2014).
Many of the biomedical citizen science projects funded by NIH sit within the broad scope of community engagement initiatives and large cooperative groups and centers. For example, the All of Us Research Program recently released their first genomic data set of almost 100,000 whole genome sequences and genotyping arrays of 165,000 participants. This data was provided by citizen participants and is available to registered users on the Researcher Workbench along with electronic health records, Fitbit data, participant survey responses, and community data from the Census Bureau (Denny and Tabak 2022). The Researcher Workbench allows users to work with the data to answer their own identified research questions. The Pilot Project Program Core at the NIEHS-funded University of Louisville Center for Integrative Environmental Health Sciences encourages a citizen science approach to solicit research applications from communities to identify and address significant environmental health problems. Two additional NIEHS-funded Core Centers, the Lake Erie Center for Oceans and Human Health Community Engagement Core and University of Kentucky Center for Appalachian Research in Environmental Sciences, have successfully incorporated citizen science activities into the centers’ work to study local environmental health issues (Stanifer et al. 2022; Chaffin et al. 2021; PEPH 2020). Finally, the Cancer Moonshot Blue Ribbon Panel identified the establishment of a network for direct patient engagement as a recommendation to advance our understanding of cancer biology and treatment (BRP 2016). As part of this network, Count Me In, a nonprofit research initiative that enables patients to share their information and experiences with researchers and other patients, was awarded a grant to directly engage with osteosarcoma and leiomyosarcoma patients to collect data, health records, and biospecimens, and to build on their successful track record of patient-partnered research (NCI 2021b).
In addition to the more canonical examples of citizen science activities described previously, citizen science principles and methodologies have expanded into related disciplines within the scope of NIH-funded biomedical research. Crowdsourcing is a powerful research tool that has been incorporated into NIH-supported biomedical research in a variety of ways including for the collection of unique biospecimens, dataset annotation, and community building. NIH also hosts and funds open innovation events such as codeathons, hackathons, bootcamps, and innovation labs to bring together teams to collaborate on challenging scientific problems while sharing unique expertise and insight from diverse backgrounds.
Foldit, EteRNA (Lee et al. 2014), and Eyewire (Cooper et al. 2018) are successful and long running examples of scientific discovery games that have received continued and additional funding beyond the NIH investment. Signaligner is another example of a serious game that has implications for further development of wearable devices; this grant supported the development of a videogame played by non-experts that crowdsourced annotations for accelerometer data collected from wrist sensors (Ponnada et al. 2019).
The development of digital tools for health research has also been advanced by NIH investment. For example, the Digital Exposure Report-Back Interface (DERBI) technology automates the production of personalized reports to present environmental chemical exposure data in an accessible and interactive manner. After donating blood, tissue, or other biospecimens, results are returned to the participants via DERBI. Users view summarized results of what chemicals were present in their samples and their environment, related potential health concerns, and see comparisons to other study participants. They are informed of possible actions to mitigate health risks as well as the overall findings of the studies that collected said data (SSI 2022). This allows participants to make informed decisions to impact their personal health and increases their health literacy.
The intramural NHGRI Dog Genome Project studies the genetic variation of dog breeds to understand traits in modern dog breeds and how inherited diseases develop in certain breeds. Many of these inherited diseases are similar to those in humans, so by identifying the underlying genes and variants that contribute to these diseases, researchers gain insight into human diseases (Ostrander et al. 2019). This project relies on the donation of DNA samples provided by dog owners. The Dog Genome Project releases datasets and other resources to the broader research community (Plassais et al. 2019). At the National Cancer Institute, the Cancer Moonshot established the NCI Comparative Oncology Program (COP) to understand cancer biology through studying naturally occurring cancers in pet dogs and cats. Through a cooperative network of veterinary hospitals, the COP conducts clinical trials using investigational drugs to evaluate new treatments for cancer (Jacob 2016). Pet animals are enrolled into clinical trials by their owners. The information gained through these trials can be applied to a more complete understanding of cancer biology and cancer therapy in humans (LeBlanc and Mazcko 2020).
The NCI Program for Natural Products Discovery (NPNPD) is a Cancer Moonshot–funded initiative that advanced natural product discovery by creating a library of natural product extracts available for translational research use (Thornburg et al. 2018). This library, the NCI’s Natural Product Repository, contains more than 230,000 compounds derived from natural sources (Wilson et al. 2020). Contributions of soil fungi collected throughout the United States were provided to the NPNPD from the University of Oklahoma Citizen Science Soil Collection Program (NCI 2019). Citizen scientists collect samples of soil from their local environment and submit them for processing and screening. After the samples are tested, data is shared openly, and citizen scientists learn about what was found in the sample they contributed and compare these samples with others that were donated (UONPDG 2021). Several products have shown promise as anti-cancer compounds (Robles et al. 2016; Du et al. 2014), and NCI makes these samples available to researchers.
At NHGRI, the Omics Compendia Commons (OMiCC) is a free online crowdsourcing platform for creating and analyzing annotated gene expression data collections from human and mouse studies. The platform takes publicly available genomic data spread out across various databases and creates a single resource that can be categorized and annotated using standard terms. Metadata can be shared for data collation and analysis across different studies to develop novel biological insights (Shah et al. 2016).
Finally, the NIH hosts a variety of events such as innovation labs and hackathons to address complex scientific challenges. At NLM, the National Center for Biotechnology Information has a successful codeathon program, where researchers and coders from diverse backgrounds work together to create tools and tutorials focused on a specific research problem. They partner with institutions and operate in tandem with conferences to run these multi-day events (NBCI 2020; ODSS 2021). In 2021, NCI organized a “DataViz+Cancer” program that consisted of a series of microlabs followed by two innovation labs. The microlabs set the stage with open conversations between cancer biologists and creative data visualization experts, and the subsequent innovation labs provided intensive, weeklong workshops for participants to collaboratively develop innovative approaches to solve challenges in cancer care and biology (NCI 2021a). These events are exceedingly valuable in the COVID era because they can pivot to virtual and hybrid spaces, allowing for participants from all areas of the country to be able to participate without the traditional socioeconomic-based challenges that accompany in-person events.
The NIH has a successful track record of supporting biomedical citizen science. While many of the citizen science activities in the NIH portfolio are standalone projects, others are components of larger community engagement initiatives. The subject matter of these activities is diverse, from environmental public health to canine clinical trials and mobile health technologies. When the entire breadth of the NIH research enterprise is considered, the opportunities for citizen science in biomedical research expands to all health sciences including behavioral and social sciences as well as implementation research.
One sign of success in NIH-supported citizen science is the combined short- and long-term impacts of these projects. For example, the initial focus of the Barcode Long Island project was to teach students about biodiversity in their local environments through basic molecular biology techniques. The long-term success of this project has been the creation of the Citizen DNA Barcode Network and the focus on national and global issues, including climate change and the spread of invasive species. The #CiteNLM Edit-a-thon has a focused goal of increasing the reliability of specific Wikipedia pages; consistent effort over time increases the accuracy of health information. Many projects benefitted from initial NIH investment and continued to find success with help from other funding sources. The health impact of naturally occurring or introduced contaminants in local environments can be studied using similar methods as those employed in Gardenroots or the Arsenic in Well-Water projects; the global expansion of Our Voice is a testament to the success of incorporating citizen science methodologies into projects in different communities. Tools like DERBI can be adapted to allow for the dissemination of citizen science–collected information back to communities, simultaneously advancing public health and increasing health literacy. Often, the value of a citizen science project may be immediately intangible, but appears as long-term benefits to the engaged community and contributes to broadening the impact of investment in biomedical research.
One challenge faced by the NIH in identifying and comparing different citizen science and crowdsourcing projects is both the scope (NIH funded 56,794 grants in 2021 [Lauer 2022]) and identification of citizen science and related terms in our highly distributed and diverse portfolio. Not all citizen science projects self-identify in the grant language. From the preliminary portfolio analysis conducted by the NIH Citizen Science Working Group in late 2021–early 2022, the vast majority of identified grants did not contain the term citizen science. They were instead identified by specific criteria: opportunities provided to participants, a bidirectional flow of information from the public to researchers, a common goal of increasing public involvement in research, or the formation of partnerships that identify research questions directly affecting the health of local communities.
Another challenge that faces funding organizations like the NIH is evaluating the impact of investment in citizen science. Quantifying the success of citizen science initiatives by measurable benefit per dollar spent, for example, is not sustainable or possible in many instances. Citizen science projects rely on the support of different communities with unique constraints, specific goals, and potential impacts that extend beyond easily quantified benefits such as data collection and availability or papers published. The value of citizen science projects may lie in community engagement and trust-building, or broadening the understanding of a specific field. Regardless, being able to identify and define citizen science will increase the ability of researchers and communities to connect, to establish common goals, and to promote the use of these methodologies within the biomedical research enterprise.
The boundaries and characteristics of biomedical citizen science are constantly expanding, and qualifications for defining an activity as citizen science are not consistent across institutions or disciplines (Wiggins and Wilbanks 2019; Guerrini et al. 2019; Eitzel et al. 2017). The NIH Citizen Science Working Group believes it would be beneficial to develop a clearly defined yet broad term specific to NIH citizen science that encompasses all activities, and to set a definition and framework for supporting and evaluating future investments. However, the diversity of ways that citizen science and other methods are employed in biomedical research is continuously evolving. The group has discussed the term community science as one option, and continues to refine the best approach, likely evolving the term(s) as the field expands and preference for terminology changes by those who apply for research funding.
With the promise of citizen science to engage diverse communities and advance public health (Cooper et al. 2021; King et al. 2019), and the pledge of NIH to address structural racism in biomedical research (Collins et al. 2021), ensuring that these methodologies are used ethically and responsibly is of paramount importance. This commitment goes beyond NIH. Communications from the Biden Administration acknowledge the use of citizen science and community-engaged research to advance government-funded science, and include a charge to develop scientific integrity guidelines, policies, and practices for these types of programs and to involve the public as participants (SIF-TAC 2022; Biden 2021).
Methods and strategies from citizen science, crowdsourcing, community-engaged research, and other related disciplines continue to have great potential to accelerate biomedical research in an ever-increasing variety of ways. Challenges remain in areas such as data governance, methods for shared credit and ownership, inclusivity, outreach, and adaptability of methods to a wider range of populations, research topics, and goals. Opportunities exist for community research to continue to provide creative avenues for biological discovery, to accelerate research, to enhance data and methods development, and to encourage technical and methodological innovation across the spectrum of biological, biomedical, and behavioral research important to the health and well-being of all.
The authors would like to thank the NIH Citizen Science Working Group for their input and contributions to this review. They would also like to acknowledge that the projects described in this report may have received additional support from other organizations.
The authors have no competing interests to declare.
Babich, R, Craig, E, Muscat, A, Disney, J, Farrell, A, Silka, L and Jayasundara, N. 2021. Defining drinking water metal contaminant mixture risk by coupling zebrafish behavioral analysis with citizen science. Nature Scientific Reports, 11(1): 17303. DOI: https://doi.org/10.1038/s41598-021-96244-4
Bailey, C, Farrell, A, Purty, T, Taylor, A and Disney, J. 2021. Development of Privacy Features on Anecdata.org, a Free Citizen Science Platform for Collecting Datasets for Climate Change and Related Projects. Frontiers in Climate, 3. DOI: https://doi.org/10.3389/fclim.2021.620100
Bauer, A and Popovic, Z. 2017. Collaborative Problem Solving in an Open-Ended Scientific Discovery Game. Proceedings of the Association for Computing Machinery on Human-Computer Interactions, 1(22). DOI: https://doi.org/10.1145/3134657
Buman, MP, Winter, SJ, Sheats, JL, Hekler, EB, Otten, JJ, Grieco, LA and King, AC. 2013. The Stanford Healthy Neighborhood Discovery Tool: a computerized tool to assess active living environments. American Journal of Preventative Medicine, 44: e41–e47. DOI: https://doi.org/10.1016/j.amepre.2012.11.028
Chaffin, JD, Bratton, JF, Verhamme, EM, Bair, HB, Beecher, AA, Binding, CE, Birbeck, JA, Bridgeman, TB, Chang, X, Crossman, J, Currie, WJS, Davis, TW, Dick, GJ, Drouillard, KG, Errera, RM, Frenken, T, MacIsaac, HJ, McClure, A, McKay, RM, Reitz, LA, Domingo, JWS, Stanislawczyk, K, Stumpf, RP, Swan, ZD, Snyer, BK, Westrick, JA, Xue, P, Yancey, CE, Zastepa, A and Zhou, X. 2021. The Lake Erie HABs Grab: A binational collaboration to characterize the western basin cyanobacterial harmful algal blooms at an unprecedented high-resolution spatial scale. Harmful Algae, 108: 102080. DOI: https://doi.org/10.1016/j.hal.2021.102080
Collins, FS, Adams, AB, Aklin, C, Archer, TK, Bernard, MA, Boone, E, Burklow, J, Evans, MK, Jackson, S, Johnson, AC, Lorsch, J, Lowden, MR, Napoles, AM, Ordonez, AE, Rivers, R, Rucker, V, Schwetz, T, Segre, JA, Tabak, LA, Hooper, MW, Wolinetz, C and NIH UNITE. 2021. Affirming NIH’s commitment to addressing structural racism in the biomedical research enterprise. Cell, 184(12): 3075–3079. DOI: https://doi.org/10.1016/j.cell.2021.05.014
Cooper, CB, Hawn, CL, Larson, LL, Parrish, JK, Bowser, G, Cavalier, D, Dunn, RR, Haklay, M, Gupta, KK, Osborne Jelks, N, Johnson, VA, Katti, M, Leggett, A, Wilson, OR and Wilson, S. 2021. Inclusion in citizen science: the conundrum of rebranding. Science, 372(6549): 1386–1388. DOI: https://doi.org/10.1126/science.abi6487
Cooper, S, Khatib, F, Treuille, A, Barbero, J, Lee, J, Beenen, M, Leaver-Fay, A, Baker, D, Popovic, Z and Foldit players. 2010. Predicting protein structures with a multiplayer online game. Nature, 466: 756–60. DOI: https://doi.org/10.1038/nature09304
Cooper, S, Sterling, ALR, Kleffner, R, Silversmith, WM and Siegel, JB. 2018. Repurposing Citizen Science Games as Software Tools for Professional Scientists. Foundations of Digital Games, 2018: 1–6. DOI: https://doi.org/10.1145/3235765.3235770
Cold Spring Harbor DNA Learning Center (DNALC). 2022a. Barcode Long Island. Available: https://dnabarcoding101.org/programs/bli/ [Accessed 17 March 2022].
Cold Spring Harbor DNA Learning Center (DNALC). 2022b. Barcoding US Ants. Available: https://dnabarcoding101.org/citizen/usants/index.html [Accessed 17 March 2022].
Denny, J and Tabak, L. 2022. All of Us: Release of Nearly 100,000 Whole Genome Sequences Sets Stage for New Discoveries. NIH Director’s Blog [Online]. Available from: https://directorsblog.nih.gov/2022/03/29/nihs-all-of-us-research-programs-first-nearly-100000-complete-human-genome-sequences-set-stage-for-new-discoveries/ [Accessed 29 March 2022].
Dong, Z, Jim, RC, Hatley, EL, Backus, AS, Shine, JP, Spengler, JD and Schaider, LA. 2015. A longitudinal study of mercury exposure associated with consumption of freshwater fish from a reservoir in rural south central USA. Environmental Research, 136: 155–62. DOI: https://doi.org/10.1016/j.envres.2014.09.029
Dong, Z, Lynch, RA and Schaider, LA. 2016. Key contributors to variations in fish mercury within and among freshwater reservoirs in Oklahoma, USA. Environmental Science: Process & Impacts, 18: 222–36. DOI: https://doi.org/10.1039/C5EM00495K
Du, L, Robles, AJ, King, JB, Powell, DR, Miller, AN, Mooberry, SL and Cichewicz, RH. 2014. Crowdsourcing natural products discovery to access uncharted dimensions of fungal metabolite diversity. Angewandte Chemie International Edition, 53(3): 804–809. DOI: https://doi.org/10.1002/anie.201306549
Eieben, CB, Siegel, JB, Bale, JB, Cooper, S, Khatib, F, Shen, BW, Foldit players, Stoddard, BL, Popovic, Z and Baker, D. 2012. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature Biotechnology, 30: 190–2. DOI: https://doi.org/10.1038/nbt.2109
Eitzel, MV, Cappadonna, JL, Santos-Lang, C, Duerr, RE, VirapongseI, A, West, SE, Kyba, C, Bowser, A, Cooper, CB, Sforzi, A, Metcalfe, AN, Harris, ES, Thiel, M, Haklay, M, Ponciano, L, Roche, J, Ceccaroni, L, Shilling, FM, Dorler, D, Heigl, F, Kiessling, T, Davis, BY and Jiang, Q. 2017. Citizen Science Terminology Matters: Exploring Key Terms. Citizen Science:Theory and Practice, 2(1): 1–20. DOI: https://doi.org/10.5334/cstp.96
Biden, JR, Executive Office of the President. 2021. Memorandum on Restoring Trust in Government Through Scientific Integrity and Evidence-Based Policymaking. The White House. Available at https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/memorandum-on-restoring-trust-in-government-through-scientific-integrity-and-evidence-based-policymaking/ (Last accessed 17 March 2022).
Guerrini, CJ, Wexler, A, Zettler, PJ and McGuire, AL. 2019. Biomedical Citizen Science or Something Else? Reflections on Terms and Definitions. American Journal of Bioethics, 19(8): 17–19. DOI: https://doi.org/10.1080/15265161.2019.1619880
Haklay, M, Dorler, D, Heigl, F, Manzoni, M, Hecker, S and Vohland, K. 2021. What is Citizen Science? The Challenges of Definition. In: et al. The Science of Citizen Science. Springer, Cham. DOI: https://doi.org/10.1007/978-3-030-58278-4_2
Jacob, JA. 2016. Researchers Turn to Canine Clinical Trials to Advance Cancer Therapies. Journal of the American Medical Association Network, 315(15): 1550–1552. DOI: https://doi.org/10.1001/jama.2016.0082
Jim, RC, Hatley, EL, Backus, AS, Dong, Z, Schaider, LA, Shine, JP, Spengler, JD and Lynch, RA. 2014. Grand Lake watershed mercury study: Mercury exposure among fish consumers and bioaccumulation in commonly consumed fish. September 2014. Available at http://www.grandlakemercurystudy.org/images/GrandLakeWatershedMercuryStudy_FinalReport_Sept2014.pdf.
Khatib, F, DiMaio, F, Foldit Contenders Group, Foldit Void Crushers Group, Cooper, S, Kazmierczyk, M, Gilski, M, Krzywda, S, Zabrankska, H, Pichova, I, Thompson, J, Popovic, Z, Jaskolski, M and Baker, D. 2011. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18: 1175–1177. DOI: https://doi.org/10.1038/nsmb.2119
Kim, JS, Greene, MJ, Zlateski, A, Lee, K, Richardson, M, Turaga, SC, Purcaro, M, Balkam, M, Robinson, A, Behabadi, BF, Campos, M, Denk, W, Seung, HS and the EyeWirers. 2014. Space-time wiring specificity supports direction selectivity in the retina. Nature, 509: 331–336. DOI: https://doi.org/10.1038/nature13240
King, AC, Campero, MI, Garcia, D, Blanco-Velazquez, I, Banchoff, A, Fierros, F, Escobar, M, Cortes, AL, Sheats, JL, Hua, J, Chazaro, A, Done, M, Espinosa, PR, Vuong, D and Ahn, DK. 2021a. Testing the effectiveness of community-engaged citizen science to promote physical activity, foster healthier neighborhood environments, and advance health equity in vulnerable communities: The Steps for Change randomized controlled trial design and methods. Contemporary Clinical Trials, 108: 106526. DOI: https://doi.org/10.1016/j.cct.2021.106526
King, AC, King, DK, Banchoff, A, Solomonov, S, Ben Natan, O, Hua, J, Gardiner, P, Rosas, LG, Espinosa, PR, Winter, SJ, Sheats, J, Salvo, D, Aguilar-Farias, N, StathiI, A, Akira Hino, A, Porter, MM, On Behalf of the Our Voice Global Citizen Science Research Network. 2020. Employing participatory citizen science methods to promote age-friendly environments worldwide. International Journal of Environmental Research and Public Health, 17(5): 1541. DOI: https://doi.org/10.3390/ijerph17051541
King, AC, Odunitan-Wayas, FA, Chaudhury, M, Rubio, MA, BaiocchiAIOCCHI, M, Kolbe-Alexander, T, Montes, F, Banchoff, A, Sarmiento, OL, Balter, K, Hinckson, E, Chastin, S, Lambert, EV, Gonzalez, SA, Guerra, AM, Gelius, P, Zha, C, Sarabu, C, Kakar, PA, Fernes, P, Rosas, LG, Winter, SJ, McClain, E, Gardiner, PA, On Behalf Of The Our Voice Global Citizen Science Research Network. 2021b. Community-based approaches to reducing health inequities and fostering environmental justice through global youth-engaged citizen science. International Journal of Environmental Research and Public Health, 18(3): 892. DOI: https://doi.org/10.3390/ijerph18030892
King, AC, Winter, SJ, Chrisinger, BW, Hua, J and Banchoff, AW. 2019. Maximizing the promise of citizen science to advance health and prevent disease. Preventative Medicine, 119: 44–47. DOI: https://doi.org/10.1016/j.ypmed.2018.12.016
Koepnick, B, Flatten, J, Husain, T, Ford, A, Silva, DA, Bick, MJ, Bauer, A, Liu, G, Ishida, Y, Boykov, A, Estep, RD, Kleinfelter, S, Norgard-Solano, T, Wei, L, Foldit players, Montelione, GT, DiMaio, F, Popovic, Z, Khatib, F, Cooper, S and Baker, D. 2019. De novo protein design by citizen scientists. Nature, 570: 390–394. DOI: https://doi.org/10.1038/s41586-019-1274-4
Lauer, M. 2022. FY 2021 By The Numbers: Extramural Grant Investments in Research. 7 March 2022. Available at https://nexus.od.nih.gov/all/2022/03/07/fy-2021-by-the-numbers-extramural-grant-investments-in-research/. (Last accessed 1 June 2022)
LeBlanc, AK and Mazcko, CN. 2020. Improving human cancer therapy through the evaluation of pet dogs. Nature Reviews Cancer, 20: 727–742. DOI: https://doi.org/10.1038/s41568-020-0297-3
Lee, J, Kladwang, W, Lee, M, Cantu, D, Azizyan, M, KimIM, H, Limpaecher, A, Yoon, S, Treuille, A, Das, R and EteRNA Participants. 2014. RNA design rules from a massive open laboratory. Proceedings of the Nationall Academies of Sciences of the United States of America, 111(6): 2122–2127. DOI: https://doi.org/10.1073/pnas.1313039111
Marizzi, C, Florio, A, Lee, M, Khalfan, M, Ghiban, C, Nash, B, Dorey, J, McKenzie, S, Mazza, C, Cellini, F, Baria, C, Bepat, R, Costentino, L, Dvorak, A, Gacevic, A, Guzman-Moumtzis, C, Heller, F, Holt, NA, Horenstein, J, Joralemon, V, Kaur, M, Kaur, T, Khan, A, Kuppan, J, Laverty, S, Lock, C, Pena, M, Petrychyn, I, Puthenkalam, I, Ram, D, Ramos, A, Scoca, N, Sin, R, Gonzalez, I, Thakur, A, Usmanov, H, Han, K, Wu, A, Zhu, T and Micklos, DA. 2018. DNA Barcoding Brooklyn (New York): A first assessment of biodiversity in Marine Park by citizen scientists. PLoS One, 13: e0199015. DOI: https://doi.org/10.1371/journal.pone.0199015
Miller, JA, Lee, V, Cooper, S and El-Nasr, MS. 2019a. Large-scale analysis of visualization options in a citizen science game. Proceedings of the Annual Symposium on Computer-Human Interaction in Play, 2019: 535–542. DOI: https://doi.org/10.1145/3341215.3356274
Miller, JA, Narayan, U, Hantsbarger, M, Cooper, S and El-Nasr, MS. 2019b. Expertise and engagement: Re-designing citizen Sscience games with players’ minds in mind. Foundation of Digital Games, 2019(6): 1–11. DOI: https://doi.org/10.1145/3337722.3337735
Mintz, E and Couch, J. 2021. NIH Citizen Science Working Group [Online]. National Cancer Institute. Available at https://www.cancer.gov/research/resources/citizen-science/working-group [Accessed 1 March 2022].
National Center for Biotechnology Information: National Library of Medicine (NBCI). 2020. Available at https://ncbi-codeathons.github.io [Accessed 31 March 2022].
National Cancer Institute Cancer Currents Blog (NCI). 2019. Nature’s Bounty: Revitalizing the Discovery of New Cancer Drugs from Natural Products. Available at https://www.cancer.gov/news-events/cancer-currents-blog/2019/cancer-drugs-natural-products-nci-program [Accessed 22 March 2019].
National Cancer Institute (NCI). 2021a. DataViz+Cancer. Available at https://apply.hub.ki/cancerplusviz/ [Accessed 30 March 2022].
National Cancer Institute (NCI). 2021b. Establish a Network for Direct Patient Engagement. Available at https://www.cancer.gov/research/key-initiatives/moonshot-cancer-initiative/implementation/patient-engagement [Accessed 20 March 2022].
National Institute of Environmental Health Sciences (NIEHS). 2012. Partnerships for Environmental Public Health Evaluation Metrics Manual. NIH Publication No. 12–7825, National Institutes of Health, Bethesda, MD. Available at https://www.niehs.nih.gov/research/supported/assets/docs/a_c/complete_peph_evaluation_metrics_manual_508.pdf. [Accessed 30 March 2022].
National Institute of Environmental Health Sciences Partnerships for Environmental Public Health (PEPH). 2020. Citizen Science Programs at the Centers for Oceans and Human Health. Available at https://www.niehs.nih.gov/research/supported/translational/peph/webinars/ohh/index.cfm [Accessed 16 March 2022]
National Institute of Environmental Health Sciences Partnerships for Environmental Public Health (PEPH). 2022. About PEPH: Partnerships for Environmental Public Health. Available at https://www.niehs.nih.gov/research/supported/translational/peph/about/index.cfm [Accessed 10 March 2022]
National Institutes of Health (NIH). 2017. What We Do- Mission and Goals. Available at https://www.nih.gov/about-nih/what-we-do/mission-goals. [Accessed 29 August 2022].
National Institutes of Health (NIH). 2021. Who We Are- Organization. Available at https://www.nih.gov/about-nih/who-we-are/organization. [Accessed 29 August 2022]
National Library of Medicine Board of Regents (RNLM). 2017. A Platform for Biomedical Discovery and Data-Powered Health: National Library of Medicine Strategic Plan 2017–2027. National Library of Medicine, Bethesda, MD.
National Library of Medicine (NLM). 2020. Congressional Justification. Available at https://www.nlm.nih.gov/about/2020CJ_NLM.pdf. [Accessed 19 March 2022]
Network of the National Library of Medicine, National Library of Medicine (NNLM). 2022. Wikipedia Edit-a-thon. Available: https://nnlm.gov/guides/wikipedia-edit-a-thon [Accessed 15 March 2022]
Office of Data Science Strategy, National Institutes of Health (ODSS). 2021. Codeathons at NIH. Available at https://datascience.nih.gov/codeathons-nih [Accessed 30 March 2022]
Ostrander, EA, Dreger, DL and Evans, JM. 2019. Canine cancer genomics: Lessons for canine and human health. Annual Review of Animal Bioscencesi, 7: 449–472. DOI: https://doi.org/10.1146/annurev-animal-030117-014523
Plassais, J, Kim, J, Davis, BW, Karyadi, DM, Hogan, AN, Harris, AC, Decker, B, Parker, HG and Ostrander, EA. 2019. Whole genome sequencing of canids reveals genomic regions under selection and variants influencing morphology. Nat Commun, 10: 1489. DOI: https://doi.org/10.1038/s41467-019-09373-w
Ponnada, A, Cooper, S, Thapa-Chhetry, B, Miller, JA, John, D and Intille, S. 2019. Designing Videogames to Crowdsource Accelerometer Data Annotation for Activity Recognition Research. Proceedings of the Annual Symposium on Computer-Human Interaction in Play, 2019:135–147. DOI: https://doi.org/10.1145/3311350.3347153
Ramirez-Andreotta, MD, Brusseau, ML, Artiola, J, Maier, RM and Gandolfi, AJ. 2015. Building a co-created citizen science program with gardeners neighboring a superfund site: The Gardenroots case study. International Journal of Public Health, 7(1): 13.
Robinson, A. n.d. EyeWire: A Game to Crowdsource Brain Mapping. Citizen Science Tool Kit Case Study. Available at https://www.citizenscience.gov/assets/files/eyewire-brain-mapping.pdf. [Last accessed 31 May 2022]
Robles, AJ, Du, L, Cichewicz, RH and Mooberry, SL. 2016. Maximiscin Induces DNA Damage, Activates DNA Damage Response Pathways, and Has Selective Cytotoxic Activity against a Subtype of Triple-Negative Breast Cancer. Journal of Natural Products, 79: 1822–1827. DOI: https://doi.org/10.1021/acs.jnatprod.6b00290
Scistarter. n.d. NLM NNLM Citizen Science Month. Available at https://scistarter.org/NLM [Accessed 12 March 2022].
Shah, N, Guo, Y, Wendelsdorf, KV, Lu, Y, Sparks, R and Tsang, JS. 2016. A crowdsourcing approach for reusing and meta-analyzing gene expression data. Nature Biotechnology, 34: 803–6. DOI: https://doi.org/10.1038/nbt.3603
Scientific Integrity Fast-Track Action Committee (SIF-TAC). 2022. Protecting the Integrity of Government Science. United States of America Executive Office of the President Office of Science and Technology Policy. Available at https://www.whitehouse.gov/wp-content/uploads/2022/01/01-22-Protecting_the_Integrity_of_Government_Science.pdf [Last accessed 17 March 2022]
Silent Spring Institute (SSI). 2022. Digital Exposure Report-Back Interface (DERBI). Available at https://silentspring.org/project/digital-exposure-report-back-interface-derbi [Accessed 15 March 2022].
Stanifer, SR, Rayens, MK, Wiggins, A, Gross, D and Hahn, EJ. 2022. Home Radon Testing in Rural Appalachia. J Rural Health, 38: 251–261. DOI: https://doi.org/10.1111/jrh.12552
Thornburg, CC, Britt, JR, Evans, JR, Akee, RK, Whitt, JA, Trinh, SK, Harris, MJ, Thompson, JR, Ewing, TL, Shipley, SM, Grothaus, PG, Newman, DJ, Schneider, JP, Grkovic, T and O’Keefe, BR. 2018. NCI Program for Natural Product Discovery: A Publicly-Accessible Library of Natural Product Fractions for High-Throughput Screening. ACS Chemical Biology, 13(9), 2484–2497. DOI: https://doi.org/10.1021/acschembio.8b00389
University of Oklahoma Natural Products Discovery Group (UONPDG). 2021. The University of Oklahoma Citzen Science Soil Collection Program. Available at https://whatsinyourbackyard.org/about/ [Last accessed 16 March 2022].
Wiggins, A and Wilbanks, J. 2019. The Rise of Citizen Science in Health and Biomedical Research. American Journal of Bioethics, 19(8): 3–14. DOI: https://doi.org/10.1080/15265161.2019.1619859
Wilson, BAP, Thornberg, CC, Henrich, CJ, Grkovic, T and O’Keefe, BR. 2020. Creating and screening natural product libraries. Natural Product Repositories, 37: 893–918. DOI: https://doi.org/10.1039/C9NP00068B