Start Submission

Reading: Citizen Science, Civics, and Resilient Communities: Informing Community Resilience Policies ...

Download

A- A+
Alt. Display

Case Studies

Citizen Science, Civics, and Resilient Communities: Informing Community Resilience Policies Through Local Knowledge, Community Values, and Community-Generated Data

Authors:

David Sittenfeld ,

Museum of Science, Boston, US
X close

Mahmud Farooque,

Arizona State University, US
X close

Brian Helmuth,

Northeastern University, US
X close

Sara Benson,

Museum of Science, US
X close

Emily Hostetler,

Museum of Science, US
X close

Francis Choi,

Northeastern University, US
X close

Nicholas Weller,

Arizona State University, US
X close

Caroline Nickerson,

SciStarter, US
X close

Katie Todd,

Museum of Science, US
X close

Darlene Cavalier

SciStarter, US
X close

Abstract

Because climate hazards present a range of potential impacts and considerations for different kinds of stakeholders, community responses to increase resilience are best considered through the inclusion of diverse, informed perspectives; local knowledge; and complementary methods of engagement. Our NOAA-funded Citizen Science, Civics, and Resilient Communities project utilizes a “science-to-civics” framework, combining participatory data collection with deliberation about potential strategies to build community resilience to climate hazards. These activities combine citizen-created resilience plans with participatory data collection activities in an effort to characterize and assess local vulnerabilities through co-creation of knowledge, while also eliciting public values about proposed policies for hazard mitigation. The project builds upon earlier activities that convened public deliberations to elicit public values and attitudes with respect to climate adaptation policies at eight US science centers. We propose that the synthesis of community data collection with these types of public fora demonstrates potential to amplify learning between resilience planning officials and diverse publics, increasing the relevance and usability of community-generated local knowledge for policymakers, and providing opportunities for citizen scientists to sustain engagement. We present evaluation results from the Forum deliberations, with particular focus on the co-generation of knowledge between and among public participants and resilience planners, summarize our first pilot of the science-to-civics model, which recently conducted a participatory community assessment in the Boston area about extreme heat, and describe a scaled national effort in 2021 that facilitated science-to-civics activities in 30 US communities.

How to Cite: Sittenfeld, D., Farooque, M., Helmuth, B., Benson, S., Hostetler, E., Choi, F., Weller, N., Nickerson, C., Todd, K. and Cavalier, D., 2022. Citizen Science, Civics, and Resilient Communities: Informing Community Resilience Policies Through Local Knowledge, Community Values, and Community-Generated Data. Citizen Science: Theory and Practice, 7(1), p.33. DOI: http://doi.org/10.5334/cstp.516
250
Views
40
Downloads
3
Twitter
  Published on 14 Oct 2022
 Accepted on 14 Aug 2022            Submitted on 12 May 2022

Introduction

Because climate hazards present a range of potential impacts and considerations for different kinds of stakeholders, community responses to increase resilience are best considered through the inclusion of diverse, informed perspectives; local knowledge; and complementary methods of engagement. Our National Oceanographic and Atmospheric Administration (NOAA)-funded Citizen Science, Civics, and Resilient Communities project (CSCRC) utilizes a “science-to-civics” framework, combining participatory data collection with deliberation about potential strategies to build community resilience (NOAA Education 2018). The CSCRC project builds upon earlier efforts that convened public deliberations to elicit public values and attitudes with respect to climate adaptation policies at eight US science centers. We propose that the synthesis of community data collection with these types of public fora demonstrates potential to amplify learning between resilience planning officials and diverse publics, increasing the relevance and usability of community-generated local knowledge for policy-makers, and providing opportunities for citizen scientists to sustain engagement.

This analysis situates these participatory resilience planning activities within the broader policy context. We first describe a rationale for the theoretical framework for “science-to-civics” processes that utilize citizen science activities, co-selected and designed in collaboration with resilience planners, as an on-ramp to deliberative participatory policy-making. We describe outcomes from two NOAA-funded Environmental Literacy projects that have implemented elements of the science-to-civics process. The Science Center Public Forums (SCPF) project developed and implemented four deliberative hazard modules with input from subject matter experts, stakeholders, and resilience decision-makers, convening public Forums at eight US science centers. We present evaluation results from the SCPF efforts and policy recommendations generated by Forum participants, with particular focus on the co-generation of knowledge between and among public participants and resilience planners. We conclude by summarizing our first pilot of the CSCRC project, which recently conducted two participatory community assessments in Boston, and subsequently expanded to citizen science and deliberation activities at nearly 30 institutions in 2020 and 2021. Science-to-civics aligns community-generated resilience plans with participatory data collection activities in an effort to characterize and assess local vulnerabilities through co-creation of knowledge, while also eliciting public values about proposed policies for hazard mitigation. These activities could help to position informal science education institutions as trusted conveners for informed community dialogue by engaging citizens in critical thinking, data collection and analysis about vulnerabilities, and potential policy responses to critical climate hazards, while sharing usable public values and priorities with civic planners.

Combining citizen science and public deliberation can help to address resilience education needs

The National Oceanographic and Atmospheric Administration’s (NOAA) Environmental Literacy Program (ELP) is focused on developing and testing methods of public education that increase community resilience to environmental hazards while engaging community members in equitable co-production of knowledge. Among the activities recommended by NOAA to accomplish these aims are “deliberative forums, citizen science, participatory decision making and mapping exercises, and scenario-based or role-playing activities and games” (Bey 2020). A number of challenges, opportunities, and best practices with respect to this type of education were identified by NOAA community resilience ELP grantees as part of a workshop co-hosted by NOAA Education staff and members of our CSCRC project team in 2017. NOAA’s summary report from the workshop recommends a number of strategies for informal learning that are applicable to convening deliberations, including recommendations to “build relationships with scientists, local government, and community organizations,” “creating a list of feasible, community-based solutions,” to “facilitate discussions rather than lecture,” and to “address values and emotions, not just knowledge (NOAA 2017).” In particular, the report identifies a pressing informal learning challenge of “ongoing engagement with participants.” As members of our group have encountered when convening deliberations on environmental topics, Forums create respectful and informed spaces at a time when such dialogues are needed to elevate the level of societal discourse, and methods for sustaining engagement afterwards would help to extend these impacts over time (Worthington 2012). Workshop attendees recommended “citizen science & civic engagement” as an opportunity for addressing this challenge. Similarly, while citizen science is a powerful tool for engaging public audiences, innovations are needed to sustain engagement among participants, address the historically high attrition rates among citizen science projects over time, and also to connect to societal issues, as has been suggested by leaders in the citizen science field (Bonney et al 2009). The summary of the ELP grantee workshop states, “Following up with the audience is the only way to successfully build resilient citizens and communities…a one-time, ten-minute presentation, or even a day-long workshop, is not enough; building resilience requires a sustained, ongoing commitment from both parties.” The science-to-civics design therefore aims to sustain participant engagement while combining complementary active learning and engagement methodologies (Newman et al 2012).

Leaders in the citizen science movement, such as Rick Bonney from the Cornell Lab of Ornithology, have pointed to the civic dimensions of co-created citizen science projects as especially promising for sustaining and building participation among citizen science volunteers: “Some PPSR participants become more engaged in community politics and more confident about asking for a place at the table in making decisions about community planning” (Bonney et al 2009). These kinds of projects can bring the citizen data collectors to the forefront of the project, demonstrating to other community members that they also can contribute to scientific understanding. Further, these kinds of projects demonstrate that everyday community members who participate in citizen science efforts have an important and unique role to play in the interpretation and translation of the results into the world of policy (Shirk et al 2012).

Citizen science falls within the scope of “Public Engagement with Science” (PES), which is defined as “intentional, meaningful interactions that provide opportunities for mutual learning between scientists and members of the public” by the American Association for the Advancement of Science (AAAS 2016). Lewenstein (2012) classifies citizen science along with “Participatory Democracy” as two of four “dimensions” of PES. In defining these types of PES activities (Storksdieck et al. 2016), AAAS explicitly recommends the combination of complementary PES methods to help address existing gaps in the field:

“By including the range of activity-types, the theory of change underscores that there is no programmatic ‘silver bullet’ for achieving the long-term vision of societal change within public engagement. Rather, a complementary suite of activities, combined with intentional goal-setting and training toward related communication skill sets, is needed.” (AAAS 2016)

Further, synthesizing deliberative forums with citizen science activities can help to increase the policy relevance of these kinds of engagements. The science-to-civics process aims to help agenda-setting by helping to identify and characterize hazard vulnerabilities to be decided on through public deliberation. Forums can also help with agenda-setting for citizen science: What are the climate impacts that communities see as most important to address or most important to collect data about? Rowe and Frewer (2005) identify three essential and defining elements of public engagement with science as “[1] agenda-setting, [2] decision-making, and [3] policy-forming activities of organizations/institutions responsible for policy development.” Our science-to-civics process is therefore designed to address each of these elements by creating opportunities for members of the public to increase understanding about local vulnerabilities through the contribution of citizen-created data about an issue pertaining to a local hazard resilience plan (agenda setting), by learning about and considering the tradeoffs of various potential resilience strategies (decision-making), and then sharing their priorities, recommendations, and values about a question of local relevance (policy-forming).

The Science-to-Civics Framework

The elements of the science-to-civics framework are summarized and aligned to the three primary phases of public engagement as identified by Rowe and Frewer (2005) in Figure 1 below.

Steps of the science-to-civics framework, connected to the three public engagement mechanisms identified in Rowe and Frewer (2005)
Figure 1 

Steps of the science-to-civics framework, connected to the three public engagement mechanisms identified in Rowe and Frewer (2005).

Agenda-Setting: Selection and Implementation of Participatory Science Activities

The first phase of the science-to-civics framework is the agenda-setting step, which begins with the selection of a locally relevant citizen science project connecting to local hazard planning priorities and to a forthcoming policy deliberation. Spitzer and Frazer (2020) assert that science museums are “positioned to engage purposefully with community groups, gather and activate entities within their region, and serve as trusted interpreters and advisors on how to act based on the implications of scientific findings that align to their mission” and thereby promote community science literacy about climate change. The agenda-setting step therefore begins by convening diverse community partners, resilience planners, and others together in community and expert co-creation sessions to listen to community priorities and collectively select and define hazards of community concern. An informal science center educator at each site meets with local resilience planning and community partners to choose appropriate projects. Activities that have been included as part of the CSCRC project include: monitoring rainfall or water quality (extreme precipitation), mapping urban heat islands (heat waves), soil moisture (drought), or photographing king tide events (sea level rise). These agenda-setting decisions are informed by input from local resilience planning partners and local civic and community stakeholders. The citizen science projects include at least one low-threshold, entry-level data collection activity that is easily accessible to everyone; this became especially important after the pivot to virtual engagement during the pandemic. In many cases, project partners also include a higher-threshold element that participants can opt into, requiring more elaborate training or the use of more complex protocols or equipment. We generally aim to recruit at least 100 participants per site for these projects but work to identify projects that can accommodate more. After project selection, the next step is the recruitment of citizen science volunteers. Citizen science volunteers are recruited by project partners, including the host science center, resilience planning partners, and various civic and/or community partners who have been engaged in the project planning.

After participants are recruited, they are introduced to the relevant project hazard and associated vulnerabilities and training about the corresponding citizen science activities and their connections to local vulnerability assessment activities. Volunteers are then oriented to the citizen science component and overarching project through online modules that were developed and hosted by SciStarter as part of a virtual project portal. These modules convey the scientific and resilience planning context for the activities, train participants to contribute data, and highlight ways for citizen science participants to interact with one another, to share perspectives, and to monitor the progress of the data collection efforts.

The final component of the agenda-setting step is the actual data collection by citizen volunteers, which occurs over a period of 1 to 2 months. During this time, project organizers and resilience planners along with subject matter experts are available to answer online questions from citizen science volunteers and provide periodic updates on progress. This includes an invitation to participate in an upcoming public forum event, which is also extended to members of the general public who have not been involved as citizen scientists.

Convening Public Deliberation for Decision-Making and Policy-Forming

The decision-making step begins with the implementation of the public forum event. This event is open to the general public, but volunteers who have participated in the citizen science project are extended special invitations.

The Forum event consists of three components:

  1. Presentation and discussion of the data collected by citizen science volunteers. The program begins by describing and recognizing the effort and contributions made by the citizen science volunteers in helping to build community resilience. This element is intended to visibly demonstrate the usefulness of the citizen-created data to resilience planning activities and is also intended to encourage other attendees of the public Forum to participate in subsequent data collection to sustain their engagement. The Forum thus informs the participants about how their data will be used and provides them an opportunity to help take action with local planners about an issue of local relevance. Additionally, the Forum is intended to encourage public attendees who have not previously participated in the citizen science project to sustain their engagement after the event by contributing data afterward. It also makes a clear connection to the locally focused resilience planning question.
  2. Deliberation. The Forums collect information in the form of recommendations and values from the participants that can be shared with resilience planners and other community and scientific audiences. As discussed in the next section, the Forum engages participants in focused deliberation about resilience strategies and their implications for various community stakeholders. Participants recommend and choose to support strategies in an atmosphere of constrained resources, considering the environmental, social, and economic tradeoffs of various policy alternatives by allocating “coins” to potential resilience actions.
  3. Locally focused resilience planning question: The policy-forming step begins with the discussion of a locally focused resilience planning question. After the deliberation about the potential resilience strategies is complete, participants transition to a topic that has been determined in collaboration with a local resilience planner, and which also connects back to the citizen science data. This component connects the decision-making step to the policy-forming step, in that it builds upon the SCPF decision-making exercise to focus specifically on locally relevant priorities for hazard mitigation and resilience planning.

Participants consider how the resilience strategies proposed and discussed in the hazard module might impact them in the context of local resilience planning and hazard vulnerabilities. These conversations are informed by the front-end agenda-setting that occurred at the project’s inception. Participants make concrete recommendations about resilience planning in their own communities.

The policy-forming activities continue with a call to action that extends beyond the Forum events, building on the relationships that have been established between data collectors, Forum participants, and resilience planners. The discussion about the locally focused resilience planning question can continue beyond the Forum event, providing opportunities to continue to engage with subject matter experts and resilience planners. Forum participants are invited to contribute their own citizen science data (or to continue doing so), and also to invite others to participate in subsequent science-to-civics activities.

Convening National Conversations: The Science Center Public Forums Project

Through our Science Center Public Forums: Community Engagement for Environmental Literacy, Improved Resilience, and Decision-Making (SCPF) project, we created four multifaceted visualization and deliberation modules designed to engage diverse publics in substantive deliberations around four hazards: heat waves, drought, extreme precipitation, and sea level rise. The SCPF modules were created with input from resilience planners, subject matter experts, and community stakeholders through iterative workshops, interviews, and reviews, and then tested with formative focus groups and two pilot forum events in 2017. Experts and planners participated in these deliberations in three ways: (1) providing input to the creation of the educational materials (2) acting as respondents to participant questions during the deliberations through in-person and online interactions, and (3) receiving reports compiled by the project team that summarize participants’ values, recommendations and priorities. The SCPF materials were piloted at science centers in Arizona and Massachusetts, and implemented at six more science centers in Alabama, California, Hawaii, Minnesota, North Carolina, and Oregon in spring of 2018 (see Figure 2 for a map of the SCPF host sites.) Forums have been demonstrated to facilitate mutual learning between publics and members of the scientific community (Bell et al 2017).

Science Center Public Forum Deliberation Host Sites (2017–2018)
Figure 2 

Science Center Public Forum Deliberation Host Sites (2017–2018).

Using a suite of materials including visualization and narrative components, each of these daylong dialogues engaged diverse groups of lay participants at eight US science centers (two in 2017 and six in 2018) in learning about hazard vulnerabilities and tradeoffs of proposed resilience strategies. Participants listened to and considered the priorities and perspectives of fellow residents and stakeholders, and worked together to formulate detailed resilience plans reflecting both current science and informed public values. The participants’ recommendations, values, and priorities were communicated to resilience planners, with the goal of mutual learning between resilience planners and members of the public.

Each SCPF deliberation module focuses on a particular hazard and engages participants in an iterative resilience planning process based upon the steps of NOAA’s Climate Resilience Toolkit (NOAA, 2018). The background materials, visualizations, and deliberation materials all have been informed by NOAA’s US Climate Resilience toolkit, the Fourth US National Climate Assessment, and other datasets at local/regional/global scales. SCPF Forum participants learn about hazard vulnerabilities and possible resilience actions and strategies through immersive planetarium graphics and geospatial visualizations of environmental and socioeconomic datasets using Google Earth. Community narratives and detailed background materials provided participants information about potential impacts on neighborhoods and representative stakeholders. The materials identified economic, social, and environmental tradeoffs of proposed resilience strategies sourced from national and local resilience plans.

Approximately 60 Forum participants at each location, selected to reflect the demographic and cultural diversity of the region, worked together in facilitated groups to consider values, priorities, and perspectives of community stakeholders. They then considered economic factors, long- and short-term considerations, uncertainties, the potential for cascading hazards, and issues of equity. Participants then explored the outcomes and stakeholder impacts of their resilience plans at the community and neighborhood levels through visualizations. We also developed and hosted workshops for disseminating the materials to K–12 classrooms, to higher education classes, and to informal educators, whom we trained to use them in other educational settings. The SCPF materials have been utilized by project partners in university courses, in public schools, as part of public meetings, and in informal community settings.

These deliberative Forum engagements were intended to accomplish two objectives— first, to inform everyday citizens about the complex tradeoffs of proposed resilience planning actions; and second, to inform resilience planners about the values and perspectives of everyday citizens, so that resilience planning decisions will be more reflective of the perspectives of the diverse public. These two complementary purposes map to three rationales for public participation in technical policy-making laid out by Fiorino (1990) and expanded on by Stirling (2008): substantive, instrumental, and normative rationales. Substantive rationales hold that the judgments of everyday people about technical topics, such as resilience, are more sound because they take more views and perspectives into consideration. Instrumental rationales relate to the ability of public participation to realize specific ends, like learning about the resilience planning process (OMB 2021). Rationales related to empowering citizens or to creating more justice planning processes are normative rationales (Hong and Page 2004). In the next section, we describe the affordances of these deliberations for environmental policy-making and some findings from their implementation at the eight science center host sites.

Setting Agendas: Aligning Public Deliberations to Resilience Planning Priorities

These rationales are important because they help frame how policy-makers view public engagement and what potential outcomes might be. The outcomes of public engagement with technical policy-making hinge, in part, on policy-makers’ perceptions of the utility of engagement. We attempted to use policy-maker involvement to further understand of this type of engagement and demonstrate that it can be relevant, credible, and salient to policy-making priorities, extending principles of actionable science (Beier et al. 2017) to public participation. Below, we discuss the development of Forums and the ways we involved policy-makers and planners in this process.

First, we identified partners via our own professional networks and recommendations from the National Oceanic and Atmospheric Administration. These partners included elected officials, university and government researchers, and planners from a variety of municipal, regional, state, and federal agencies. Municipal, county, state, and federal agencies represented were related to resilience planning, including emergency management, public health, water resources, public works, and sustainability planning. We conducted brief phone calls with potential partners to gauge their interest in the project and willingness to provide input. Additionally, we used reports from cities, regional governments, states, and federal agencies to identify issues related to each extreme hazard and resilience strategies to help scope our interactions with officials.

We then convened two workshops to inform the content development for the deliberation materials: one in Phoenix, Arizona on heat waves and drought, and one in Boston, Massachusetts on extreme precipitation and sea level rise. At each workshop, partners worked through an example deliberative activity to become familiar with the types of engagement activities we sought to develop. Several partners delivered brief “lightning” talks on issues related to their expertise and to the various hazards. For example, an official from a county health department reported on the impact of heat waves on human health in central Arizona. Following the talks, partners split into small groups in which a member of our team led an activity to 1) identify salient and relevant issues related to each hazard, 2) identify strategies communities are considering to cope with those hazards, and 3) identify existing resources and data related to those issues. Groups then shared each of these issues in an open forum format, allowing for additional discussion and prioritization. The information collected at these workshops served as inputs into the content development processes for Forum activities. In addition to these workshops, our team used a variety of academic, professional, and governmental literature to guide content development.

These workshops and the literature review represented an agenda-setting step in our planning and development process. These agenda-setting activities accomplished a few things. First, they allowed us to secure commitments from a variety of policy-makers and experts to provide feedback at various points in the project. Second, they allowed us to ground our activities in specific recommendations from the policy community. By using those recommendations, we sought to ensure that the activities and content we developed represented the resilience policy community’s current priorities. Further, an explicit agenda-setting step emulates agenda-setting that happens in formal policy processes, providing an easily relatable step for policy-makers and experts advising our effort.

Because of the national scope of the project, we elected to create hypothetical communities for each hazard module. The communities were based upon real locations and included authentic environmental, social, and geospatial data, but in an effort to reduce personal bias among the participants, they were anonymized to allow for some distance from each of the eight host sites. In the time that we were developing these materials, a number of analogous projects utilized similar strategies. For example, Marin County Community Development Agency created the popular “Game of Floods,” which employs a hypothetical community called “Marin Island” (County of Marin 2021). Similarly, the National Academy of Science created an “Extreme Event Game” applying disaster management techniques to hypothetical communities for earthquakes, floods, and hurricanes (LabX and Koshland Science Museum 2021). Our hazard modules are based upon the communities of Louisville, Kentucky (heat); Charleston, South Carolina (Sea Level Rise); Grand Rapids, Michigan (Extreme Precipitation); and Reno, Nevada (drought). The subsequent locally focused resilience planning questions that were convened in each location after these hypothetical modules were developed by the host institutions and their partnering resilience planners, in service of connecting the ideas in the Forums to specific local resilience planning priorities.

After several internal rounds of activity and content development, we turned back to partners for content review. Partners were provided outlines of the activity, including descriptions of specific hazards, resilience strategies, and the impacts of those resilience strategies. We used in-person conversations and phone calls with partners to review those descriptions and to identify improvements. For example, after reviewing draft materials for the drought activity, we clarified pieces of the activity that referred to the feasibility of grey water reuse systems. This process served to hone content and to make it more reflective of resilience priorities currently in consideration by policy-makers. It also served as an opportunity for us to reassess our coverage of a very complex subject.

Importantly, our activity does not lay out locally specific policy choices as they are considered in professional resilience planning communities. In this way, it is abstracted and different from formal policy-making processes. However, features of the activity honed through the agenda-setting and review processes described above emulated considerations from policy-makers and experts. Specific tradeoffs, the inclusion of limited resources, and various pieces of each resilience strategy were chosen to create an activity that emulates the considerations resilience planning professional must consider.

Data collection and analysis

Data collection and analysis activities focused on four overarching parameters: (1) participant learning, assessed through formative and summative program evaluation; (2) individual and group resilience planning recommendations, collected and quantitatively analyzed from completed resilience planning worksheets; (3) participants’ stated motivations and underlying values for their resilience planning preferences, collected from qualitative responses by participants and associated table observations; and (4) policy-maker reflections with respect to the participants’ recommendations and suggestions for improvements to maximize the utility of the outputs for local resilience planning.

Program evaluation to assess participant learning

The summative evaluation of the Science Center Public Forums project involved a multi-site data collection effort through which evaluators gathered post-surveys from participants at each of the eight forum sites around the country. All summative evaluation activities were covered under the Museum of Science, Boston’s Institutional Review Board, protocol number 2015.12. The events were advertised locally by the host museums, using newsletters, social media, email lists, and other marketing approaches. Interested people were invited to apply to participate, and participants were selected such that they represented the demographic diversity of each museum’s surrounding region in terms of age, gender, income levels, educational experience, ethnicity, and employment status. At the end of the events, participants had the opportunity to complete a paper post-survey before they left. All respondents were over the age of 18 and provided informed consent in accordance with IRB protocols. Across the eight sites, 367 post-surveys were collected. Fifteen percent of respondents were between the ages of 18 and 24, 38% were between the ages of 25 and 44, 32% were between the ages of 45 and 64, and 16% were older than 65. Sixty percent of sample participants were female, and 39% were racial or ethnic minorities. In terms of annual household income, 23% made less than $25,000 a year, 22% between $25,000 and $49,999, 35% between $50,000 and $99,999, and 20% $100,000 or more. For educational attainment, 36% held a graduate or professional degree, 30% held a bachelor’s degree, 26% had some college, 6% had a high school degree or equivalency, and 1% had no high school diploma. Fifty three percent were employed, 6% were unemployed, 15% were retired, 15% were students, and 11% described their employment status as “other.”

The post-survey used a range of self-report items, including retrospective pre-post questions, which minimize respondents’ tendency to inflate their pre-survey knowledge ratings (Rennie and Johnson 2007). Below, we share a selection of quantitative data points that relate to policy-related efforts. Evaluators analyzed the data using a mix of descriptive and nonparametric inferential statistics, which are appropriate owing to the non-normal distributions of the data. The number of responses varies slightly in some cases because participants did not have to respond to all of the survey questions. The full evaluation took a mixed-methods approach, gathering a combination of quantitative and qualitative data. Full results and more information about the evaluation are in the full SCPF summative evaluation report, including details and results from pre-surveys that were additionally collected but are out of the scope of this analysis (Todd et al. 2018).

The evaluation assessed the extent to which participants met a range of goals including:

  • increased knowledge, awareness, and understanding of natural hazards, resilience efforts, interactions between nature and society, and scientific processes;
  • engagement in discussions and use of data during the forums;
  • increased interest in research about natural hazards and related societal issues; and
  • more supportive attitudes towards community resiliency measures.

This analysis looks across these goals, and highlights three themes: 1) participants engaged in rich discourse that mirrors civic processes, 2) through this discourse participants learned multi-faceted information about interactions between climate and resilience, and 3) the forums fostered increased interest that could lay the groundwork for future participation in policy work.

1) Through the forums, participants engaged in civic practices of hearing from others, sharing their own opinions, and working with others to make shared recommendations.

On the post-surveys, respondents indicated the extent to which they agreed or disagreed that they had done several activities (see Figure 3). The data show that 99% agreed or strongly agreed they heard others’ viewpoints about what actions should be taken to reduce the impacts of climate-related hazards (n = 355), 98% agreed or strongly agreed that they shared their views about what actions should be taken to reduce the impacts of climate-related hazards (n = 357), and 91% agreed or strongly agreed their group’s resilience plan reflected their personal views (n = 353). The data from these questions displayed similar patterns across all of the event sites; there were no statistically significant differences based on the location of the event.

Responses to post-survey question, “Thinking about your experience at the forum and with the forum materials, how much do you agree or disagree with the statements below?”
Figure 3 

Responses to post-survey question, “Thinking about your experience at the forum and with the forum materials, how much do you agree or disagree with the statements below?”

These three actions—hearing others’ views, sharing one’s own views, and working with others to develop a shared plan—are key building blocks of civic process. The fact that over 90% of all participants agreed or strongly agreed with each of these statements shows that the forum format can be effective in creating an opportunity to practice these skills that can be applied in a wide range of contexts.

2) Forum participants learned about multi-directional interactions between humans and their environments.

On the post-surveys, participants indicated how much they knew about various topics before and after the event on retrospective pre/post questions. Respondents reported statistically significant increases in their agreement that they knew about a range of topics, a sampling of which is shown in Figure 4. Seventeen percent of participants indicated that they knew a lot about the types of impacts that surrounding environments have on local communities, and this percentage rose to 53% after the forum (Wilcoxon Signed Ranks Test; n = 354, Z = –12.915, p < 0.001, r = 0.686). Sixteen percent reported they knew a lot about the types of impacts communities have on their surrounding environment before the forum, and 52% indicated that they knew a lot about this topic afterwards (Wilcoxon Signed Ranks Test; n = 356, Z = –12.553, p < 0.001, r = 0.665). There were no statistically significant differences in these data based on the site where the events took place.

Responses to post-survey question, “How much did you know about the following topics before the forum, and how much do you know after?”
Figure 4 

Responses to post-survey question, “How much did you know about the following topics before the forum, and how much do you know after?”

These data demonstrate that forum participants felt that they learned about multi-directional impacts between communities and environments. This illustrates a deeper level of public engagement than the simple learning of facts (McCallie et al. 2009). The learning involves development of understanding that systems have reciprocal, complex, and interrelated interactions. This recognition of complexity is valuable for weighing options when considering potential policies.

3) After the forums, participants reported greater interest in areas that could set the stage for future policy involvement.

As part of the post-survey, forum participants indicated how much they agreed or disagreed with several statements about their attitudes toward climate resilience efforts and their interest in learning more about them. The questions asked respondents to select a rating for their perspectives before the forum and then afterwards (see Figure 5). Respondents reported that, after the forum, they had statistically significantly stronger agreement that they were interested in learning about how climate-related hazards may affect their community (45% strongly agreed before the forum and 81% agreed afterwards; Wilcoxon Signed Ranks Test: n = 359, Z = –11.337, p < 0.001, r = 0.598). Similarly, there was stronger agreement after the forum that it is important for participants’ local community to develop and implement a resilience plan (49% strongly agreed before the program and 88% strongly agreed afterwards; Wilcoxon Signed Ranks Test: n = 359, Z = –11.730, p < 0.001, r = 0.619). Thus, participants left the forum reporting that they were interested in continued learning about the forum topic and were more supportive of resilience efforts in their communities. As with the previous data, there were no statistically significant differences in these data based on the geographic location of the different events.

Responses to retrospective pre/post-survey question, “Please rate your agreement with the following statements before you participated in the forum, and then after.”
Figure 5 

Responses to retrospective pre/post-survey question, “Please rate your agreement with the following statements before you participated in the forum, and then after.”

While interest and support do not necessarily predict future participation, they may lay the groundwork for civic participation. Future studies—including the evaluation of the upcoming project that melds forums and citizen science—can investigate the extent of behavior change after a forum. Collaborations between forum organizers and policy-makers could provide direct opportunities for Forum participants to extend their interest and support into ongoing, applied action.

Individual and group resilience plans, policy recommendations, and motivations

In addition to the summative evaluation discussed above, we also analyzed the participants’ resilience planning recommendations and their potential utility for local resilience planners.

Based on recommendations from resilience planners in our design workshops, we structured the deliberation in a format that allowed us to easily tabulate resilience priorities that were prioritized and selected by participant groups and by individuals. Participants supported resilience plan elements connecting to their priorities by allocating “coins” to the resilience strategies of their preference. Across all sites, we tabulated the selected plans for the relevant hazards, and used these tabulations and qualitative elements from the discussions to create summaries and customizable templates (see Figure 6 for an example) for the informal science education hosts to share with their resilience planning and community partners. Below we report a snapshot of those recommendations because a full accounting of recommendations across all sites would be expansive and beyond the scope of this work.

A summary element from the Phoenix, AZ Extreme Heat Forum, visualizing participants’ resilience plan allocations and the proportion of total resilience plans
Figure 6 

A summary element from the Phoenix, AZ Extreme Heat Forum, visualizing participants’ resilience plan allocations and the proportion of total resilience plans.

A majority (22 of 38 or 58%) of tables at 5 sites that considered extreme precipitation chose plans involving substantial investment in various green infrastructure solutions. Two-thirds of tables across 4 sites that considered SLR chose plans that included managed retreat. For the 2 sites that considered drought, 63% of tables chose plans that emphasized conserving water resources and increasing watershed stewardship efforts. At the 3 sites that considered extreme heat, no majority preference emerged, perhaps demonstrating the lack of focus extreme heat has received as a public health, emergency, or resilience risk. Seven of 24 tables recommended strategies focused on decreasing the urban heat island, 6 of 24 chose plans focused on protecting key infrastructure from extreme heat, and 7 of 24 tables chose plans that invested in smaller efforts targeted towards safety, infrastructure, and cooling.

Reflections and lessons learned from local resilience officials

We conducted interviews with local resilience planners and outreach specialists who informed the content development, observed the deliberations, and/or responded to participant questions. These interviews focused on what about the Forums was interesting, useful, and relevant to the public officials who observed or contributed to them. This focus allowed us to evaluate the policy utility of the Forum activities and to improve activities and future efforts. Notably, these interviews took place after forums but before our team compiled voting results from each forum and before the analysis and reporting of qualitative data from participant discussions, meaning interviewees had not yet seen a summary of the recommendations from the public. A total of 9 officials participated in these semi-structured interviews. Transcribed interviews were qualitatively coded using an iterative open coding approach to highlight major themes (Saldaña 2016). These themes are discussed below.

Interviewed officials highlighted the ability of Forums to inform participants about hazards, resilience strategies, and the complexities and challenges of planning for resilience. For example, one interviewee who works on resilience planning in a large city stated:

“…it was really fascinating to watch [participants at the forum] struggle with a lot of the things that I and my colleagues struggle with on a regular basis in terms of no clear solutions that are going to fix everything, winners and losers, not enough money to do everything, things that you think are going to work sometimes don’t work how you think they’re going work.”

Similarly, an official in another city noted that the forums forced people to grapple with challenges of making policy. In this way, policy officials highlighted the importance of Forums to build civic capacity in general, in addition to content knowledge about resilience. The observations of officials were supported by participant evaluation data reported above (see Figures 4 and 5). Further, this finding provides support for one design aspiration in the development of Forums activities: Emulate the difficulties and tradeoffs inherent in actual resilience planning.

Interviewees also valued the potential for these Forums to forge ongoing relationships with communities. Based on these interviews officials and policy-makers come away from forums interested in deploying the model in their own jurisdiction:

“…if you were able to make connections with [policy officials in our community] and could bring some of the participants of the [forum] back to present to them in some way, or to help author a presentation of some sort, I think that could be very meaningful.” – Outreach Specialist

This sentiment aligns with observations of involved policy-makers from past forum efforts on other topics (Tomblin et al. 2017).

As the quote above demonstrates, officials often see Forums as just one tool in a larger toolbox to foster engagement amongst communities and policy-makers. While Forums were a good first step in creating engagement, interviews pointed to a need to work more closely with local communities to create other opportunities for engagement and foster relationships among Forum conveners and resilience officials.

Interviewed officials were disappointed that the Forum activity components, such as maps and resilience plans, weren’t specific to their city. We made the decision not to customize the Forum activity components for every host location to make it easier for other institutions to utilize the materials, though we did provide support for hosts who wanted to do so. Interviewees nonetheless viewed the activities as covering the same suite of issues and considerations that their communities face:

“when I first heard…that it wasn’t going to be specific to [our city], I was sort of disappointed, because I thought; oh, that’s a missed opportunity for us to explore these issues, particularly for our city, but after going through it, I saw that in many ways, a lot of the issues that you picked in particular, are universal in nature, and the narrative in the issues for any given city are probably very similar.”

While these national-scale findings highlight a challenge of scaling forum activities across communities, while still maintaining relevance to local policy-making priorities, we have since created and implemented science-to-civics activities at the hyperlocal level in response to this feedback, and others have as well (Science Museum of Virginia 2020).

Finally, officials contrasted the Forums with existing modes of engaging public audiences. One resilience official compared the Forum to her past experiences with other public engagements, stating:

“[The forum] shifted the focus of that engagement in a way that made it much more accessible for me, where I could listen, and ponder things without having to be in a reactionary mode.”

She contrasted this experience (Figure 10) with the “usual” approach to public engagement on resilience issues in which the public provides comments on policy proposals from officials. Importantly, the role of science centers as neutral convening institutions allowed officials to observe and listen to citizen concerns and preferences fostering learning by the officials themselves. Officials were given time to present to participants during the Forums about specific challenges, opportunities, and efforts underway in the community. Many participants continued discussions with officials during breaks and after the Forums were over. While this effort did not directly feed into specific policy processes, it opened the window for further engagement focused on constructively listening to and accounting for citizen preferences rather than passive acceptance of feedback on existing proposals. In this way, the forums served a capacity-building role for more than just participants and convening ISE institutions. Involving officials in innovative public engagement helps demonstrate the utility and importance of such engagement beyond administrative requirements to check the box of public consultation.

Participants deliberating at a Forum in Phoenix, Arizona
Figure 10 

Participants deliberating at a Forum in Phoenix, Arizona.

In summary, our experience involving officials demonstrated:

  • – the difficulty of scaling forum activities across communities while maintaining relevance to local policy context;
  • – the need to situate forums as part of a broader engagement between communities and policy-making processes to create resilience communities;
  • – the ability of forums to foster learning by both officials and participants; and
  • – the need for humility and creative thinking regarding the direct policy impact of forums.

The CSCRC project has adapted these materials in response to the feedback and experiences from public and resilience planning audiences and synthesized them with relevant citizen science projects pertaining to climate hazards as outlined in the science-to-civics framework presented earlier. We conclude below by summarizing our first pilot efforts to combine the SCPF forums with citizen science activities pertaining to extreme heat; subsequent work has extended these activities to all four project hazards in nearly 30 US communities.

Wicked Hot Boston—Piloting the Science-to-Civics Framework for Extreme Heat

We implemented our first pilot of the science-to-civics framework over the summer of 2019 in a campaign called “Wicked Hot Boston” (Benson et al 2020; Sittenfeld et al 2021; Hostetler et al 2019). We engaged volunteers in participatory urban heat island mapping and making policy recommendations for mitigating the impacts of extreme heat events through resilience strategies identified in the SCPF heat waves deliberation module. The outcomes of our recently concluded pilot study will inform future modifications and expanded efforts at multiple sites around the US across all of the SCPF project hazards.

We began by convening meetings with resilience planning partners in Boston, Cambridge, and Brookline. These planners shared a need for high-resolution urban heat island data. While Climate Ready Boston, the City of Cambridge, and the Metropolitan Area Planning Council had previously conducted urban heat island mapping utilizing data satellites or aircraft (City of Cambridge 2017), high-resolution data and evening temperatures afforded by the CAPA strategies would be more helpful for vulnerability assessment and adaptation efforts. Through the support of NOAAs’ Climate Program Office, we partnered with researchers from CAPA Strategies, the Science Museum of Virginia, and Portland State University to follow the protocol for community mapping described in Shandas et al (2019). We also partnered with ISeeChange to engage dozens of others in an online extreme heat observation, eliciting community photographs, measurements, and observations of extreme heat. A portal on the SciStarter website engaged participants in making observations via ISeeChange, sharing opportunities for the heat mapping, and inviting them to the culminating Wicked Hot Boston Forum event.

Our participant recruitment for the citizen science activities (Figures 7 and 8) utilized a number of complementary methods. We participated in large in-person events such as community festivals and farmers’ markets, attended public meetings convened by our civic partners, and facilitated hands-on interpretations on the floor of the Museum. We also utilized social media and emails from the Museum and our civic and community partners and worked with local community groups in an effort to engage participants from diverse communities, particularly those especially vulnerable to extreme heat events. We successfully recruited approximately 50 volunteers to map 10 polygons in Boston, Cambridge, and Brookline in July and August of 2019. The study areas were determined in collaboration with our civic partners and also included elements suggested by community members at public meetings. Over 100 posts were also submitted in the Boston area Extreme Heat investigation on ISeeChange.

Wicked Hot Boston Recruitment Brochure
Figure 7 

Wicked Hot Boston Recruitment Brochure.

Wicked Hot Boston Citizen Scientists in Cambridge, MA
Figure 8 

Wicked Hot Boston Citizen Scientists in Cambridge, MA.

As described in the theoretical outline of the science-to-civics process, our Wicked Hot Boston Forum occurred in September 2019 and presented results of the citizen science activities to the broader community (Hostetler, Nickerson, Sittenfeld, and Benson, 2019). A poster session shared the community-generated heat maps and ISeeChange posts with public participants and community resilience planners. Then the SCPF heat module engaged participants in learning and thinking about proposed strategies to make communities more resilient to extreme heat events. Participants considered and discussed potential strategies for reducing temperatures through cooling centers or shade projects, protecting infrastructures such as electrical grids and transportation hubs, and prioritizing health and safety among those populations most vulnerable to extreme heat events. A panel of resilience planners from the three communities where the mapping was implemented shared their ongoing strategies for building resilience to extreme heat events and commented upon the participants’ recommendations from the SCPF deliberation. The Forums also served as recruitment tools for subsequent citizen science projects pertaining to other project hazards. For example, a subsequent campaign engaged Boston-area participants in photographing impacts of a Boston-area King Tide event, and then convened a deliberation employing the SCPF sea level rise Forum (Benson, Nickerson, and Sittenfeld 2021). We thus view the science-to-civics process as a continuing cycle of policy engagement, learning, and co-generation of knowledge as depicted in Figure 9 below.

The Citizen Science, Civics, and Resilient Communities (CSCRC) process of citizen science, public Forum deliberation, and resilience decision-making
Figure 9 

The Citizen Science, Civics, and Resilient Communities (CSCRC) process of citizen science, public Forum deliberation, and resilience decision-making.

After evaluating outcomes of our pilot study, our team made revisions in response to our formative evaluation and feedback from project partners. The model has since been extended across all four SCPF hazards and tested at the 8 SCPF institutions in the project’s second year, and then was disseminated to an additional 20 institutions for implementation in the culminating phase of the project. Through our project portal on SciStarter, we have observed more than 48,000 views of the project pages across the sites, and 1,400 clicks to make project contributions. We have also employed the science-to-civics framework in summer 2021 for heat mapping activities in the Mystic River Watershed, focusing on issues prioritized by community partners, including addressing historic redlining, eliciting qualitative perspectives from local residents, and visualizing differences across municipalities (Museum of Science 2022). Our forthcoming summative evaluation of the CSCRC project has assessed changes in citizen science and Forum participants’ learning and attitudes with respect to extreme heat and other hazard vulnerabilities and potential resilience strategies at different points along the science-to-civics trajectory. While the pandemic made assessment more challenging, we have found that the science-to-civics modules and the CSCRC process demonstrate potential to sustain engagement and amplify environmental literacy among Forum and citizen science participants, while contributing citizen-created data, local knowledge, and community values to hazard resilience planning. Our preliminary summative data indicate statistically significant learning increases across each of our learning objectives, as was the case for the SCPF Forums. These deliberative and citizen science components thus employ complementary active learning methodologies while connecting in a coordinated fashion to issues of local policy relevance and community resilience.

Data Accessibility Statement

In accordance with the IRB policies for this project and to protect the rights of participants, data from the post-surveys are not publicly available. However, anonymous data may be shared upon request. Please contact David Sittenfeld at dsittenfeld@mos.org with any data inquiries. SciStarter provides on-demand access of anonymous summary statistics to project organizers upon request. Individuals can view the number and frequency of their own contributions to affiliate projects via their SciStarter dashboard.

Acknowledgements

This resource was prepared by the Museum of Science, Boston and Northeastern University under award NA18SEC0080008 from the Environmental Literacy Program of the National Oceanic and Atmospheric Administration (NOAA), U.S. Department of Commerce.

The authors would like to thank Ira Bennett from Arizona State University for his support in conceptualizing these activities during the initial SCPF phase of the activities.

Competing Interests

The authors have no competing interests to declare.

References

  1. American Association for the Advancement of Science. 2016. Theory of Change for Public Engagement with Science. Online at https://www.aaas.org/sites/default/files/content_files/2016-09-15_PES_Theory-of-Change-for-Public-Engagement-with-Science_Final.pdf. 

  2. Beier, P, Hansen, LJ, Helbrecht, L and Behar, D. 2017. A How-to Guide for Coproduction of Actionable Science. Conservation Letters, 10: 288–296. DOI: https://doi.org/10.1111/conl.12300 

  3. Bell, L, Lowenthal, C, Sittenfeld, D, Todd, K, Pfeifle, S and Kollmann, EK. 2017. Public engagement with science: A guide to creating conversations among publics and scientists for mutual learning and societal decision-making. Boston, MA: Museum of Science. http://publicengagementwithscience.org/guide. 

  4. Benson, S, Sittenfeld, D, Shandas, V, Hoffman, J, Baur, K, Harrington, S and Cavalier, D. 2020. Wicked Hot Boston: Connecting Citizen Science to Extreme Heat Events Through Urban Heat Mapping and ISeeChange. American Meteorological Society Annual Meeting, Boston, MA. https://ams.confex.com/ams/2020Annual/webprogram/Paper370872.html. 

  5. Benson, S, Nickerson, C and Sittenfeld, D. 2021. “Wicked High Tides: Citizen Scientists Plan for Sea-Level Rise”. Online at https://www.discovermagazine.com/environment/wicked-high-tides-citizen-scientists-plan-for-sea-level-rise. 

  6. Bey, G, McDougall, C and Schoedinger, S. 2020. Report on the NOAA Office of Education Environmental Literacy Program Community Resilience Education Theory of Change. Washington, DC: National Oceanic and Atmospheric Administration. DOI: https://doi.org/10.25923/mh0g-5q69 

  7. Bonney, R, Ballard, H, Jordan, R, McCallie, E, Phillips, T, Shirk, J and Wilderman, CC. 2009. Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education. A CAISE Inquiry Group Report. Washington, DC: Center for Advancement of Informal Science Education (CAISE). 

  8. City of Cambridge, MA. 2017. Climate Change Vulnerability Assessment (CCVA) Report – Part 2. Retrieved from http://www.cambridgema.gov/CDD/Projects/Climate/~/media/F93208C3B12D4AACBD3E0F3A712F68C7.ashx. 

  9. County of Marin. “Game of Floods”. Online at https://www.marincounty.org/depts/cd/divisions/planning/csmart-sea-level-rise/game-of-floods. Accessed November 2021. 

  10. Fiorino, DJ. 1990. Citizen participation and environmental risk: A survey of institutional mechanisms. Science, Technology & Human Values, 15(2): 226–243. DOI: https://doi.org/10.1177/016224399001500204 

  11. Hong, L and Page, S. 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Nat. Acad of Sciences 101, no. 4616385–16389. DOI: https://doi.org/10.1073/pnas.0403723101 

  12. Hostetler, E, Nickerson, C, Sittenfeld, D and Benson, S. 2019. “Boston is WICKED hot. This is what they’re doing about it.” Discover. Online at https://www.discovermagazine.com/the-sciences/boston-could-become-wicked-hot-this-is-what-theyre-doing-about-it. 

  13. LabX and Koshland Science Museum. 2018. Extreme Event Game. Online at https://labx.org/extreme-event/. Accessed November 2021. 

  14. Lewenstein, B. 2012. Public Engagement. Online at CAISE website: http://www.informalscience.org/news-views/public-engagement. 

  15. McCallie, E, Bell, L, Lohwater, T, Falk, JH, Lehr, JL, Lewenstein, BV, Needham, C and Wiehe, B. 2009. Many Experts, Many Audiences: Public Engagement with Science and Informal Science Education. A CAISE Inquiry Group Report. Washington, DC: Center for Advancement of Informal Science Education (CAISE). http://caise.insci.org/uploads/docs/public_engagement_with_science.pdf. 

  16. Museum of Science. 2022. Wicked Hot Mystic Executive Summary. Retrieved from https://www.mos.org/explore/public-events/wicked-hot-mystic. 

  17. National Oceanographic and Atmospheric Administration. 2018. “Climate Resilience Toolkit: Steps to Resilience”. Retrieved from NOAA website at https://toolkit.climate.gov/#steps. 

  18. National Oceanographic and Oceanic Administration. 2017. NOAA Environmental Literacy Program Resilience Education Grantee Workshop. Retrieved from NOAA website at http://www.noaa.gov/sites/default/files/atoms/files/PDF-ELP2017GranteeWorkshopReport-110117-NOAA.pdf. 

  19. Newman, G, Wiggins, A, Crall, A, et al. 2012. Citizen science futures: emerging technologies and shifting paradigms. Frontiers in Ecology and the Environment, 10: 298–304. DOI: https://doi.org/10.1890/110294 

  20. NOAA Office of Education. 2018. “2018-2019 Environmental Literacy Grants”. Retrieved from https://www.noaa.gov/office-education/elp/grants/apply/2018-environmental-literacy-grants. 

  21. Office of Management and Budget (OMB). Study to Identify Methods to Assess Equity: Report to the President (July 2021). Available online at: https://www.whitehouse.gov/wp-content/uploads/2021/08/OMB-Report-on-E013985-Implementation_508-Compliant-Secure-v1.1.pdf. 

  22. Rennie, LJ and Johnston, DJ. 2007. Visitors’ perceptions of changes in their thinking about science and technology following a visit to science center. Visitor Studies, 10(2): 10. DOI: https://doi.org/10.1080/10645570701585194 

  23. Rowe, G and Frewer, L. 2005. A typology of public engagement mechanisms. Science, Technology, & Human Values, 30: 251–290. DOI: https://doi.org/10.1177/0162243904271724 

  24. Saldana, J. 2016. The Coding Manual for Qualitative Researchers. (3rd ed.). Sage. 

  25. Science Museum of Virginia. 2020. “Bridging Scientific Data and Lived Experiences to Equitably Prepare for Climate Change.” Retrieved from https://smv.org/learn/blog/bridging-scientific-data-and-lived-experiences-equitably-prepare-climate-change/. 

  26. Shandas, V, Voelkel, J, Williams, J and Hoffman, J. 2019. Integrating Satellite and Ground Measurements for Predicting Locations of Extreme Urban Heat. Climate, 7: 5. DOI: https://doi.org/10.3390/cli7010005 

  27. Shirk, JL, Ballard, HL, Wilderman, CC, Phillips, T, Wiggins, A, Jordan, R, McCallie, E, Minarchek, M, Lewenstein, BV, Krasny, ME and Bonney, R. 2012. Public participation in scientific research: a framework for deliberate design. Ecology and Society, 17(2): 29. DOI: https://doi.org/10.5751/ES-04705-170229 

  28. Sittenfeld, D, Smith, N, Benson, S, Drapkin, JK, Nickerson, C, Baur, K and Hoffman, JS. Wicked Hot Boston: Community Science for Building Extreme Heat Resilience and Addressing Public Health Disparities. American Meteorological Society 2021 Virtual Annual Meeting. https://ams.confex.com/ams/101ANNUAL/meetingapp.cgi/Paper/384318. 

  29. Spitzer, W and Fraser, J. 2020. Advancing Community Science Literacy. Journal of Museum Education, 45(1): 5–15. DOI: https://doi.org/10.1080/10598650.2020.1720403 

  30. Stirling, A. 2008. “Opening Up” and “Closing Down”: Power, Participation, and Pluralism in the Social Appraisal of Technology. Science, Technology & Human Values, 33(2): 262–294. DOI: https://doi.org/10.1177/0162243907311265 

  31. Storksdieck, M, Stylinski, C and Bailey, D. 2016. Typology for Public Engagement with Science: A Conceptual Framework for Public Engagement Involving Scientists. Corvallis, OR: Center for Research on Lifelong STEM Learning. Online at https://www.aaas.org/sites/default/files/content_files/AAAS_Typology.pdf. 

  32. Todd, K, Kollmann, EK, Gregory, M and Weitzman, O. 2018. Science Center Public Forums: Summative Evaluation Report. Boston, MA: Museum of Science, Boston. 

  33. Tomblin, D, Pirtle, Z, Farooque, M, Sittenfeld, D, Mahoney, E, Worthington, R, Gano, G, Gates, M, Bennett, I, Kessler, J, Kaminski, A, Lloyd, J and Guston, D. 2017. Integrating Public Deliberation into Engineering Systems: Participatory Technology Assessment of NASA’s Asteroid Redirect Mission. Astropolitics, 15(2): 141–166. DOI: https://doi.org/10.1080/14777622.2017.1340823 

  34. Worthington, R, Cavalier, D, Farooque, M, Gano, G, Geddes, H, Sander, S, Sittenfeld, D and Tomblin, D. 2012. Technology assessment and public participation: From TA to pTA: Expert and Citizen Assessment of Science and Technology (ECAST). Retrieved from https://ecastnetwork.files.wordpress.com/2013/01/ecast-report-ta-to-pta-rev1.pdf.