Start Submission

Reading: Maintaining User Engagement in an Infectious Disease Surveillance-Related Citizen Science Pr...

Download

A- A+
Alt. Display

Research Papers

Maintaining User Engagement in an Infectious Disease Surveillance-Related Citizen Science Project

Authors:

Shannon Smith,

Boston University School of Public Health, US
X close

Kara C. Sewalk ,

Boston Children’s Hospital, US
X close

Fernando Donaire,

Boston Children’s Hospital, US
X close

Lauren Goodwin,

Boston Children’s Hospital, US
X close

Ariel Zych,

Science Friday, US
X close

Adam W. Crawley,

Ending Pandemics, US
X close

John S. Brownstein,

Boston Children’s Hospital, US
X close

Kristin Baltrusaitis

Havard T.H. Chan School of Public Health, US
X close

Abstract

Over the past fifteen years, infectious disease surveillance has evolved to include online citizen science projects that implement active digital data collection. Flu Near You (FNY) is an online participatory surveillance system in the United States that collects weekly health reports from its volunteers. Because high engagement levels and consistent participation of users are essential to accurately track disease and estimate burden, FNY implemented three measures: (1) adding a new feature in that allows website visitors to submit a health report without registering, (2) administering user surveys at the end of each influenza season, and (3) partnering with Science Friday (SciFri) (a weekly public radio program that discusses topics relevant to science, nature, and technology) with the goals of increasing the number of weekly participant reports, improving the frequency of user reporting, and understanding user motivation. In this paper, we evaluate the impact of these three measures. The number of registered users who submitted only one report decreased after allowing users to submit reports without registering. The survey indicated that respondents were primarily motivated by the importance of disease tracking and by a desire to participate in a citizen science project. Finally, users who registered from SciFri were highly engaged and also motivated by a desire to participate in a citizen science project.

How to Cite: Smith, S., Sewalk, K.C., Donaire, F., Goodwin, L., Zych, A., Crawley, A.W., Brownstein, J.S. and Baltrusaitis, K., 2021. Maintaining User Engagement in an Infectious Disease Surveillance-Related Citizen Science Project. Citizen Science: Theory and Practice, 6(1), p.7. DOI: http://doi.org/10.5334/cstp.302
105
Views
36
Downloads
1
Twitter
  Published on 08 Mar 2021
 Accepted on 17 Nov 2020            Submitted on 10 Dec 2019

Background

Early detection is a vital aspect of preventing the spread of infectious diseases (Why early detection of outbreaks is so important. 2017). One such closely monitored disease is influenza, which is responsible for 140,000 to 960,000 hospitalizations and 12,000 to 79,000 deaths per year in the United States (U.S.) (Disease Burden of Influenza 2019). Most infectious disease surveillance systems are based on outpatient visits to healthcare providers and laboratory-confirmed cases. Although these sentinel surveillance systems provide information on where, when, and what influenza viruses are circulating, they lack the ability to capture instances of disease when medical care is not sought. Furthermore, they are not capable of presenting data in real time when there is a delay between the onset of symptoms and receiving medical care or when cases are not immediately reported to local health departments by clinicians (Thompson 2006; Paolotti et al. 2014).

Over the past two decades, the widespread proliferation of mobile devices and wearable technology has significantly changed the landscape of epidemiological data gathering and evolved into a field known as Digital Epidemiology (DE) (Salathé 2018; Choi et al. 2016). DE provides an informal, complementary approach to traditional sentinel surveillance methods by leveraging data generated outside of the public health system through digital data sources such as Google internet searches (Ginsberg et al. 2009), Twitter posts (Signorini et al. 2011; Dredze et al. 2014; Chen et al. 2016), and cloud-based electronic health records (Santillana et al. 2016). While these alternative data sources provide real time information about trends and general patterns of disease activity, the data is not generated specifically for epidemiologic purposes (Salathé 2018).

One source of active digital data collection is online participatory syndromic surveillance systems (Smolinski et al. 2017). Through these systems, participants volunteer to report health information via online or mobile communication technologies on a weekly basis. The first of these systems, de Grote Griepmeting, or the Great Influenza Survey, started in 2003 in the Netherlands and Belgium (Marquet et al. 2006). Since that time, multiple systems have been established throughout Europe, Australia, the U.S., and Japan (Paolotti et al. 2014; Dalton et al. 2009; Smolinski et al. 2015; Fujibayashi et al. 2018). These systems actively engage the general public in reporting and provide timely information about disease trends within the community, thereby providing a mechanism for members of the community to become citizen scientists (Wójcik et al. 2014; Smolinski et al. 2017b, Kullenberg and Kasperowski 2016). Flu Near You (FNY) is a U.S.-based online participatory syndromic surveillance system that was developed in 2011 through collaboration between HealthMap of Boston Children’s Hospital and Ending Pandemics (formally Skoll Global Threats Fund) (Smolinski et al. 2015; https://flunearyou.org). In exchange for participating in FNY, users can visualize local influenza-like illness (ILI) activity on maps, connect with local public health organizations, and find nearby locations offering influenza vaccines.

Two important issues for any citizen science system are motivating contributions from participants and ensuring the quality of these contributions (Prestopnik et al. 2017). For participatory syndromic surveillance systems, consistent participation by users is necessary to accurately track ILI over time and produce end-of-year estimates of ILI burden. However, many users of these systems report only once, and these one-time users are more likely to report ILI symptoms (Baltrusaitis et al. 2017). In August 2017, FNY added a new feature that allows website visitors to submit a health report without registering. Allowing reports without registration was aimed at reducing the initial barrier to participaton presented by the registration process and potentially reducing the bias from one-time reports.

Understanding the motivations of citizen science participants is key to attracting new participants and retaining old ones (Prestopnik and Crowston 2011). Starting with the 2015–2016 influenza season, FNY began administering user surveys at the end of each influenza season. These surveys collect information on reporting habits, on how users interact with the site, and on users’ motivations for joining and continuing to use FNY. The surveys also ask about demographic information not collected upon registration with the site, which allows for further understanding of the representativeness of the FNY user base. In addition, these surveys provide users with the opportunity to give feedback about their overall experiences with the platform.

Previous studies related to motivations for initial and continued engagement in citizen science projects have focused on intrinsic motivations, such as preexisting enthusiasm for the scientific topic and gaining a sense of helping, versus extrinsic motivations, such as game-based rewards and monetary compensation (Aristeidou et al. 2017; Rotman et al. 2012; Reed et al. 2013; Raddick et al. 2011; Nov et al. 2011a; Iacovides et al. 2013; Crowston and Prestopnik 2013). Results suggest that members with intrinsic motives have enhanced participation frequency and longer participation (Nov et al. 2011b; Borst 2010; Eveleigh et al. 2014). During the 2018–2019 influenza season, FNY partnered with Science Friday (SciFri), a weekly national public radio program that discusses topics relevant to science, nature, and technology (About SciFri 2019). The goal of this partnership was to provide SciFri listeners and current FNY users with factual information about influenza virus, symptoms, spread, and vaccination in the form of two half-hour interviews with influenza experts and high-risk patients broadcast as part of the public radio show and podcast. Online, broadcast, and SMS text content addressed misconceptions about influenza and vaccines and promoted a better understanding of disease surveillance and data literacy. This collaboration aimed to register new FNY users who are science enthusiasts with strong intrinsic motivations, and increase user participation through SMS text-based reminders.

In this paper, we summarize the effectiveness of the guest user feature in increasing the number of weekly symptom reports and in improving the frequency of registered user reporting. Specifically, we compare the total number of weekly reports, the total number of registered participants, and the median number of reports per participant before and after the implementation of this feature. We also summarize yearly user survey data for four influenza seasons (2015–2016 to 2018–2019) and compare results from year to year to demonstrate if and how the user base and motivations are changing. Finally, we compare the frequency of user reporting and the motivations of users who registered through the SciFri page with those of users who registered through the FNY website.

Methods

Data collection

Following registration, FNY users are sent a reminder email each Monday with a link to a brief weekly report. These reports first ask how the user is feeling. If a user reports “Not feeling well,” the user has the opportunity to select symptoms they experienced during the previous week, e.g., headache, body aches, diarrhea, fatigue, chills, nausea, shortness of breath, rash, and runny nose. (Figure 1). The user is then asked follow-up questions about the date of illness onset and whether the user sought a health professional for care. At the start of each season, users are also asked if they have received the influenza vaccine. We define participants as FNY users and their household members. Influenza surveillance seasons were defined as Morbidity and Mortality Weekly Report (MMWR) week 40 through week 20, which typically corresponds with the start of October through mid-May. Our study of FNY data received approval from the Boston Children Hospital’s Institutional Review Board.

Figure 1 

Screenshots of Flu Near You.

Guest user analysis

The implementation of the guest user feature was analyzed to understand the frequency of user reporting and participant representation before and after this change. FNY data from two influenza seasons before the implementation (2015–2016 and 2016–2017) and two influenza seasons after the implementation (2017–2018 and 2018–2019) were included in the analysis. We summarized the number of participants, the number of users, the total number of FNY symptom reports, the median symptom reports per week, and the number of reports per participant for registered participants only across all four influenza seasons as well as for registered participants and guest users combined for the 2017–2018 and 2018–2019 influenza seasons. We also compared the demographics of registered participants across all four influenza seasons. Continuous variables are presented as median (25th percentile, 75th percentile), and categorical variables are presented as n (%). Significant differences across seasons were assessed using the Kruskal-Wallis test for continuous variables and Chi-square tests for categorical variables.

Survey data analysis

Surveys for each influenza season were created using SurveyMonkey or SurveyGizmo. Surveys were administered to registered FNY users in June of each study year. Prior to distribution, survey questions were fine-tuned with input from contributing FNY partners and stakeholders including individuals from local health departments, the Centers for Disease Control and Prevention (CDC), and universities. Surveys from the previous four influenza seasons, 2015–2016 through 2018–2019, were used in the analysis.

Only users with complete surveys were included in the analysis. Summaries of responses to selected survey questions were tabulated for each year. Questions included in the analysis were

  1. “How likely are you to recommend Flu Near You on a scale of 0 to 10 (with 0 being least likely and 10 being most likely)?”
  2. “Why did you sign up for Flu Near You?”
  3. and “What motivates you to continue using Flu Near You?”

Questions about users’ influenza vaccination status, whether FNY influenced them to get vaccinated, and whether FNY could influence those who were not vaccinated to receive a vaccine in the future were also analyzed. In addition, information about users’ reported level of educational attainment and occupation were summarized. Results were compared across each study year. Questions discontinued by the most recent FNY survey (2018–2019) were excluded from the analysis.

Each survey respondent’s number of symptom reports submitted to FNY per year was compared with the survey question “How frequently do you report to Flu Near You?” Survey respondents were compared with all registered FNY users to examine how they differed by number of reports. For this analysis, only users registered as of June 1, 2019 were included. When analyzing influenza seasons before the 2018–2019 season, users who registered after that season were excluded from the analysis. For example, when comparing 2015–2016 survey respondents with all FNY users, users who registered during the 2016–2017 season or later were excluded.

Science Friday listener analysis

SciFri listeners registered for FNY using a unique landing page that tagged users in the database as SciFri users. SciFri listeners also could sign up for weekly text reminders that included facts and tips related to influenza. All FNY participants were able to track the weekly percent ILI activity for the SciFri cohort, the FNY cohort, and the CDC Influenza-Like Illness Surveillance Network (CDC-ILINet) through a time-series visualization embedded on the FNY website (CDC 2019). We summarized the number of participants, the number of users, the total number of symptom reports, the median symptom reports per week, and the number of reports per participant for registered participants. Additionally, annual user survey responses from the 2018–2019 season from SciFri users were compared with those from users who did not register through SciFri. Chi-square tests of independence were used to assess differences in responses between these two groups.

Results

Guest user analysis

The descriptive statistics of FNY reporting for the four influenza seasons are shown in Table 1. The 2015–2016 influenza season had more registered participants (n = 47,893) compared with the 2016–2017 (n = 30,638), 2017–2018 (n = 31,558), and 2018–2019 (n = 29,849) influenza seasons. However, the 2015–2016 influenza season had the smallest median number of reports per registered participant—2 (1, 12). Following this season, the median number of reports per registered participant increased to 8 (1, 24). This change in reporting habits is captured in Figure 2, which shows that the relative percentage of registered participants who submitted only one report has decreased since the 2015–2016 influenza season. Although the 2018–2019 influenza season had the fewest number of registered participants, it had the greatest median number of reports per week from registered participants—11,561 (10,535, 12,022). Across all four seasons, approximately two-thirds of the registered participants were primary users. During the 2017–2018 influenza season, there were 132,504 reports from guest users. The number of reports from guest users during the 2018–2019 influenza season decreased to 55,890. Although these guest reports increased the median number of weekly reports, the median number of reports per participant decreased to 1.

Table 1

Descriptive statistics of Flu Near You (FNY) reports for the 2015–2016, 2016–2017, 2017–2018, and 2018–2019 influenza seasons. Continuous variables are displayed as median (25th quartile, 75th quartile), and categorical variables are displayed as n (%).


VARIABLE 2015–2016 2016–2017 2017–2018 2018–2019

REGISTERED REGISTERED REGISTERED REGISTERED + GUEST REGISTERED REGISTERED + GUEST

Total number of participants1 47,893 30,638 31,558 164,062 29,849 85,739

Number of users2 33,011 (68.9%) 20,183 (65.9%) 20,588 (65.2%) 153,088 (93.3%) 19,824 (66.4%) 75,708 (88.3%)

Total number of reports 378,919 333,628 350,636 483,142 372,363 428,280

Weekly reports 11,449 (10,834, 12,160) 10,433 (9823, 11,290) 10,416 (9895, 11,189) 12,208 (11,178, 15,642) 11,561 (10,535, 12,022) 13,115 (11,379, 14,259)

Number of reports per participant 2 (1, 12) 4 (1, 23) 5 (1, 23) 1 (1, 1) 8 (1, 24) 1 (1, 2)

1 Participants are defined as FNY users and registered household members.

2 Users are defined as individuals who submit a FNY report.

Figure 2 

Distributions of the number of registered FNY participant reports during the (a) 2015–2016, (b) 2016–2017, (c) 2017–2018, and (d) 2018–2019 influenza seasons.

Table 2 shows the descriptive demographic statistics of registered FNY participants for all four influenza seasons. Since the 2015–2016 influenza season, the percentage of male participants increased significantly from 34% to 39% (p < 0.001). Although the median age of all participants increased from 50.8 (34.4, 62.3) to 52.8 (33.6, 64.4), the percentage of participants in the 5–17 age group increased significantly from 8.47% to 12.94% (p < 0.001).

Table 2

Descriptive statistics of registered Flu Near You (FNY) participant demographics for the 2015–2016, 2016–2017, 2017–2018, and 2018–2019 seasons. Continuous variables are displayed as median (25th quartile, 75th quartile), and categorical variables are displayed as n (%).


VARIABLE LEVEL 2015–2016 2016–2017 2017–2018 2018–2019 P-VALUE

Total N 47,893 30,638 31,558 29,849

Sex Male 16,258 (34.33%) 11,295 (36.87%) 11,715 (37.16%) 11,660 (39.11%)

Female 29,679 (62.67%) 18,636 (60.83%) 18,712 (59.36%) 17,506 (58.73%)

Unspecified 1,418 (2.99%) 707 (2.31%) 1,097 (3.48%) 644 (2.16%) < 0.0011

Age Median (IQR) 50.8 (34.4, 62.3) 53.6 (36.4, 64.1) 52.8 (35.7, 64.3) 52.8 (33.6, 64.4) < 0.0012

Age group < 5 1,006 (2.1%) 538 (1.76%) 499 (1.58%) 528 (1.77%)

5–17 4,055 (8.47%) 2,914 (9.51%) 3,302 (10.46%) 3,861 (12.94%)

18–49 16,349 (34.14%) 9,310 (30.39%) 9,889 (31.34%) 8,806 (29.5%)

50–64 14,516 (30.31%) 10,038 (32.76%) 9,774 (30.97%) 9,055 (30.34%)

65+ 8,366 (17.47%) 6,727 (21.96%) 7,080 (22.43%) 6,922 (23.19%)

Unspecified 3,601 (7.52%) 1,111 (3.63%) 1,014 (3.21%) 677 (2.27%) < 0.0011

1 Chi-square test.

2 Kruskal-Wallis test.

User survey data analysis

The analysis of survey data included 4,711 responses for the 2015–2016 influenza season, 1,333 responses for 2016–2017, 4,519 responses for 2017–2018, and 5,128 responses for 2018–2019. A summary of all survey questions with the number and proportion of responses per year is shown in Table 3. When asked how likely on a scale of 0 to 10 users were to recommend FNY to a friend or colleague, with 0 being least likely and 10 being most likely, survey respondents were likely to recommend the service overall, with 10 being the most common response for all four influenza seasons. Responses to this question by year can be seen in Figure 3. There was a clear increase in the number of participants who responded 10 to this question, as 1,957 out of 4,711 (41.54%) 2015–2016 respondents reported that they would rate their likeliness to recommend FNY a 10 compared with 2,711 out of 5,128 (52.87%) 2018–2019 respondents. The first influenza season where greater than half of participants rated their likeliness to recommend FNY a 10 was 2018–2019. Responses in the middle of the scale (3–8) decreased, as 399 out of 4,711 (8.19%) 2015–2016 respondents rated their likeliness to recommend FNY a 5 compared with 304 out of 5,128 (5.93%) 2018–2019 respondents, with decreases observed between each influenza season. Responses at the lowest end of the scale (0–2) were fairly consistent across years and were low overall; less than 2% of participants each year rated their likeliness to recommend FNY to a friend or colleague a 0, 1, or 2.

Table 3

2018–2019 user survey responses. Values are represented as n (%).


RESPONSES N (%)

INFLUENZA SEASON

SURVEY QUESTION RESPONSE 2015–2016 2016–2017 2017–2018 2018–2019

N 4,711 1,333 4,519 5,128

On a scale of 0 to 10, with 0 being “very unlikely” and 10 being “very likely,” how likely would you be to recommend Flu Near You to a friend? 5 or lower 627 (13.31) 111 (8.35) 519 (11.48) 542 (10.56)

6 217 (4.61) 46 (3.45) 173 (3.83) 179 (3.49)

7 476 (10.10) 107 (8.03) 330 (7.30) 331 (6.45)

8 779 (16.54) 209 (15.68) 700 (15.49) 709 (13.83)

9 655 (13.90) 195 (14.63) 623 (13.78) 656 (12.79)

10 1,957 (41.54) 665 (49.89) 2174 (48.11) 2,711 (52.87)

How did you first hear about Flu Near You? Doctor or healthcare worker 77 (1.63) 29 (2.18) 60 (1.33) 54 (1.05)

Family or friend 391 (8.30) 98 (7.35) 288 (6.37) 219 (4.27)

News story (TV, radio, newspaper) 589 (12.50) 166 (12.45) 346 (7.66) 1,053 (20.52)

School/university 151 (3.21) 27 (2.03) 99 (2.19) 83 (1.62)

Workplace 315 (6.69) 89 (6.68) 243 (5.38) 189 (3.64)

Facebook 1,031 (21.88) 268 (20.11) 377 (8.34) 344 (6.71)

Internet search 1,208 (25.64) 323 (24.23) 605 (13.39) 534 (10.41)

Twitter 20 (0.42) 10 (0.75) 14 (0.31) 12 (0.23)

Other 929 (19.72) 323 (24.23) 202 (4.47) 453 (8.83)

I don’t remember N/A N/A 2,285 (50.56) 2,187 (42.65)

Why did you sign up for Flu Near You? Please select your primary motivation from the list below. I consider disease tracking to be important and wanted to help 3,312 (70.30) 963 (72.24) N/A N/A

I want to know when people around me are sick 998 (21.18) 287 (21.53) N/A N/A

I wanted to learn more about the flu 123 (2.61) 22 (1.65) N/A N/A

I was sick and wanted to submit this information 134 (2.84) 27 (2.03) N/A N/A

Other 144 (3.06) 34 (2.55) N/A N/A

Why did you sign up for Flu Near You? Please check all that apply. I was sick and wanted to submit this information N/A N/A 275 (6.09) 264 (5.15)

I wanted to learn more about the flu N/A N/A 879 (19.45) 971 (18.94)

I want to know when people around me are sick N/A N/A 2,302 (50.94) 2,393 (46.67)

I consider disease tracking to be important and wanted to help N/A N/A 3,737 (82.70) 4,270 (83.27)

I wanted to take part in a citizen science project N/A N/A 1,820 (40.27) 2,658 (51.83)

I joined as a part of a study or promotion from my area health department N/A N/A 112 (2.45) 112 (2.18)

A friend or colleague recommended that I join N/A N/A 211 (4.67) 175 (3.41)

Other N/A N/A 115 (2.54) 119 (2.32)

Did you receive a seasonal influenza vaccine for the most recent flu season? Yes 3,671 (77.92) 1,085 (81.40) 3,649 (80.75) 4,370 (85.22)

No 1,025 (21.76) 246 (18.45) 857 (18.96) 739 (14.41)

I don’t remember 15 (0.32) 2 (0.15) 13 (0.29) 19 (0.37)

Did participating in Flu Near You influence your decision to get a flu vaccine for this most recent flu season? Planned to get vaccine, Flu Near You helped to remind 555 (15.14) 155 (14.29) 453 (12.42) 506 (11.59)

No-would have gotten vaccine regardless 3,047 (83.12) 910 (83.87) 3,106 (85.17) 3,750 (85.81)

Probably would not have gotten vaccine without Flu Near You 64 (1.76) 20 (1.84) 67 (1.84) 90 (2.06)

Not applicable N/A N/A 21 (0.58) 24 (0.55)

Do you think that continued participation in Flu Near You might make you more likely to get an influenza vaccine in the future? No plans to vaccinate 419 (40.88) 103 (41.87) 388 (45.27) 323 (43.71)

No—may vaccinate, not because of Flu Near You 336 (32.70) 77 (31.30) 287 (33.49) 221 (29.91)

Yes 270 (26.34) 66 (26.83) 182 (21.24) 195 (26.39)

For classification purposes only, please indicate your level of education. Less than high school degree 32 (0.68) 13 (0.98) 21 (0.46) 14 (0.27)

High school degree 962 (20.42) 310 (23.26) 786 (17.39) 772 (15.05)

College degree 1934 (41.04) 510 (38.26) 1,786 (39.52) 2,077 (40.50)

Graduate degree 1,744 (37.03) 490 (36.76) 1,843 (40.78) 2,188 (42.67)

Rather not say 38 (0.81) 10 (0.75) 83 (1.84) 77 (1.50)

For classification purposes only, please indicate your occupation. Accommodation and food service 23 (0.49) 2 (0.15) 17 (0.38) 19 (0.37)

Administrative and support 115 (3.29) 43 (3.22) 161 (3.56) 152 (2.96)

Agriculture, forestry, fishing, and hunting 22 (0.47) 3 (0.23) 27 (0.60) 31 (0.60)

Arts, entertainment, and recreation 81 (1.72) 20 (1.50) 75 (1.66) 109 (2.13)

Construction 35 (0.74) 5 (0.38) 25 34 (0.66)

Educational services 406 (8.62) 117 (8.78) 444 (9.83) 534 (10.41)

Finance and insurance 89 (1.89) 16 (1.20) 72 (1.59) 99 (1.93)

Health care and social assistance 880 (18.68) 225 (16.88) 725 (16.04) 732 (14.27)

Information 101 (2.14) 32 (2.40) 124 (2.74) 139 (2.71)

Management of companies and enterprises 52 (1.10) 11 (0.83) 59 (1.31) 33 (0.64)

Manufacturing 60 (1.27) 18 (1.35) 61 (1.35) 66 (1.29)

Military/armed forces 12 (0.25) 1 (0.08) 10 (0.22) 13 (0.25)

Mining, quarrying, and oil and gas extraction 6 (0.13) 1 (0.08) 4 (0.09) 2 (0.04)

Other services 280 (5.94) 72 (5.40) 218 (4.82) 233 (4.54)

Professional, scientific, and technical services 443 (9.41) 106 (7.95) 436 (9.65) 553 (10.78)

Public administration 81 (1.72) 30 (2.25) 80 (1.77) 95 (1.85)

Real estate and rental and leasing 49 (1.04) 12 (0.90) 40 (0.86) 49 (1.00)

Retail trade 85 (1.80) 18 (1.35) 83 (1.84) 103 (2.01)

Retired 1,583 (33.61) 544 (40.81) 1,638 (36.25) 1,909 (37.23)

Student 55 (1.17) 12 (0.90) 29 (0.64) 39 (0.76)

Transportation and warehousing 23 (0.49) 5 (0.38) 32 (0.71) 37 (0.72)

Unemployed 173 (3.67) 34 (2.55) 134 (2.97) 119 (2.32)

Utilities 10 (0.21) 4 (0.30) 11 (0.24) 16 (0.31)

Wholesale trade 6 (0.13) 2 (0.15) 14 (0.31) 12 (0.23)

Figure 3 

Proportion of responses to the survey question “How likely are you to recommend Flu Near You to a friend or colleague?” by influenza season. FNY: Flu Near You.

When asked how they first heard about FNY, “Internet search” was the most common response from 2015–2016 to 2017–2018. Many users said that they first heard about FNY through Facebook for the 2015–2016 and 2016–2017 influenza seasons (1,031 out of 4,711 [21.88%] and 268 out of 1,333 [20.11%], respectively), but only 377 out of 4,519 (8.34%) 2017–2018 respondents and 344 out of 5,128 (6.71%) 2018–2019 respondents reported Facebook as their source. “I can’t remember” was added as an option for the 2017–2018 influenza season, and about half chose this response during that season. It continued to be the most common response for the 2018–2019 influenza season, though there was a decrease from 2,285 out of 4,519 (50.56%) to 2,187 out of 5,128 (42.65%). During the 2018–2019 influenza season, 1,053 out of 5,128 (20.52%) respondents said that they had heard about FNY from a news story, which was an increase from the previous year, when this option was selected by 346 out of 4,519 (7.66%) users, and from the previous high of 589 out of 4,711 (12.50%) users for the 2015–2016 influenza season.

The most common reason that users reported signing up for FNY across all four influenza seasons was “I consider disease tracking to be important and wanted to help.” During the first two influenza seasons included in the study, users were asked to choose one option only, during which time the percentage of users who selected this response was 3,312 out of 4,711 (70.30%) for 2015–2016 and 963 out of 1,333 (72.24%) for 2016–2017. For the 2017–2018 season, additional responses were added, and users were able to select more than one response, at which point the percentage of those who chose this response increased to 3,737 out of 4,519 (82.70%). The second most common response until 2017–2018 was “I want to know when people around me are sick.” This response was overtaken during the most recent flu season by “I wanted to take part in a citizen science project,” a response added for the 2017–2018 influenza season. Users’ motivations to continue using FNY were similar to their reason for signing up. “Flu Near You provides valuable information that I want to contribute to” was the most common reason given in the first two seasons included the study, for which users had to choose only one response between two (the other being “To stay informed about disease activity in my area,” which received about one-third of responses for both years).

For the 2017–2018 and 2018–2019 influenza seasons, more options were added that were worded similarly to those for the question about signing up for FNY, and users were asked to select any response they found applicable. “I consider disease tracking to be important and I want to help” was the most common response. The second-most-common response in 2017–2018 was “I want to know when people around me are sick;” similar to the responses to the question about signing up for FNY, this response was overtaken by “I enjoy contributing to a citizen science project” for the 2018–2019 influenza season. There were 746 out of 2,923 (28.07%) users who selected this option and who reported first hearing about FNY from a news source, compared with 307 out of 2,205 (13.92%) who did not.

Users were asked to report whether they received an influenza vaccine during the most recent season. Most users across the four-year study period reported having been vaccinated, with a range of “yes” responses between 3,671 out of 4,711 (77.92%) for the 2015–2016 influenza season and 4,370 out of 5,128 (85.61%) in 2018–2019. The 2018–2019 influenza season had the highest proportion of “yes” responses, with an increase of 4.21% compared with the year with the second-highest proportion of “yes” responses (2016–2017). When users answered “yes” to this question, they were asked “Did participating in FNY influence your decision to get a flu vaccine for this most recent flu season?” The majority of respondents reported that they would have received a vaccine regardless of their participation in FNY; this response was consistent across influenza seasons, with a percentage range of 83.12% (2015–2016) to 85.51% (2018–2019). Between 11.89% and 15.14% of respondents across the four influenza seasons reported that they had been planning to get the vaccine and FNY helped remind them to do so. Each year, 1.59% to 2.06% of survey respondents reported that they would not have received the influenza vaccine if they were not reminded to do so from FNY. Users who responded “no” to the original question about their vaccine status were asked, “Do you think that continued participation in Flu Near You might make you more likely to get an influenza vaccine in the future?” The most common response to this question, with between 40.88% and 45.27% of unvaccinated users reporting, was “No, I did not get the vaccine and have no plans to do so in the future.” Between 21.24% and 26.83% of users reported that FNY may influence them to receive a vaccine in the future.

Overall, FNY survey respondents report high educational attainment. Fewer than 1% of users each year reported that they had less than a high school degree, and greater than 75% each year reported having at least a college degree. Between 36.76% and 42.67% of users each year reported having a graduate-level education. For occupation, the most common response selected each influenza season was “retired,” with 1,909 out of 5,128 (36.32%) 2018–2019 respondents reporting they were retired. The second most commonly selected occupational category was health care and social assistance, which received at least 10% of responses during all four influenza seasons. Other categories that received many responses each year were educational services and professional, scientific, and technical services.

Most users responded that they are just as likely to report to FNY when they are healthy as they are when they are sick during all four influenza seasons. However, as shown in Figure 4, the percentage of users who selected this option increased across each study year. For the 2015–2016 season, 3,122 out of 4,711 (66.28%) users reported that they were just as likely to report when healthy compared with 4,156 out of 5,128 (81.05%) for the most recent influenza season, and there were clear increases between each season. The percentage of users who report only when sick dropped from 1,532 out of 4,711 (32.53%) for the 2015–2016 influenza season to 642 out of 5,128 (18.37%) for the most recent season. Most users said that when they report symptoms, they report any symptoms that they’ve had during the week and not just those that they believe are associated with the flu. However, 22.59% to 24.46% of users still said that they report only influenza symptoms. There was a higher percentage of users during the most recent influenza season who said that they have never reported symptoms to FNY (500 out of 5,128, or 9.75%) compared with previous years, possibly reflecting the increase in those who report whether they are healthy or sick. In terms of what symptoms FNY users associate with influenza, the most definitive symptoms, with over 75% of users selecting these options during every influenza season, were fatigue, fever, and body aches. Chills/night sweats, sore throat, cough, and headache were also commonly associated with influenza; over 50% of users selected these as perceived influenza symptoms during every season.

Figure 4 

Proportion of responses to the survey question “Do you consider yourself more or less likely to report to FNY when you have symptoms of illness?” by influenza season. FNY: Flu Near You.

Most survey respondents self-reported that they submitted reports to FNY “every week or almost every week,” with 3,991 out of 4,979 (80.16%) users selecting this option for the 2018–2019 influenza season. Survey responses to this question were compared with the actual number of reports that users submitted to FNY for the four influenza seasons, and it was found that answers to this question are fairly accurate. For the 2018–2019 season, users who said they submitted reports “every week or almost every week” submitted an average of 25.61 (6.87) reports out of the 33 weeks of the influenza season included in the analysis, which corresponds to a mean participation rate of approximately 78%. This value is compared with a participation rate of approximately 46% for those who said they reported “at least twice a month,” 18% for those who said they reported “sometimes,” and 8% for those who said they report to FNY “not very often” or “never.” Similar trends were also seen for the 2015–2016, 2016–2017, and 2017–2019 influenza seasons. These results are detailed in Table 4.

Table 4

Number of responses to the question “How often do you report to Flu Near You?” Mean number of reports out of 521 and mean participation rate2 for survey respondents for four influenza seasons. Number of responses is reported as n (%), and average number of reports is presented as mean (standard deviation).


RESPONSE

NEVER/NOT VERY OFTEN SOMETIMES MOST OF THE TIME EVERY WEEK OR ALMOST EVERY WEEK

2015–2016

Responses 57 (1.55) 163 (4.44) 698 (19.02) 2,752 (74.99)

Mean number of reports 5.77 (6.81) 10.25 (9.72) 24.98 (12.82) 38.26 (11.83)

Mean participation rate 11.10% 19.71% 48.03% 73.58%

2016–2017

Responses 24 (1.81) 44 (3.31) 203 (15.23) 1,057 (79.59)

Mean (sd) number of reports 2.50 (3.38) 7.86 (8.15) 25.01 (14.02) 41.06 (10.64)

Mean participation rate 4.81% 15.12% 48.11% 78.96%

2017–2018

Responses 63 (1.40) 227 (5.03) 765 (16.96) 3,456 (76.61)

Mean (sd) number of reports 5.83 (8.56) 10.93 (10.41) 26.23 (13.07) 42.90 (9.47)

Mean participation rate 11.20% 21.03% 50.44% 82.51%

2018–2019

Responses 49 (0.98) 221 (4.44) 718 (14.42) 3,991 (80.16)

Mean (sd) number of reports 2.6078 (4.9156) 5.9012 (5.9623) 15.0615 (8.7417) 25.6110 (6.8659)

Mean participation rate 7.90% 17.89% 45.64% 77.61%

1 For 2018–2019, the average number of reports is out of 33, not 52.

2 Defined as average number of reports divided by 52 (33 for 2018–2019).

Science Friday listener analysis

During the study period, 3,196 SciFri listeners registered for FNY, and 2,982 registrants submitted at least one symptom report. An additional 588 household members were registered by users, bringing the total SciFri cohort to 3,570 participants. Overall, 45,702 symptom reports were submitted by SciFri participants, and the median number of symptom reports per week was 1,644 (1,535, 1,710). SciFri users submitted a median of 11 reports (2, 21) during the 2018–2019 season, which corresponds with a mean participation rate of 40.74%. As shown in Figure 5, symptom reports submitted by SciFri participants made up approximately 10–15% of the total weekly FNY symptom reports. Compared with non-SciFri-registered FNY users who submitted at least one symptom report during the 2018–2019 influenza season, SciFri users were younger—55.2 (35.3, 66.3) versus 57.8 (44.8, 66.6) and had a greater percentage of males (38.5% versus 29.5%).

Figure 5 

Number of symptom reports per week for all registered Flu Near You (FNY) participants and guests, registered FNY participants, and SciFri participants during the 2018–2019 season.

There were 990 out of 2983 (33.2%) SciFri users who completed the annual 2018–2019 user survey, and they represented 19.31% of all 2018–2019 survey respondents during this year. This cohort was compared with survey respondents who did not sign up through SciFri (Table 5). SciFri users rated their likeliness to recommend FNY similarly to non-SciFri users and also gave high scores overall; 519 out of 990 (52.42%) rated their likeliness to recommend the service a 10, compared with 2,129 out of 4,138 (52.97%) non-SciFri users. There were 88 out of 990 (8.89%) SciFri users and 454 out of 4,138 (10.97%) non-SciFri users who gave a score of 5 or lower.

Table 5

2018–2019 user survey responses and p-values for SciFri users versus non-SciFri users. Values are represented as n (%).


SURVEY QUESTION RESPONSE SCIFRI (n = 990) NON-SCIFRI (n = 4,138) P-VALUE

On a scale of 0 to 10, with 0 being “very unlikely” and 10 being “very likely,” how likely would you be to recommend Flu Near You to a friend? 5 or lower 88 (8.89) 454 (10.97) 0.072

6 43 (4.34) 136 (3.29)

7 57 (5.76) 274 (6.62)

8 138 (13.94) 571 (13.80)

9 145 (14.65) 511 (12.35)

10 519 (52.42) 2,192 (52.97)

Why did you sign up for Flu Near You? Please check all that apply. I was sick and wanted to submit this information 18 (1.82) 246 (5.94) < 0.001

I wanted to learn more about the flu 133 (13.43) 838 (20.25) < 0.001

I want to know when people around me are sick 283 (28.56) 2,110 (50.99) < 0.001

I consider disease tracking to be important and wanted to help 885 (86.36) 3,385 (81.80) < 0.001

I wanted to take part in a citizen science project 768 (77.58) 1,890 (45.67) < 0.001

I joined as a part of a study or promotion from my area health department 8 (0.81) 104 (2.51) 0.001

A friend or colleague recommended that I join 7 (0.71) 168 (4.06) < 0.001

Other 22 (2.22) 97 (2.34) 0.911

Did you receive a seasonal influenza vaccine for the most recent flu season? Yes 881 (88.99) 3,489 (84.32) 0.001

No 106 (10.71) 633 (15.30)

I don’t remember 3 (0.30) 16 (0.39)

For classification purposes only, please indicate your level of education. Less than high school degree 0 14 (0.34) < 0.001

High school degree 69 (6.96) 703 (16.99)

College degree 390 (39.39) 1,687 (40.77)

Graduate degree 519 (52.42) 1,669 (40.33)

Rather not say 12 (1.21) 65 (1.57)

The reasons for joining the site differed between SciFri users and non-SciFri users. While believing in the importance of disease tracking was the most commonly selected reason in both groups, non-SciFri users weremore likely to select “I want to know why people around me are sick;” 2,129 out of 4,138 (50.99%) non-SciFri users selected this option, compared with 283 out of 990 (28.56%) SciFri users. More SciFri users (768 out of 990, or 77.58%) reported being motivated by wanting to participate in a citizen science project than non-SciFri users (1,890 out of 4,138, or 45.67%).

SciFri users were slightly more likely to report having been vaccinated for influenza than non-SciFri users. A total of 881 out of 990 (88.99%) users reported that they had received a vaccine for the most recent influenza season compared with 3,489 out of 4,138 (84.32%) non-SciFri users. SciFri users are highly educated: 519 out of 990 (52.42%) survey respondents who signed up from SciFri had graduate degrees, compared with 1,669 out of 4,138 (40.33%) survey respondents who did not sign up from SciFri.

Discussion

Citizen science projects have the potential to complement traditional sentinel surveillance systems by reaching populations that do not access health care systems and areas with limited surveillance data; however, the success of participatory syndromic surveillance systems depends on participants who are dedicated to providing accurate data (Nov, Arazy, and Anderson 2014). Citizen science initiatives play a unique role in supporting public health, particularly for disease surveillance activities (Dalton et al. 2009). This role is two-fold in that the data that individuals provide increases our collective understanding of disease risk and transmission patterns while the direct engagement of individuals offers an opportunity to provide information about disease risks and empowers citizens to address needs or concerns in their communities (Smolinski et al. 2017a).

The number of new participants that register for FNY varies each season and is typically influenced by the severity of the influenza season, media coverage, and advertising efforts, which vary each season. For example, more than 7,000 individuals registered in January 2013 after FNY was featured in a prominent national news program. Although the direct impact of the guest implementation is unknown due to the confounding factors listed above, the number of registered participants, the total number of reports, and the median number of reports per week were similar to the previous two years. However, the median number of reports per registered participant increased, and the percentage of registered participants who submitted only one report decreased after guest reporting was introduced. The guest reports also increased the total number of reports during each influenza season. Because of these promising results, FNY will continue the using the guest feature.

FNY survey respondents are highly engaged with the platform based on their higher rates of participation compared with the overall FNY user base. FNY survey respondents generally had altruistic reasons for signing up for and for continuing to participate in the project. Their primary motivation both for joining and continuing to use the site across all four influenza seasons is their belief in the importance of disease tracking, to which they want to meaningfully contribute by submitting their weekly symptom reports. Since desire to participate in a citizen science project was added as a motivation that users could select on the survey for the 2017–2018 season, about half of users chose this for a motivation for joining the site, and this option overtook the more self-serving reason of “I want to know when people around me are sick” during the 2018–2019 season as the second-most-common response. This was especially apparent in the SciFri cohort; 77.58% reported that they signed up for FNY because of a desire to participate in citizen science, whereas only 28.56% reported that they were motivated by wanting to know when people around them are sick. These findings support previous research that users with intrinsic motivations have greater participation frequency and duration (Nov et al. 2011a; Borst 2010; Eveleigh et al. 2014).

The careful analysis and comparison across years of surveys will drive future development of surveys and recruitment efforts. For example, the option “I don’t remember” will be removed from the question “How did you first hear about FNY?” in future surveys. Also, because most survey respondents stated they were very likely to recommend FNY to a friend, a recruitment campaign that asks users to invite two or three friends at the start of influenza season may be an effective low-cost way to build the user base. This approach was successful for the Australian system, FluTracking.net (Dalton et al. 2017).

One limitation of the survey data analysis is that the FNY survey respondents may not be a representative sample of all FNY users. Specifically, the median number of reports submitted within an influenza season by FNY survey respondents was larger than the overall FNY user base. In addition, FNY survey respondents had a higher percentage of vaccination compared with the overall FNY user base. They are also highly educated compared with the general U.S. population. According to U.S. census data from 2018, 35% of adults over the age of 25 have Bachelor’s degrees, and 13% had masters and/or doctorate and/or professional degrees; whereas 80% of FNY survey respondents during the same year (2017–2018 survey) reported having at least a college degree, and 41% of those respondents reported also having a graduate degree (Educational Attainment in the United States. 2018). Importantly, FNY and other systems are also not representative of the general population in sex and age distribution. Females and middle-aged individuals are over-represented in these systems (Baltrusaitis et al. 2017; Bajardi et al. 2014; Koppeschaar et al. 2017). Of note, SciFri users were younger and had a greater percentage of males compared with non-SciFri users.

Despite these limitations, participatory surveillance projects such as FNY engage citizen scientists to contribute their information to support scientific research while providing resources for individuals to learn about disease risk and to track symptoms in the community. The FNY population provides valuable, longitudinal data for both researchers and public health officials to understand patterns of disease in near-real time. This type of data collection directly from the public compliments traditional sentinel surveillance measures and provides a flexible surveillance method for emerging diseases. For example, in response to the COVID-19 pandemic, the FNY team pivoted the crowdsourced reporting structure of the platform to create COVID Near You (CNY), a symptom surveillance tool aimed to use citizen-reported health information to supplement tracking of COVID-19. This platform aims to increase understanding of the burden of COVID-19 across North America, identify hotspots of disease activity, and inform the public on reported symptom behaviors, in the absence of population-level and representative testing (COVID Near About us 2020).

In the future, we will assess the impact of the COVID-19 pandemic on user behaviors. Specifically, we will compare the number of new registrations, the frequency of user reporting, and the motivations of users before, during, and after this global pandemic. We will also compare user motivations and reporting behaviors of FNY users with CNY users over the same period. Of particular interest is whether the reporting habits of CNY users who use the SMS text-based response system differ from FNY and CNY users who use the email-based response system.

Conclusions

Following the addition of an option that allows FNY website visitors to submit health reports as guests without registering, we found that the number of registered users who submit only one symptom report decreased and the total number of weekly reports submitted increased. FNY survey respondents are highly engaged with the platform because of their belief in the importance of infectious disease tracking and their interest in participating in citizen science projects. The FNY partnership with SciFri succeeded in registering users highly engaged with science, and these users had notably higher levels of interest in citizen science. Engagement of users is an important aspect of maintaining citizen science projects and ensuring that they are effective in collecting and presenting data.

Data Accessibility Statement

Data is available upon request for public health research. Contact flunearyou@healthmap.org for information.

Ethics and Consent

Our study of FNY data received approval from the Boston Children Hospital’s Institutional Review Board.

Acknowledgements

The authors acknowledge all the participants who contributed their time and information to the FNY system.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

KB, SS, KCS, AWC, AZ, and JSB conceived the research. SS, KB, FD, and LG conducted the statistical analysis. SS, KB, and KCS drafted the manuscript. All authors contributed to the final version of the paper.

References

  1. About SciFri. https://www.sciencefriday.com/about/ (Last accessed Sept 24 2019). 

  2. About us. https://www.covidnearyou.org/#!/who-we-are (Last accessed May 7 2020). 

  3. Aristeidou, M, Scanlon, E and Sharples, M. 2017. Profiles of engagement in online communities of citizen science participation. Computers in Human Behavior. DOI: https://doi.org/10.1016/j.chb.2017.04.044 

  4. Bajardi, P, Vespignani, A, Funk, S, Eames, KT, Edmunds, WJ, Turbelin, C, Debin, M, Colizza, V, Smallenburg, R, Koppeschaar, CE, Franco, AO, Faustino, V, Carnahan, A, Rehn, M and Paolotti, D. 2014. Determinants of follow-up participation in the internet-based european influenza surveillance platform influenzanet. Journal Medical Internet Research, 16(3): e78. DOI: https://doi.org/10.2196/jmir.3010 

  5. Baltrusaitis, K, Santillana, M, Crawley, AW, Chunara, R, Smolinski, M and Brownstein, JS. 2017. Determinants of participants’ follow-up and characterization of representativeness in flu near you, a participatory disease surveillance system. Journal Medical Internet Research Public Health Surveillance, 3(2): e18. DOI: https://doi.org/10.2196/publichealth.7304 

  6. Borst, I. 2010. Understanding Crowdsourcing: Effects of motivation and rewards on participation and performance in voluntary online activities. ERIM Ph.D. Series Research in Management 2010. 

  7. Chen, L, Tozammel Hossain, KSM, Butler, P, Ramakrishnan, N and Prakash, BA. 2016. Syndromic surveillance of flu on Twitter using weakly supervised temporal topic models. Data Mining and Knowledge Discovery. DOI: https://doi.org/10.1007/s10618-015-0434-x 

  8. Choi, J, Cho, Y, Shim, E and Woo, H. 2016. Web-based infectious disease surveillance systems and public health perspectives: a systematic review. BioMed Central Public Health. DOI: https://doi.org/10.1186/s12889-016-3893-0 

  9. Crowston, K and Prestopnik, N. 2013. Motivation and data quality in a citizen science game: a design science evaluation. Proceedings of the Annual Hawaii International Conference on System Sciences. DOI: https://doi.org/10.1109/HICSS.2013.413 

  10. Dalton, C, Carlson, S, Butler, M, Cassano, D, Clarke, S, Fejsa, J and Durrheim, D. 2017. Insights from flutracking: thirteen tips to growing a web-based participatory surveillance system. Journal Medical Internet Research Public Health Surveillance, 3(3): e48. DOI: https://doi.org/10.2196/publichealth.7333 

  11. Dalton, C, Durrheim, D, Fejsa, J, Francis, L, Carlson, S, d’Espaignet, ET and Tuyl, F. 2009. Flutracking: A weekly australian community online survey of influenza-like illness in 2006, 2007, and 2008. Communicable Diseases Intelligence Quarterly Report, 33(3): 316–22. 

  12. Disease Burden of Influenza. Centers for Disease Control and Prevention. https://www.cdc.gov/flu/about/burden/index.html (last accessed 25 June 2019). 

  13. Dredze, M, Cheng, R, Paul, MJ and Broniatowski, D. 2014. HealthTweets.org: A platform for public health surveillance using Twitter. AAAI Workshop – Technical Report. 

  14. Educational Attainment in the United States. 2018. United States Census Bureau. https://www.census.gov/data/tables/2018/demo/education-attainment/cps-detailed-tables.html (last accessed 14 August 2019). 

  15. Eveleigh, A, Jennett, C, Blandford, A, Brohan, P and Cox, AL. 2014. Designing for dabblers and deterring drop-outs in citizen science. In Proceedings of the 32nd annual ACM conference on human factors in computing systems 2014. DOI: https://doi.org/10.1145/2556288 

  16. Fujibayashi, K, Takahashi, H, Tanei, M, Uehara, Y, Yokokawa, H and Naito, T. 2018. A new influenza-tracking smartphone app (flu-report) based on a self administered questionnaire: Cross-sectional study. Journal Medical Internet Research MHealth, 6(6): e136. DOI: https://doi.org/10.2196/mhealth.9834 

  17. Ginsberg, J, Mohebbi, MH, Patel, RS, Brammer, L, Smolinski, MS and Brilliant, L. 2009. Detecting influenza epidemics using search engine query data. Nature, 457(7232): 1012–1014. DOI: https://doi.org/10.1038/nature07634 

  18. Iacovides, I, Jennett, C, Cornish-Trestrail, C and Cox, AL. 2013. Do games attract or sustain engagement in citizen science?: A study of volunteer motivations. CHI ‘13 Extended Abstracts on Human Factors in Computing Systems. DOI: https://doi.org/10.1145/2468356.2468553 

  19. Koppeschaar, CE, Colizza, V, Guerrisi, C, Turbelin, C, Duggan, J, Edmunds, WJ, Kjelso, C, Mexia, R, Moreno, Y, Meloni, S, Paololotti, D, Perrotta, D, van Straten, E and Franco, AO. 2017. Influenzanet: citizens among 10 countries collaborating to monitor influenza in Europe. Journal Medical Internet Research Public Health Surveillance. DOI: https://doi.org/10.2196/publichealth.7429 

  20. Kullenberg, C and Kasperowski, D. 2016. What is citizen science? – A scientometric meta-analysis. PLoS ONE. DOI: https://doi.org/10.1371/journal.pone.0147152 

  21. Marquet, RL, Bartelds, AIM, van Noort, SP, Koppeschaar, CE, Paget, J, Schellevis, FG and van der Zee, J. 2006. Internet-based monitoring of influenza like illness (ILI) in the general population of the Netherlands during the 2003–2004 influenza season. BioMed Central Public Health. DOI: https://doi.org/10.1186/1471-2458-6-242 

  22. Nov, O, Arazy, O and Anderson, D. 2011a. Dusting for science: Motivation and participation of digital citizen science volunteers. In Proceedings of the 2011 iConference. DOI: https://doi.org/10.1145/1940761.1940771 

  23. Nov, O, Arazy, O and Anderson, D. 2011b. Technology-mediated citizen science participation: A motivational model. In Proceedings of the AAAI international conference on weblogs and social media (ICWSM). 

  24. Nov, O, Arazy, O and Anderson, D. 2014. Scientists@Home: What drives the quantity and quality of online citizen science participation? PloS ONE, 9(4): e90375. DOI: https://doi.org/10.1371/journal.pone.0090375 

  25. Paolotti, D, Carnahan, A, Colizza, V, Eames, K, Edmunds, J, Gomes, G, Koppeschaar, C, Rehn, M, Smallenburg, C, Turbelin, C, Van Noort, S and Vespignani, A. 2014. Web-based participatory surveillance of infectious diseases: the Influenzanet participatory surveillance experience. Clinical Microbiology and Infection, 17–21. DOI: https://doi.org/10.1111/1469-0691.12477 

  26. Prestopnik, N and Crowston, K. 2011. Gaming for (Citizen) Science: Exploring motivation and data quality in the context of crowdsourced science through the design and evaluation of a social-computational system. 2011 IEEE Seventh International Conference on e-Science Workshops. DOI: https://doi.org/10.1109/eScienceW.2011.14 

  27. Prestopnik, N, Crowston, K and Wang, J. 2017. Gamers, citizen scientists, and data: Exploring participant contributions in two games with a purpose. Computers in Human BAehavior. DOI: https://doi.org/10.1016/j.chb.2016.11.035 

  28. Raddick, MJ, Bracey, G, Gay, PL, Lintott, CJ, Murray, P, Schawinski, K and Vandenberg, J. 2011. Galaxy Zoo: Exploring the motivations of citizen science volunteers. Astronomy Education Review. DOI: https://doi.org/10.3847/AER2009036 

  29. Reed, J, Raddick, MJ, Lardner, A and Carney, K. 2013. An exploratory factor analysis of motivations for participating in Zooniverse, a collection of virtual citizen science projects. 46th Hawaii International Conference on System Sciences. DOI: https://doi.org/10.1109/HICSS.2013.85 

  30. Rotman, D, Preece, J, Hammock, J, Procita, K, Hansen, D, Parr, C and Jacobs, D. 2012. Dynamic changes in motivation in collaborative citizen-science projects. Proceedings of the ACM 2012 conference on computer supported cooperative work. DOI: https://doi.org/10.1145/2145204.2145238 

  31. Salathé, M. 2018. Digital epidemiology: what is it, and where is it going? Life Science Social Policy, 14(1): 1. DOI: https://doi.org/10.1186/s40504-017-0065-7 

  32. Signorini, A, Segre, AM and Polgreen, PM. 2011. The use of Twitter to track levels of disease activity and public concern in the U.S. during the influenza A H1N1 pandemic. PloS one, 6(5): e19467. DOI: https://doi.org/10.1371/journal.pone.0019467 

  33. Smolinski, MS, Crawley, AW, Baltrusaitis, K, Chunara, R, Olsen, JM, Wójcik, OP, Santillana, M, Nguyen, A and Brownstein, JS. 2015. Flu Near You: Crowdsourced symptom reporting spanning 2 influenza seasons. American Journal Public Health, 105(10): 2124–30. DOI: https://doi.org/10.2105/AJPH.2015.302696 

  34. Smolinski, MS, et al. 2017a. Finding outbreaks faster. Health Security, 15(2): 215–220. DOI: https://doi.org/10.1089/hs.2016.0069 

  35. Smolinski, MS, et al. 2017b. Participatory Disease Surveillance: Engaging communities directly in reporting, monitoring, and responding to health threats. Journal Medical Internet Research Public Health Surveillance, 3(4): e62. DOI: https://doi.org/10.2196/publichealth.7540 

  36. Thompson, WW, Comanor, L and Shay, DK. 2006. Epidemiology of seasonal influenza: use of surveillance data and statistical models to estimate the burden of disease. Journal Infectious Disease, 194(suppl 2): S82–91. DOI: https://doi.org/10.1086/507558 

  37. Why early detection of outbreaks is so important. 2017. Canada Family Physician, 63(2): 133. 

  38. Wójcik, OP, Brownstein, JS, Chunara, R and Johansson, MA. 2014. Public health for the people: participatory infectious disease surveillance in the digital age. Emerging Themes Epidemiology. DOI: https://doi.org/10.1186/1742-7622-11-7 

comments powered by Disqus