Participation in CS programmes

Participation takes multiple forms in CS programmes and refers to either the level of volunteers’ involvement in scientific activities or the type of activities and intellectual effort needed. Shirk et al. (2012) classified projects based on the level of volunteers’ involvement in scientific activities into: (a) contractual projects initiated by scientists to address a community need, (b) contributory projects designed by scientists in which the public collects data, (c) collaborative projects in which the public, in addition to collecting data, refines the project design, analysis, and dissemination, (d) co-created projects designed in collaboration with the public, and (e) collegial contributions initiated by non-professional members of the public who conduct research independently.

For online CS programmes in particular, Ponciano et al. (2014) classified participation in two Zooniverse projects (Galaxy Zoo, Milky Way) based on levels of engagement of volunteers into transient and regular. Transient users completed tasks only once and did not return to the platform again. They participated sporadically in CS activities, most likely due to a help request made by others. In contrast, regular users returned to the platform at least once to complete more tasks. They could be described as volunteers who actively seek opportunities for participation in CS projects. Online participation was found to be mainly transient, yet regular volunteers completed the largest number of tasks. Similarly, Eveleigh et al. (2014) described participation in terms of high and low contributions, with the latter referring to small input and little involvement and the former to regular and significant participation, including social participation in forums. In a follow-up analysis, Ponciano and Brasileiro (2014) proposed five engagement profiles: Hardworking (hard work, yet leaving the project soon), spasmodic (short period of contributions with irregular periodicity), persistent (link to the project for long, yet with few active days), lasting (similar to persistent profile, yet remaining linked to a project for a shorter time period), and moderate (the shorter the period of time volunteers are linked to a project, the more days they come back to the project to complete tasks). Volunteers were found mainly in the moderate profile with a minority in the persistent profile. Lurking members also were detected in CS programmes, indicating non-active participation (no contributions) (Aristeidou et al. 2017).

In the broader Human-Computer Interaction (HCI) literature, participation, referred to as “engagement,” has been classified on a continuum based on the type of activities that users engage with and the intellectual contribution required. Preece and Shneiderman (2009) describe online participation as initiated by reading activities, developed into contributing data in the form of a question or a picture, extended to collaborative activities such as an article creation in Wikipedia, and reaching leading activities, for example, mentoring novices. The number of participants decreases from reading to leading activities. Similarly, Haythornthwaite and De Laat (2010) classified online activities as lightweight (simple, often directed by others) or heavyweight (complex and time consuming, such as academic writing).

In the case of Zooniverse, projects are more relevant to contributory forms of participation as described by Shirk et al. (2012), yet the public is processing, rather than collecting data, as described by Bonney et al. (2016), where data are originally collated and presented in the platform by scientists. In terms of the types of data processing activities that Zooniverse hosts, we categorised these into (a) “tasks requesting a response to a question” such as “Are there any penguins in this image?” in the Penguin Watch project; (b) “free text entry tasks” such as “Add keywords to describe each illustration” in the Science Gossip project; (c) “marking tasks” such as Project Plumage requesting to mark-up different views of bird specimens; and (d) “identification tasks” such as those found in the majority of camera trap projects on Zooniverse (e.g., Camera CATalogue), which ask volunteers to identify the species they can see from a set list. What determines effort or intellectual contribution is how easy or difficult the question being asked in each project is as well as the series of tasks (and accompanied time devotion) participants are asked to address within each project. An easy task could be a binary question, such as the first task on the Penguin Watch project, which asks: “Are there any penguins in this image?” Most people can recognise a penguin and answer “yes” or “no” when they see one. On the contrary, the yes/no question “Are there Meridiani-type polygonal ridges visible in this image?” in the Planet Four: Ridges project is rather complex, due to lack of familiarity with the scientific terms and the data in question for a majority of people. An example of a time-consuming task is the second task on the Penguin Watch project, which requests participants to mark the locations of all the penguins in a given image; although identifying penguins is a relatively easy activity, in some cases hundreds of penguins should be marked in a single image. In this study, we examined (as part of RQ2) whether and how different task types relate to participation patterns, as a means to understand whether projects with, for example, “easy” or less time-consuming tasks are more or less often chosen by young people.

Capturing participation in informal learning settings

Learning Analytics (LA) typically refer to data generated from learners’ interaction with, for example, a Learning Management System (LMS) and their application to improving teaching and learning. LA have been used less frequently to understand participation and learning in informal learning settings (Ferguson et al. 2016). Such studies have examined learning at the workplace (e.g., Littlejohn 2017), interactions in technology-mediated social systems such as online forums (e.g., Kloetzer, Schneider, and Costa 2016), and a few CS projects examining adult participation (Aristeidou et al. 2017; Dittus, Quattrone, and Capra 2016; Morais, Santos, and Raddick 2015; Ponciano and Brasileiro 2014). In this paper, we use LA as a methodology that allows a fine-grained analysis of young people’s participation (aged 5–19) in Zooniverse projects as a first step towards understanding and improving their learning experiences in online CS. We perceive participation as a requirement of learning; understanding how young people participate in CS programmes will enable us to identify how or what types of participation relate to, facilitate, or hinder learning processes and outcomes and improve the design of CS projects accordingly. Also, we perceive online CS projects as “informal learning settings,” as these have not been designed with an explicit learning or curriculum objective, learning is rather random and spontaneous, and is less likely to lead to any form of recognition.

CS studies that have made use of LA propose or develop a set of metrics for capturing online participation for adult volunteers. Amsha et al. (2016) explicitly proposed the use of LA for understanding learning and participation in online CS programmes. They introduced and tested a non-exhaustive set of indicators for demographic, learning, engagement, creativity, and collaboration related LA, such as total number of tasks over the activity duration (engagement-related LA), average percentage of tutorial completion (learning-related LA), length of blog posts (creativity-related LA), and number of artefacts shared (collaboration-related LA). Ponciano and Brasileiro (2014) captured Zooniverse engagement over time considering points of engagement, periods of sustained engagement, disengagement, and re-engagement. Their metrics (adopted also in this study) capture time potentially linked to a project, days actually linked to a project, active days, time devoted to tasks per day, and elapsed time between two active days. Morais, Santos, and Raddick (2015) highlighted the evolution in number of volunteers and classifications over time and showed that the majority of classifications in Zooniverse were completed in the first 600 days of the project, and sharply declined afterwards. They also explained “bursts” of activity and analysed volunteers joining or leaving a project in short intervals. They cluster volunteers into committed, potential, and curious based on the following metrics: Relative duration activity, frequency of activity per day, and classifications completed within a single day.

Dittus, Quattrone, and Capra (2016) compared the design of three humanitarian mapping projects using cohort and task analysis and first-time contributor’s performance. Projects designed as sustained initiatives that engage volunteers through various means, such as social media and mapathons, exhibited high retention rates. In contrast, projects with complex tasks were found to demotivate first time contributors. Improvements in performance were not related to increased retention. Aristeidou et al. (2017) examined participation of volunteers in an online community where participants could not only contribute to CS projects but also initiate and pursue their own CS activities. They expanded the Ponciano and Brasileiro (2014) engagement metrics by proposing the lurking ratio, that is, the proportion of days that a participant was lurking (logging in a platform, browsing, yet not contributing) in relation to the total days that they visited the project.

Process of recruitment and consent

Aligning with the broader LEARN Citizen Science project objectives, we aimed to identify and examine the participation of at least 100 volunteer young people (ages 5–19) on Zooniverse projects that are related to the Natural History Museum (NHM) of London, UK. This number target was rather arbitrary given the lack of any previous examinations referring to the numbers of young people possibly joining online projects that could inform our study. The only NHM active project at the time was Project Plumage, asking participants to mark-up photos of different coloured birds. This activity aimed to help scientists understand feather colour evolution over time. Due to the low number of participants in the project (about 2,000 at the time) and to increase the number of volunteers, we ran various promotional campaigns (emails, newsletters to Zooniverse volunteers) originally raising awareness only for Project Plumage. The marketing campaign lasted for two and a half months. This approach resulted in increasing the overall number of registered volunteers to around 3,000, yet the number of young people attracted through this publicity did not meet our initial target of 100 young people. This resulted in expanding the scope of the study to other Zooniverse projects that focused on NHM-related research questions or collections, including Science Gossip, Notes from Nature (participants transcribe photos of specimens involving many NHM digitized collections), London Bird Records (participants transcribe historical records of birds), Orchid Observers, and Penguin Watch. This follow-up campaign lasted for two months.

Users interested in the study followed an email weblink. An information sheet briefed potential participants about the aim and procedures of the study, and an online form invited them to consent in participating (if over 16 years old), or to provide their guardians’ contact details (if younger than 16 years old) for researchers to confirm consent over the phone with guardians. Although 159 young people originally signed up for the study, the final number was 104, as 22 of them were duplicates or their usernames did not exist on the Zooniverse database, 19 did not provide consent, and 14 did not make any contribution to the platform. Log data extracted from the platform for analysis were anonymised, that is, Zooniverse usernames were changed to, e.g., User1, User 2. The study received ethical clearance from the university of the lead author.

Sample characteristics

The great majority of the sample were female (67%, n = 70), 29% were male, three users selected “other,” and one preferred not to disclose gender information (total n = 104). In contrast to existing CS projects and the predominance of male adults in CS activities (Brossard et al. 2005; Price and Lee 2013), young participants were found to be mainly female. This insight warranted further investigation and resulted in examining gender differences in RQ1, in particular comparing the number of classifications made between male and female and RQ2, comparing the gender distribution in the choice of CS projects in which young people participate. In terms of age, a great majority (n = 99) was aged 16–19, two users were aged 13–15, two users were aged 10–12, and one user was under 10 years old. In terms of the device used to access Zooniverse, a great majority (n = 76) used a desktop computer, 16 used a mobile device, and 11 used both devices.

Process of data analysis

Engagement metrics

The four engagement metrics were compared to adult Zooniverse users (see Ponciano and Brasileiro 2014). Young participants, engagement metrics (a) M = 0.31, SD = 0.44; (b) M = 0.02, SD = 0.11; (c) M = 0.49, SD = 0.36; (d) M = 49.93, SD = 52.08 were found to be relatively less active than adults during the days they are linked to the platform and to devote considerably less time daily on Zooniverse tasks. Compared to adults, engagement metrics (a) M = 0.40, SD = 0.40; (b) M = 0.44, SD = 0.54; (c) M = 0.20, SD = 0.30; (d) M = 18.27, SD = 43.31, they were found to be linked to the platform for a longer period of time, and they were less systematic in their visits.

Engagement profiles

Participants with two or fewer active days on the platform formed a separate profile labelled as “visitors.” The rest of the participants were clustered according to the metrics of Ponciano and Brasileiro (2014) (“daily devoted time” was excluded, see previous sections). The hierarchical clustering algorithm, with the dendrogram cut-off vertical lines at the scales values of “25,” “20,” and “15” (Figure 1), indicated between two and five clusters as the interval to be tested, suggesting that four clusters is the optimal solution in terms of number of clusters (Figure 2). An overview of box-plots (Figure 3) revealed an outlier far away from the lower whisker of cluster 1, and this was excluded from the dataset. The K-means algorithm was used to classify the data into four clusters. Clusters were labelled based on the core activity of the cluster (Figure 4). In two cases, labels were adopted from the study of adult Zooniverse users.

Cluster 1: Systematic engagement (n = 5). This category demonstrates the largest activity ratio, combined with moderate relative activity duration and zero variation in periodicity. Young people in this category visited the platform at regular time intervals and were very active during their stay on Zooniverse. However, the moderate relative activity duration showed that compared to some other profiles, these participants did not remain linked to Zooniverse for a long time. Young people in this category were found to be all female, aged 16–19, who contributed from 257 to 1,625 classifications to 9–33 projects. One participant shared her account with her family. A representative of this category is User 29; she registered with Zooniverse on the 7th of August and contributed 420 classifications to 19 projects, on the 7th, 8th, 9th, and 10th of August.

Cluster 2: Moderate engagement (n = 16). Members of this category exhibited moderate relative activity duration, low activity ratio, and low variation in periodicity. The latter shows that their visits were at a constant rate, yet they were neither linked to the platform for a very long time nor very active during that period. This category consisted of 11 female and five male users, who did not share their accounts with others (e.g., family, friends). Only one member was younger than 10 years old; the rest of the participants were aged 16–19. Members of this category contributed between 62 and 3,171 classifications to 1–25 projects. An example, User 94 (female, 16–19 years old), registered on the 29th of June 2018 and contributed 62 classifications to three projects on the 29th of June and 12th and 19th of July.

Cluster 3: Casual engagement (n = 8). Members of this cluster were characterised by high variation in periodicity and relative activity duration, while they had extremely low activity ratio levels. Although they remained linked to Zooniverse for a long time, they had inconstant visits and were not very active during the period that they were linked to the platform. They were five female and three male participants, aged 16–19, who did not share their accounts with other people. They contributed from 65 to 651 classifications to 2–27 projects. An example, User 43 was female and registered in March 2013. She contributed 635 classifications to 27 projects. Although registered in 2013, she made her first classification on the 21st of July 2015, followed by more classifications in the following months.

Cluster 4: Lasting engagement (n = 40). This category is characterised by the largest relative activity duration, low variation in periodicity, and low activity ratio. Members of this category remained linked to Zooniverse the longest, did not visit the platform very regularly, and had a small number of active days during the long period that they were linked to Zooniverse. The majority of young people are found in this cluster. This category consists of 28 female, nine male, and three who declared themselves as “other.” Apart from two members, the rest were aged 16–19. One member was aged 13–15, and another was aged 10–12 and shared her account with her mother. Members of this cluster contributed from 38 to 4,558 classifications to 3–42 projects. An example, User 47 was female, 16–19 years old, and registered on the 8th of March 2018. She had only three active days since registration, during which she contributed 449 classifications to five projects.

Cluster 5: Visitors (n = 34). Members of this cluster contributed to projects one or two days only, and therefore their variation in periodicity could not be calculated. Their activity ratio is nearly as large as that of Cluster 1, but this was accompanied by low relative activity duration, indicating active days during their short stay in Zooniverse. Metric values in this cluster and the dendrogram suggested two sub-groups of this category. The first subgroup (n = 12) was linked to the project for longer than the second, but it was active only one or two days during this time period. It consisted of eight female and four male users, all aged 16–19, with none sharing their account with others. Nine members contributed to two projects only, while one user contributed to 16 projects. For example, User 79 (female, 16–19 years old) registered with the platform in April 2016, and had only two active days since then, in December 2016 and July 2017. She made 28 contributions to two projects. The second subgroup (n = 22) was linked to the project for a maximum of two days and was very active during those days. The group included 13 female users, eight male and one participant who did not disclosed their gender. All but two participants were aged 16–19; one was aged 13–15 and one was aged 10–12. One participant, who was aged 16–19, shared her account with family and friends. Although the majority of this category (n = 13) contributed to a single project only, there were users who contributed to as many as ten projects. An example, User 34 had a single active day. She was female, 16–19 years old, and registered on 29 June 2018. She contributed 64 classification to a single project, Project Plumage.

Figure 5 shows examples of how participation changed over time based on year quarters (e.g., 2015 Q3: third quarter of 2015) for specific users from each engagement profile. For example, User 75 was a systematic user (Cluster 1), and all of their activity was recorded in the second quarter of 2018. User 60 (Cluster 2) visited the platform regularly between the second quarter of 2016 and the first quarter of 2017, yet they did only a few tasks on the platform. User 46 (Cluster 3) had visits on the platform over a long period of time (between 2015 and 2018), but these visits were rather random and accompanied by low activity.

Frequency of contributions

Fifty percent of participants (n = 52) were found to have 200 or fewer classifications over the period of this study. Six participants presented rather unusual behaviour, exhibiting more than 2,000 classifications, and seven were found to have between 1,600 and 2,000 classifications (median = 199). Figure 6 presents the distribution of classifications across participants; the majority contributed few tasks whereas a few participants made the majority of contributions.

Female participants made 35,129 classifications, whereas male participants made 26,221. Participants classifying themselves as “Other” contributed 1,308 classifications (total number of classifications made 62,659). An independent Kruskal-Wallis test showed no statistically significant differences in the number of contributions made between male and female (p = .657, NS; p = .05), suggesting that the above number difference is random.

Project popularity amongst young people

As expected due to our initial recruitment strategy and the emphasis on a specific NHM project, the most popular Zooniverse project chosen by young people was Project Plumage (n = 48). The second and third most popular were projects not explicitly targeted through the project, Plastic Tide (n = 24) and Camera CATalogue (n = 21). Yet, the Plastic Tide was systematically promoted by Zooniverse after its launch in April 2017. Camera CATalogue is more likely to reflect young people’s “natural” interests in specific CS projects, in particular an interest in identifying big cats and other wildlife in pictures captured by motion-activated camera traps set around the world. Figure 7 shows the distribution of the most popular projects across gender. Follow-up Kruskal-Wallis independent tests, comparing the distribution of male and female participants across all projects that young people joined, revealed no statistically significant differences for either male (p = .504, NS) or female (p = .555, NS; p < .05), suggesting that males and females participate in similar Zooniverse projects.

In a follow-up analysis, we identified and mapped the most popular projects and their corresponding tasks to the five engagement profiles to explore whether task types relate to specific participation patterns. Types of project tasks were as follows:

1. Drawing: This task is used to make markings on an image. The various drawing tools include points, lines, rectangles, and ellipses.
2. Survey: This task asks volunteers to choose from a predefined list of options, e.g., a list of species likely to be found in camera trap images.
3. Question: This task is used to ask multiple choice questions about an image.
4. Text: This task allows volunteers to write free text and is used mostly for transcription projects.
5. Dropdown: This task allows volunteers to set a dropdown list of options, e.g., a list of US States.
6. Combo: This task allows volunteers to add multiple tasks on the same page, e.g., a text task and a question task shown at the same time.
7. Slider: This task allows volunteers to set a slider option for choosing between a range of values, e.g., to choose from 1–10 how many animals are in an image.
8. Shortcut: This task creates a tick box that can be used to skip to the next image, e.g., “There are no animals in this image.”

Figure 8 shows the distribution of task types across the five engagement profiles. The most popular tasks are answering single and multiple choice questions and taking a survey, while combo tasks were chosen only by visitors. Although the graph may point to distinct choices of tasks amongst profiles, a chi-square analysis showed no statistically significant differences between engagement profiles and project tasks of the most popular projects within each cluster (χ²(1,38) = 19.847 p = 0.705, NS).

Co-chosen projects

The most popular projects chosen together by young people were the Project Plumage, the Plastic Tide, and Notes from Nature (Figure 9). These projects linked indirectly with all other projects, including the less popular ones. The dominance of these three projects is more likely explained by the promotion of Project Plumage and Notes from Nature in this study, and the fact that Plastic Tide was systematically promoted by Zooniverse over the last few years. Regarding the degree of centrality, Plastic Tide was the project with the most connections, followed by Project Plumage and Bash the Bag.

Figure 9 also shows the four sub-communities of projects that tend to be chosen together by young people based on the number of ties between each pair of nodes. The graph suggests that the purple and the green groups, with 35% and 34% graph coverage respectively, were more dominant than the orange and blue ones with 16% and 15% coverage respectively. Examples of projects that belong to the same group are Plastic Tide and Seabird Watch (in orange) – both projects are UK-based, focus on the environment, and have received major publicity in the UK – and Bash the Bug and Where are my Body Organs (in blue), which are both biomedical in nature and may suggest that participants had a special interest in biomedicine. Overall, this analysis suggests that targeted publicity and personal interests may explain why young people join specific groups of projects and not others. Given the modularity algorithm result of 0.13, it is noted that there were no dense connections within the groups, yet many connections with projects of other groups. Users co-chose projects from more than one of the coloured groups. A follow-up frequency analysis showed that the majority of users (n = 58) contributed to up to eight projects, while eighteen participants contributed to only one project.