Item features and cognitive processes in assessments of SR

Assessments should represent the processes and strategies necessary for participants to perform on tasks that test a psychological construct (this is known as construct representation; e.g., Embretson 1983). SR skills depend on cognitive processes—such as identifying the variables under investigation (i.e., information encoding and retrieval)—as well as on the use of cognitive strategies—such as controlling several variables to avoid biases in the investigation (e.g., the control-of-variables strategy) (Morris et al. 2012). Items that assess SR skills usually include a scientific problem that can be solved by identifying the relevant variables and controlling other variables to facilitate unbiased conclusions. Previous research on SR questionnaires has explored how particular item features—such as the SR skill being investigated, the number of independent variables, and the research context—influence the thinking processes involved in the identification and control of variables (Krell 2018; Mannel, Walpuski, and Sumfleth 2015). In the development of SR items, researchers have to account for those item features so that the SR assessment instrument represents those cognitive processes and allows for valid interpretations of test scores (Hartig and Frey 2012). If the different item features are accounted for, it is possible to calculate the influence of each item feature in relation to the items’ overall difficulty. Significant sources of item difficulty that are not related to the psychological construct pose a threat to validity because they suggest that other abilities are needed to solve the item in addition to the intended abilities. Hence, the identification of such sources of item difficulty has the potential to improve the validity of assessments. Moreover, the identification of sources of item difficulty that are related to the psychological construct can guide item development (Messick 1995).

In the construction of valid assessments, previous research distinguished between two kinds of item features (Opfer, Nehm, and Ha 2012). First, deep-structure item features are held constant across all items because they aim to assess the cognitive processes and strategies related to SR, such as the identification and control of variables. Second, item surface features embed items in specific contexts of CS research designs (e.g., Follett and Strezov 2015). Individuals with high-level SR skills master the assessment despite the varying contexts (i.e., item surface features), but individuals with low-level SR skills are more likely to be distracted by such item surface features (Opfer, Nehm, and Ha 2012). If item features and the related cognitive processes are identified, it is possible to explain how an assessment instrument works, and this contributes to construct validity (Fischer 1995, 2005; Hartig and Frey 2012). Hence, we explore how different item features that indicate the cognitive processes required to solve the item contribute to item difficulty in SR assessments. From previous research, we identified two deep-structure item features that are essential for the assessment of SR: (1) the feature that one of the three different SR skills (i.e., forming hypotheses, testing hypotheses, and analyzing data) is required to solve the item, and (2) the feature that the number of independent variables (i.e., one or two independent variables) has to be accounted for to solve the item. Furthermore, three item surface features that might distract participants from successfully applying their SR skills were examined: research context, text complexity, and the use of specialist terms.

The three SR skills of forming hypotheses, testing hypotheses, and analyzing data are deep-structure item features—they relate to the cognitive processes and strategies of identifying and controlling variables—and have been shown to significantly influence item difficulty (Hammann et al. 2008; Mannel, Walpuski, and Sumfleth 2015; Krell 2018). Based on comparisons of item difficulties in the formal education context, research suggested that the SR skill of testing hypotheses requires different knowledge than forming hypotheses and analyzing data: the SR skills of forming hypotheses and analyzing data seem to require profound domain-specific content knowledge, while the SR skill of testing hypotheses is more closely, but not exclusively, related to knowledge of the processes (Hammann et al. 2008). Furthermore, previous research indicates that assessment items on forming hypotheses and testing hypotheses typically presuppose one part of the inquiry as given (i.e., the items provide either the research design or the hypothesis). For example, assessment items on testing hypotheses provide a hypothesis and ask the participant to propose a valid research design to test it. Items on data analysis, however, require participants to relate two parts of the inquiry process, that is, the research design and the observations (Krell 2018). In the formal learning context, studies revealed that assessment items on forming hypotheses and testing hypotheses are typically easier to solve for participants than assessment items on analyzing data (e.g., Krell 2018). In CS projects, all three SR skills are neither easy for participants nor common so that we aimed at comparing the item difficulties for the three SR skills in the informal education context of a CS project. Testing hypotheses will serve as the reference category in our analysis, that is, the item difficulty of the SR skills of forming hypotheses and analyzing data will be compared against it.

Item complexity is the second deep-structure item feature we identified. Item complexity in SR assessments is defined as the number of variables that individuals need to keep in mind to answer an item, that is, whether the hypothesis, the research design, or the data refer to one or to two independent variables (Mannel, Walpuski, and Sumfleth 2015). Thinking of more than one independent variable at once increases the cognitive load, that is, the amount of information that individuals need to process (Kauertz et al. 2010). Therefore, the item complexity contributes to the item difficulty in an assessment of SR skills (e.g., Kauertz et al. 2010; Krell 2017).

Although the investigation of hypotheses as one form of scientific reasoning spans the sciences, SR skills must be applied in the various contexts of research. The research context is considered an item surface feature because individuals can apply their SR skills to different contexts, while the underlying thinking processes of variable identification and control remain the same. Previous research has indicated that SR also depends on domain-specific knowledge (Fischer et al. 2014). Individuals need to have domain-specific knowledge of the respective research context to identify the investigated variables and to represent them in a mental model (Morris et al. 2012). Domain-specific knowledge enables the adequate representation of variables that are relevant to problem-solving (Fischer et al. 2014). Especially unfamiliar contexts have been shown to make items more difficult to solve (Le Hebel et al. 2017). Hence, the context of the items on SR also contributes to the items’ difficulty (Krell 2018).

Two further item surface features that can contribute to the items’ difficulty are text complexity and the use of specialist terms. In the natural sciences, language is determined by its functional grammar, including specialist terms and complex sentences (Fang 2006). In SR assessments, items typically include a text-based description of a problem that can be solved by applying SR skills. The problem description often employs specialist terms and words and sentences with an above-average length. These constructs are due to the functional grammar that is used in the language of the natural sciences. The length of words and sentences (i.e., text complexity) and the use of specialist terms are both considered item surface features as they influence the readability of scientific texts. Individuals have to follow the grammar in the text and understand the specialist terms to be able to represent the problem mentally. Both text complexity and specialist terms seem to influence the item difficulty (Stiller et al. 2016; Krell, Khan, and van Driel 2021).

This study describes the development of a flexibly adaptable SRQ that systematically considers two deep-structure item features (the skills of forming hypotheses, testing hypotheses, and analyzing data; the two levels of item complexity) and three item surface features (the research context; the text complexity; the use of specialist terms). We compared the fit of statistical models on SR skills to provide empirical evidence for the item features’ contribution to item difficulty. To do so, we first tested a descriptive Rasch model (one-parameter logistic model [1PLM]) that does not differentiate between the item features. Then, we compared the descriptive Rasch model against a basic linear logistic test model (LLTM) that accounts for three of the item features, that is, the SR skills, the levels of complexity, and the research contexts, and against an extended LLTM that additionally takes text complexity as well as specialist terms into account. Our research provides valuable insights for the increasing CS community and researchers who aim to assess participants’ SR skills in CS projects by suggesting a blueprint and guidelines for the development of SRQs, using item features that can be adapted to different CS-relevant contexts.

Instrument development

We applied an established blueprint for the systematic development of a multiple-choice SRQ (Krell 2018). The blueprint accounts for two deep-structure item features by addressing three skills of SR and two levels of item complexity. Furthermore, the blueprint allows for the contextualization in our particular CS projects (i.e., on wildlife ecology, bat ecology, and air pollution) by adapting the surface features to different research contexts. Finally, the blueprint guides the structure of the language (i.e., text complexity and specialist terms) and the figures (example in Figure 1; see Supplemental file 1: Appendix 1 for the blueprint used in this study).

To adapt the blueprint and contextualize it in authentic CS research, experts first identified research designs within actual research on the respective topics (e.g., flight distance of urban wild boars: Stillfried et al. 2017; effect of artificial light at night and tree cover on bats: Straka et al. 2019). Second, the experts reviewed the research regarding its central variables, hypotheses, design, and the data obtained from this research. Third, we adopted the respective variables of the chosen research contexts to each of the three SR skills (i.e., forming hypotheses, testing hypotheses, and analyzing data). To differ between item complexities, we varied the number of independent variables under consideration by using two levels, that is, the consideration of one or two independent variables (i.e., low and high item complexity). The SRQ comprised three contexts (wildlife ecology, bat ecology, and air pollution) for three SR skills (forming hypotheses, testing hypotheses, analyzing data) and two item complexity levels (one independent variable and two independent variables). The complete crossing of the three contexts, three SR skills, and two complexity levels resulted in 3 × 3 × 2 = 18 items in total (Table 1).

With regard to the remaining two item surface features, we did not purposefully vary the length of words and sentences or the specialist terms between the items; the blueprint aimed to keep the complexity of language comparable for all items. All 18 items had a comparable structure (see example in Figure 1). First, the text stem introduced a research design with all relevant dependent and independent variables. Second, the picture represented the setup of this research and named all independent variables. Third, the question prompted participants to provide a valid hypothesis, suggest an additional setup, or draw a valid conclusion. Fourth, each item provided four answer options.

Although we aimed to keep the complexity of language comparable, the different research contexts required using words and terms of different length and familiarity in the assessment items. We analyzed the text complexity and the use of specialist terms to control for their effects on item difficulty. To monitor the influence of text complexity on item difficulty, we calculated the Flesch Reading Ease Index (FRE; Flesch 1948) in its German adaptation that accounts for the mean sentence length and the mean number of syllables per word. For the FRE, values below 60 indicate a high text complexity, that is, sentences are longer and a word has more syllables. Furthermore, we counted the percentage of specialist terms (ST) in every item because the ability to identify the variables being investigated (i.e., the cognitive process of information encoding) also depends on knowledge of specialist terms. We formed a list of specialist terms that are not commonly used in everyday language (e.g., transect, particulate measure, flight distance) and consistently applied it to all items. The number of specialist terms varied across the 18 items depending on the respective research contexts. The less tangible research context of air pollution used more specialist terms than the research contexts of wildlife ecology and bat ecology. More than seven specialist terms in 100 words (ST > 7%) are considered cognitively demanding (Kulgemeyer and Starauschek 2014).

Participants

Participants were recruited via media sources such as radio, newsletter, or posters in public places. They could apply to participate in one of the two field seasons of the CS project. Given the diversity of sociodemographic factors within the city’s districts and to ensure an equal distribution of participants across the city, citizens were selected for participation based on the location where they lived. The participants were evenly distributed across the districts of the city of Berlin by the design of this study. N = 374 citizens participated, of which 198 were identified as being female and one as having a non-binary gender. Their mean age was M = 53.22 (SD = 11.92; range: 25–81). As in many CS projects, this sample was well educated, with most participants holding an upper secondary school certificate (82.6%) and fewer participants holding a certificate from the upper secondary vocational track (15.5%). Furthermore, more than half of the participants held a university degree (59.9%) and some also even had a doctoral degree (11.5%).

Procedure

To participate in one of the two field seasons, participants signed up on an online platform. For two months, the participants formed an online community to share and analyze the data they had collected, as well as to discuss their findings with other participants. Participants filled in the questionnaire before and after they took part in the field season. In this study, we report on data that was collected from two field seasons, one in fall 2018 and one in spring 2019. In detail, we analyzed the answers of participants to the SRQ before the project. The reason for analyzing the data (i.e., participants’ answers to the questionnaire) collected before participation in the CS project was that this assured that the SR skills assessed had not been explicitly trained by participation in the CS project. Participants gave their informed consent for this study, and an external ethics board approved the SRQ.

Data analysis

To estimate how the different item features contributed to the items’ difficulty, we applied the LLTM. The LLTM assumes that item difficulty is a linear combination of the different item features (Fischer 1995, 2005). The LLTM belongs to the Rasch models, a family of established psychometric models applied in psychological and educational research (Embretson and Reise 2000). The family of Rasch models includes descriptive psychometric models, such as the 1PLM, which allows for the holistic estimation of individual person ability (θ_s) and item difficulty (β_i) parameters. In the 1PLM, it is assumed that the probability of a correct item response depends only on θ_s and β_i (Embretson and Reise 2000):

In contrast to descriptive models such as the 1PLM, explanatory models consider different item features to estimate each feature’s influence analytically (Wilson, Boeck, and Carstensen 2008). From this perspective, the LLTM can be seen as an item explanatory model because it replaces the β_i parameter with a linear combination of the basic parameters α_k : ${{\beta }^{\prime }}_i={\Sigma }_{k=1}^N({\alpha }_k{\chi }_{\mathrm{ik}})$M2 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\beta'_i} = \Sigma _{k = 1}^N({\alpha_k}{\chi_{ik}}) \] \end{document} (Fischer 1995). The LLTM splits up the item difficulty of whole items (i.e., β_i parameter in the 1PLM) into the individual contribution of different item features to the item difficulty (i.e., α_k parameters). Hence, if an LLTM can be shown to fit the given data, the estimated parameters α_k provide a measure of each item feature’s contribution—such as the different SR skills, the levels of item complexity, and the research context—to the item difficulty.

To evaluate the model fit of an LLTM, a two-step procedure is proposed: First, the 1PLM has to fit “at least approximately” (Fischer 2005, p. 509) to the data because further decomposition should concern a unidimensional measure of SR skills. Second, the decomposition of β_i needs to be checked for empirical validity. For this purpose, item difficulty parameters estimated in the 1PLM and the LLTM can be compared (e.g., graphically or by calculating the Pearson correlation), assuming that they positively correlate (Baghaei and Kubinger 2015). Furthermore, the Akaike information criterion (AIC), the Bayesian information criterion (BIC), and the log-likelihood difference test can be applied to compare the fit of both models, as well as the plausibility of different LLTMs (Fischer 2005; Wu, Adams, and Wilson 2007). The AIC and BIC are relative fit indices that allow model comparison, but they do not allow an absolute evaluation of model fit. The higher the AIC and BIC values, the more the data deviates from the specified model (Wu, Adams, and Wilson 2007). In the present study, we used the software ACER Conquest (Wu, Adams, and Wilson 2007) and the R package eRm (Mair and Hatzinger 2007) for parameter estimation.

Specification of a descriptive Rasch model (1PLM)

We specified a one-dimensional 1PLM that reflected the view of SR as a general ability without disentangling the influence of different item features, for example, the SR skills or the text complexity. Hence, the 1PLM provides an estimation of item difficulty without an account of specific item features. The 1PLM showed appropriate mean square (MNSQ) item-fit statistics (0.7 ≤ MNSQ ≤ 1.8; not distorting measurement; Wright and Linacre 1994). The item separation reliability was very high (rel._SEP = .98) and the person reliability was good (rel._EAP/PV = .74) compared with previous SR assessments (Hartmann et al. 2015: 0.54; Mannel, Walpuski, and Sumfleth 2015: 0.76; Krell 2018: 0.64).

In the Wright map in Figure 2, we inspected the distribution of item difficulties in the SRQ and the distribution of person ability in our sample on the same linear scale (equal-interval logits) as computed based on marginal maximum likelihood (MML) estimation. Higher logit values indicate a higher person ability (see Figure 2; green dots represent distribution of participants). The estimated person ability followed a Gaussian distribution. Furthermore, higher logit values indicate a greater item difficulty. For example, Item 4 (i.e., forming hypotheses with two independent variables in the context of urban wildlife ecology; see Supplemental file 2: Appendix 2) was the most difficult item with a difficulty of 1.25 logits (see Figure 2, blue triangles). Based on their difficulty, the items were evenly scattered across the person ability in our sample.

Specification of an explanatory Rasch model (LLTM)

To further explain the difficulty of items, we specified two LLTMs, that is, a basic model and an extended model (see Table 2 for item features that are included in the models). The smaller values obtained from the AIC and the BIC suggested a better fit between model and data for the extended model compared with the basic model (Table 2), as did the log-likelihood difference test (p < .001); however, the extended model still showed an inferior fit compared with the 1PLM based on both the smaller information criteria AIC and BIC and the significant log-likelihood difference test (p < .001). These findings indicate the least deviation between model and data for the 1PLM, followed by the extended model and the basic model.

The item difficulty parameters estimated in the 1PLM positively correlated with those estimated in the basic model (r = .58, p = .011, 95% CI [0.51, 0.64]) and the extended model (r = .62, p = .006, 95% CI [0.55, 0.68]). This means that about 34% (basic model: 95% CI [26, 41]) or 39% (extended model: 95% CI [30, 46]) of the individually estimated item difficulties in the 1PLM can be explained with the respective parameters specified in the LLTMs. The graphical model tests of the basic model in Figure 3a and the extended model in Figure 3b reveal that the items were scattered around the 45° line moderately well.

The estimated α_k parameters (Table 3) showed that all item features contributed significantly to the items’ difficulty because their 95% CI did not include zero. For example, the SR skills of forming hypotheses and analyzing data seemed to be rather difficult (i.e., relatively high positive α_k parameter) compared with testing hypotheses, which served as the reference category in our comparison. As already found in the item parameters of the 1PLM, a higher item complexity reduced item difficulty (i.e., negative α_k parameter). The consideration of text complexity and specialist terms (item surface features) in the present study reduced the estimated effect of the context on item difficulty: The α_k parameters for the contexts of wildlife and of bats were smaller in the extended model compared with the α_k parameters for the contexts in the basic model. This indicates that the difficulty of different research contexts is to some degree related to the use of specialist terms and the complexity of the text that is used to describe the research context.

Implications

Our research provides practical implications for evaluating ILOs in CS (Jordan et al. 2012) as it shows how item features in questionnaires influence the item difficulty. We suggest that practitioners and researchers in CS account for the different SR skills and the number of variables when developing questionnaires to evaluate participants’ SR skills for the investigation of hypotheses. Regarding SR skills that have been less common in evaluations of CS projects, such as forming hypotheses (Stylinski et al. 2020), the blueprint might help to standardize assessment instruments in the different research contexts. We further suggest accounting for the research context in which SR skills have to be applied. The research context influences the item difficulty in our study—in addition to the deep-structure item features that directly relate to SR.

In our study, the explanatory modeling based on item features provided evidence for the validity of the assessment. Following our theoretical assumptions, our results indicate that the research context, the SR skills, and the item complexity accounted for 34% (or 39% with the item surface features text difficulty and specialist terms added) of the item difficulty that individual citizens encountered while solving the assessment items on SR. The substantial amount of variance explained can be traced back to the systematic development of the SRQ in regard to item features. We suggest that practitioners use this blueprint when adapting the SRQ to the research context and participant sample of their CS project.

Furthermore, the analysis of item features revealed that the item difficulty differs depending on the SR skills, the item complexity, and the research context in an assessment of SR. This confirms that the development of items and the interpretation of test scores in SR assessments should consider the particular item features. When comparing participants’ proficiency in SR across CS projects that assess different SR skills or SR in different disciplines, researchers should consider that the test items are not equally difficult. Similar to our overview of item features in this SR assessment on the investigation of hypotheses, future research should describe which science inquiry skills were tested in which contexts and using how many variables in the assessment. Given the number of research contexts and the different science inquiry skills addressed by CS projects, further research on the evaluation of SR in samples of citizen scientists should systematically explore item feature effects.