Phase 1: Developing a citizen science version of the Housing Enabler Screening Tool

The goal of the first phase was to develop a version of the HE Screening Tool (Carlsson et al. 2009) for implementation in the Housing Experiment app, which a member of the public with limited general knowledge about housing accessibility could administer in a valid and reliable way. Researchers and the VA specialist met frequently during this phase. The researchers ensured that high scientific quality was maintained, while the VA specialists used their experience from previous citizen science projects to make sure that the items were easily understood by members of the public with only limited knowledge of housing accessibility and research. An older adult (a User Board member) also gave input. The guiding principles were to a) use simple yet accurate language and terminology; b) keep sentences short; c) avoid double negatives; and d) reduce the number of items considerably without compromising the reliability and validity of the HE. The 60-item list from the HE screening tool was reduced and developed into a list of yes/no questions. As the Housing Experiment focused on entrances and indoor environments, the exterior surroundings section from the HE Screening Tool (20 items) was not included. Six items were excluded due to known negligible variance from previous research. That is, as those items had been present as environmental barriers in more than 95% of all dwellings in previous projects, it was not meaningful to collect further data on these barriers. To enhance readability and comprehension, five items that included two or more measurements were divided into two or three questions. Being particularly careful to avoid collecting any personal data of a sensitive nature, the team developed project-specific questions to include basic descriptive information in the app.

Phase 1 resulted in a set of 39 questions on the presence of environmental barriers at entrances and indoors. Overall, as exemplified in Figure 1, these questions ended up shorter than the HE Screening Tool items, and the language had less technical jargon. Seven descriptive questions addressed type of dwelling, zip code, building year, whether any resident in the dwelling used mobility devices (yes/no), and whether any resident was 65–79 years or 80 years or older (yes/no).

Phase 2: Developing a print mock-up

The goal of Phase 2 was to decide on the format and design of the app and develop a print mock-up version. At this stage, the team was complemented with a design and communications specialist and an additional team member from VA, experienced in leading citizen science projects. They contributed with knowledge on user-friendly design and communication of citizen science projects, however, with limited experience in developing products for older adults. Two employees from a tech development company, which was commissioned for the technological solutions, were engaged to contribute with their knowledge on design, technical features, and solutions. The tech developers and researchers had previously collaborated on projects including HE methodology. An expert with a PhD in digital accessibility and universal design was consulted for feedback on for example colours, fonts, contrasts, and text size. Format and design suggestions were discussed and revised iteratively among researchers, VA specialists, and tech developers. The ambition was to come up with a user-friendly design for both experienced and inexperienced smartphone users.

The results were visualised in a print mock-up version including all questions established in Phase 1. The mock-up included a front page and four sections. Each section had an introductory information page. The first and second sections included information and the seven descriptive questions about the dwelling and the residents. The third and main section included the 39 yes/no questions where the measurements of the dwelling were reported. When needed, pop-up windows were to be available with information or illustrations to help the user take the measurement correctly and to answer the question. In the fourth section, the data collected were to be uploaded to an online open access database. In addition, the person/s reporting the data were able to share their age, write a free text comment, and choose whether they wanted a summary of the data they had collected to be emailed to them. See Figure 2 for example on design.

Phase 3: Developing an app prototype

In Phase 3, the tech developers programmed a prototype of the app, based on the results of Phases 1 and 2, in both iOS and Android versions. Frequent online meetings were held with researchers, VA specialists, and tech developers to discuss challenges regarding content, user-friendliness, design, and technical solutions. At the end of Phase 3, the prototype was ready for a first round of testing. During Phase 3, no members of the public were involved, but when the prototype was ready, it needed to be thoroughly tested, which took place in Phase 4.

Phase 4: Testing and improving usability

The aim of the Phase 4 testing was to improve the usability of the prototype with the goal of arriving at the operational app’s first version. Usability was tested by five older adults as well as all individuals hitherto involved in the development. The older adults were recruited via the Thematic Collaboration Initiative and the senior citizen organisations involved. The older adults who volunteered were fairly experienced in using digital technology and smartphones, they were 65 years or older, both men and women, had no previous experience of citizen science projects, and did not live in any type of age-restricted special housing. They received written instructions on how to use the prototype to measure and record data on environmental barriers in their own dwellings, and were asked to answer a study-specific questionnaire about the app, inspired by the System Usability Scale (Brooke 1996). Instead of a purely quantitative survey approach, we chose a qualitative one to get a deeper understanding of the possible technical and handling issues with the app, to adjust the features in the best possible way. Hence, the original 5-point Likert scale was complemented with open-ended questions, together with a question on what kind of information they thought was needed in an online Housing Experiment instruction manual.

The results of Phase 4 showed that the five older adults were satisfied overall with the usability both in terms of handling the app and in understanding what should be measured and reported. Areas that we identified as needing improvement concerned information pages, opening and closing pop-up information and illustration windows, terminology used for some of the measurements regarding steps and stairs, and clarity of one pop-up illustration window indoors. For example, one older adult commented: “In the questions about the entrance it was sometimes unclear whether they regarded the entrance to the apartment or the entrance to the building, or both.” Another thought the pop-up windows could be improved: “Possibly be made larger and easier to click. And place the closing button in another corner of the screen.”

Revisions to improve the usability were made, and the usability test was repeated. This time, another group of members of the public were approached—students in the Occupational Therapy Bachelor’s program at Lund University (n = 26). The students were undergoing training to administer professional home environment assessments, to identify environmental barriers and fall hazards, and to suggest home modifications for clients with functional limitations, which was considered an adequate setting for additional usability testing. The first author instructed the students on how to use the app to measure and record data on environmental barriers in their own dwellings and to answer the aforementioned usability survey.

Overall, the university students were satisfied with the usability of the app and considered it easy to use. They felt confident using it and thought most people would learn to use it very quickly (see Table 2). Most comments concerned the function of emailing a summary of the data they had collected to the user. Apparently, depending on what email client the participants had on their smartphones, some emails could not be sent, the response was delayed, or the layout of the email appeared as code. In addition, some measurements still needed refinement or clarifications, and additional pop-up illustration windows were wanted. Again, the results of the test were discussed by the involved researchers, VA specialists, and tech developers, followed by revisions into a beta version of the app.

Phase 5: Beta version testing for reliability

To test inter-rater reliability, 18 older adults (65 years and older; eight women, ten men) were recruited via the Thematic Collaboration Initiative, local senior citizen organisations, and patient organisations. They were invited to use the app to measure and record data on environmental barriers in a two-room apartment made available by one of the team members. To counteract any undue influence of the measurements and recording of data among the participants, monitored by one researcher (first and last authors), a maximum of two participants in each section/room of the apartment was allowed at the same time. The participants were instructed to use their own smartphones. However, in the few cases where the app (still in development mode) download failed, participants borrowed a smartphone for the test situation. The two researchers assisted during the testing, provided instructions, and monitored the test situations to ensure compliance. The visits lasted 30–45 minutes each, and the participants received a SEK 200 gift card when the test was completed. To produce data to be used as the gold standard for the evaluation of inter-rater reliability, the last author (developer of the HE methodology) used the app to measure and record data in the apartment. Percent agreement for each participant and each item was calculated by pairwise comparisons between the participant assessment and the gold standard. Using a standard level for accepted agreement (McHugh 2012), rates lower than 80% were considered to warrant further scrutiny and tracing of potential sources of error.

The results of phase 5 showed that of the 39 questions, 24 (61%) had an agreement of 80% or more. The causes of low agreement for the 15 items were varied. Some were caused by misunderstanding questions, forgetting to measure all areas of the dwelling section before answering the question (Are any of the thresholds higher than 1.5 cm?), and not being meticulous enough when using a ruler. Another reason was interpretations needed to answer questions that did not include measurements (Is there enough space for a shower chair in the shower area?). One question, on stairs at the entrance, was misunderstood by several participants. They incorrectly answered “no” and then the app by default filtered out the following four questions regarding bannisters, step height etcetera. We used the results of the inter-rater reliability test for additional app improvements This optimisation included dividing one more question into two questions, so from here on the app included 40 questions. We revised general information on how to measure, and we made further clarifications regarding certain measures in the app as well as in the online manual.

Phase 6: Final tests and finishing touches

While the development process this far had been focused on the main target group (i.e., older adults), a final testing step was made to include other potential members of the public who also might have had an interest in participating in the Housing Experiment. Accordingly, twelve participants including children (aged 10–15 years, n = 5), adults of working age (n = 3), and older adults (n = 4) carried out a final test. They were instructed to use the app in their own dwellings and then answer the usability survey (see Phase 4).

The results of this final test showed that the language and terminology needed further improvement to be comprehended by children. For example, we had to replace the Swedish words for “passing through” and for “kitchen counter” with synonyms. In addition, further clarification and information about certain measurements were necessary. For example, we added a reminder to measure threshold height on both sides of the threshold. There was also a need of reminders to use the pop-up information and illustration windows. To maximise the size of the text despite limited space, we changed font style after the feedback provided. Some coding errors were detected and fixed. In addition, feedback from the children implied that completing all the measurements felt tiresome. Thus, on some information pages between sections we added encouraging feedback such as “The entrance section is complete! You are halfway through! Click “next” to move on to the kitchen.” After the final language refinements, we used the Swedish Lix readability index (Björnsson, 1968) to compare the readability of the original HE Screening Tool questions with the app questions. We found that the average readability score decreased from 45 (average difficulty, cf. newspaper articles) in the original, to 34 (easy to read, cf. popular literature) in the app version (Björnsson 1968). As can be seen in Table 3, the proportion of texts with high (difficult) readability scores was drastically reduced. After these final adjustments we considered, the app ready to use in the Housing Experiment.