The “scientific elite” often has been seen as having a monopoly on the development of knowledge in academic institutions collectively referred to as “ivory towers.” By virtue of their advanced education, specialized skills, and knowledge, researchers have been granted authority, credibility, influence, prestige, and power in the production of knowledge. More recently, however, research has been housed in various types of institutions–university, government, and industrial entities–which have forged new partnerships to create and share knowledge and ultimately contribute to the growth of a “knowledge based society” (Etzkowitz and Leydesdorff 2000). Citizen inclusion also has gained prominence with the emergence of citizen science, democratic science, and community-based science; this has served to further decentralize and extend the creation of knowledge beyond traditional venues (Bottles 2011; Brown 1998; Wallerstein and Duran 2010).
In health sciences, patient involvement has become more important. As “human subjects,” patients have always had a role to play in medical research, but typically it has been a passive one. With the institutionalization of research ethics, emerging patient-partner theories drew attention to research-subject autonomy, welfare, well-being, and the fair distribution of risks and benefits (Veatch 1987). Currently, Patient Engagement in Research (PER) initiatives actively promote the inclusion of patients in health research beyond their traditional role as research participants. PER is broadly defined as the: 1) valuation, mobilization, and legitimization of patient experiential knowledge of living with a particular health condition; 2) conduct of research that focuses on patients’ concerns, participation, and outcomes; and 3) active partnership among a variety of actors (researchers, clinicians, decision-makers, institutions, patients, families) (Rouleau et al. 2018). Some PER initiatives even have “patient partners in research” (PPRs) who participate as collaborators with the researchers (or even co-researchers); the idea behind this collaboration is to recognize patient knowledge to be as important as scholarly knowledge within health research. As such, patients are not only important, they may be considered “equal partners” with researchers. PPRs are valued for their experiential knowledge and are increasingly recognized as repositories of knowledge that escapes researchers (Crocker et al. 2016; Litva et al. 2002; Moorhead et al. 2013).
In this article we will briefly contextualize how PER projects have varied from mere tokenism to the full involvement and empowerment of participant partners–the former as the subject of criticism, and the latter promoted as an ideal (Ives, Damery, and Redwod 2013). We argue that because patient partner contribution is significant and valuable, its lack of recognition may be considered a significant undervaluation. To promote the former and mitigate the incidence of the latter, the implementation of PER should require that institutions establish appropriate recognition of PPRs’ contribution as an ethical requirement. We will explore types of recognition that could be attributed to PPRs, including different types of financial reward or compensation, personal recognition, training and knowledge development, academic recognition, and altruistic rewards. This is intended to initiate further dialogue with PPRs to design a system of recognition that duly acknowledges their contribution to research. We then discuss the potential application of this recognition model to other contributions made by citizen scientists that may not be PPRs, such as the contributions made in environmental sciences.
Public engagement is central to a number of emerging trends in health research, including patient involvement, patient activation, customer engagement, patient-centered care research, community-based research, and citizen science. All of these trends explore the various ways of including patients in research activities. PER has gained traction as evidenced by its increased popularity in specialized institutes and methodological units, dedicated funds, research networks, and academic conferences and journals (Bombak and Hanson 2017; Caron-Flinterman, Broerse and Bunders 2005a; Fleurence et al. 2014; Magwood et al. 2012).
In recent decades, citizen organizations have created and supported movements that promote patient engagement in research. For example, women’s groups have had a significant effect on biomedical research and healthcare in many areas by promoting reproductive rights, providing awareness and inquiries about sterilization abuse of women of color or of First Nations populations, ensuring the integration of breast cancer practices in medical care, and promoting the reduction of the over-medicalization of pregnancy and childbirth (Norsigian 1992). During the HIV outbreak in the 1980s, citizen movements put significant pressure on the FDA to make medication more accessible, enable more experimental use, and provide better public information about the use of placebos in clinical trials. Disability rights movements have been ongoing since the 1800s (Albrecht, Seelman, and Bury 2001). The slogan “nothing about us, without us” became commonplace among contemporary disability movements in which individuals advocated to be included and valued contributors to society, which led to the American Disability Act of the 1990s prohibiting discrimination in all areas of public life (e.g., jobs, schools, transportation) (Charlton 1998).
Moreover, the development of interactive information technology has allowed individuals to network on a global scale. Although there are limitations regarding confidentiality or privacy and the validity of information being shared through social media, the online space has allowed patients not only to have greater access to health information but also to benefit from increased social and emotional support (Moorhead et al. 2013). There are online forums dedicated to all types of health issues, including cancer, arthritis, HIV/AIDS, ankylosing spondylosis, epilepsy, and fibromyalgia (Flickinger et al. 2017; van Uden-Kraan et al. 2008; White and Dorman 2001). Some sites also include chat rooms to foster greater discussion among researchers, patients, and health care providers. This has enabled patient-centered initiatives by fostering ongoing communication and establishing relationships and trust between researchers and partners (Kaye et al. 2012). Recently, some patient organizations have been promoting patient-driven studies. For example, PatientsLikeMe—a private online community originally aimed at individuals with amyotrophic lateral sclerosis (ALS)—now has individuals conducting an observational study of lithium and ALS (Bottles 2011).
In 2012, Lucien Engelen attended a number of conferences about patients needs where, paradoxically, there were no patients, either on stage or in the audience. Engelen launched Patients Included™ to demand that scientists who preach the importance of patient involvement to “walk the talk” and take the necessary steps to include patients (Engelen 2012). Inclusion is not limited to a single interview or conversation, but rather entails a series of cooperative endeavours. To engage patients meaningfully, Patients Included™ operates on the basis of two charters: One for conferences and one for journals (requirements in Table 1).
|Scientific venue||Requirements for Patient Inclusion|
The British Medical Journal (BMJ) also has pioneered a greater role for patients in research. As early as 1999, the BMJ published a thematic issue on patient partnership in research (Coulter 1999) and in 2000, published an issue written by patients who participated in research (Richards and Godlee 2014). In 2012, the BMJ Opinion introduced “Patient Perspectives” to allow patients to engage with the BMJ community and BMJ contributors and to discuss many topics of interest to the patient community, such as healthcare communication, access to records, compassion in care and research, and the use of terms such as “patient-centred care” and “shared decision-making.” In 2014, BMJ launched the patient partnership strategy, which added patient-editors to the journal and a patient panel to advise the editorial staff (BMJ 2018). In addition, BMJ has just announced that all submissions to BMJ Open will now require a patient and public involvement statement (Aldcroft 2018); more specifically, at the end of the Methods section, each paper is required to describe patient involvement in informing the development of research questions, study design, recruitment, and dissemination of results.
A growing number of venues publish research about PER and PPRs (Huan Xu and Wong 2017). In addition to the BMJ series, journals that either have a main focus on PER or are patient-friendly include, among others: Research Involvement and Engagement (BMC 2019), Journal of Participatory Medicine (Society for Participatory Medicine 2017), The Patient-Patient-Centered Outcomes Research (SpringerNature 2018), Value in Health (International Society for Pharmacoeconomicas and Outcomes Research 2018), and Health Expectations (Wiley Online Library 2018). In 2015, the first journal co-edited by a researcher and a patient, named Research Involvement and Engagement, was launched. The journal seeks to be as accessible as possible both by being open access and by its interdisciplinary perspective dedicated to the co-production of knowledge. The growing venue for PER scholarship is a sign of the importance, and the many ways, of involving patients and recognizing their contribution to research.
To offer further institutional support for PER, major funding agencies have included patient involvement as a funding criterion in the governance, priority setting, and conduct of research. The National Institute of Health Research (NIHR) in the UK, the Canadian Institutes of Health Research (CIHR), and the Patient-Centered Outcomes Research Institute (PCORI) in the US have stated that health research should aim to meaningfully involve patients as partners rather than simply as participants (CIHR 2018; NIHR n.d.; PCORI 2017). New funding schemes are expressly designed to support the active involvement of patients. For instance, PCORI is dedicated to funding research projects that involve patients, caregivers, and the healthcare community, while CIHR has largely invested in both funding opportunities and methodological units supporting PER across the country (CIHR 2018).
Health researchers – often from a separate class and walk of life than many in society – might be disconnected from the values enshrined in the populations that they study. Tapping into “experiential knowledge” from PPRs surely provides relevant and different insights than those provided through scholarly education. The development of experiential knowledge is well explained by Caron-Flinterman et al. (2005b: 2567): “experiential knowledge arises when these experiences are converted, consciously or unconsciously, into a personal insight that enables a patient to cope with individual illness and disability. When patients share experiential knowledge, the communal body of knowledge exceeds the boundaries of individual experiences.” In this context, it has been suggested that experiential knowledge may be a worthy complement to scholarly research (Caron-Flinterman, Broerse, and Bunders 2005b).
Experiential knowledge may create a significant epistemological shift by tapping into worlds of knowledge that are still underdeveloped in health-based research. Proximity to individuals who may openly share experiential knowledge may create empathy and compassion, which in turn may motivate researchers to better prioritize the needs of patients in their work. Moreover, experiential knowledge may provide insight into patient populations typically excluded–intentionally or unintentionally–from research. The study of vulnerable populations is often challenging because of methodological difficulties such as recruiting and retaining participants and ill-suited measurement or instrumentation that may bias data collection (Flaskerud and Winslow 1998). Representation from such populations may help to adapt research methods, tailor context-specific interventions, and develop ways to fairly and effectively disseminate and translate research results (Fagan et al. 2016).
A study by Bélisle-Pipon, Rouleau, and Birko (2018) found that early career researchers suggested a list of contributions that may benefit from the involvement of both PPRs and researchers. We have included these contributions in Figure 1 to show how all steps of the research can be provided using a mix of scholarly knowledge and experiential knowledge. In this “ideal” model, the tasks may be shared in various ways, but what creates synergy is the mixed input from diverse types of knowledge. However, not all PER projects are created equal. They can involve participation in any of the range of research responsibilities, from governance and priority setting, to contributions in research conduct (identification of research needs, research design, recruitment, data collection, analysis, and interpretation), and finally, to knowledge transfer and the evaluation of research impacts.
If experiential knowledge is used simply when convenient and given no (or insignificant) recognition, then this suggests a degree of tokenism and undervaluation. The most integrated form of PER research happens when PPRs are co-contributors to research on equal footing with researchers. According to Shippee et al. (2015), PER comprises four main components: Patient initiation (initiating lay individuals to the research process); building reciprocal relationships (valuing PPRs as equal partners and important components of a research team); co-learning (researcher is open to learning from PPRs and avoids monopolizing agendas and conversations); and re-assessment and feedback (continually evaluating and improving team practices and fostering PPRs empowerment). An authentic collaboration—or, as stated by Crocker (2016), a “fully intertwined” partnership—has more chance of yielding results that reflect PER-expected outcomes (Wilson et al. 2015).
Various types of PER outcomes have been the object of many empirical studies. In the UK, Mockford et al. (2012) conducted a systematic review to evaluate the impact (in terms of services and economic costs) of the broad concept of “user involvement” (N = 248 studies) in the UK National Health Service health care (1997–2009); their findings suggest that the literature contains insufficient detail to adequately assess parameters, especially regarding economic costs. Domenecq et al. (2014) also conducted a systematic review of studies in peer-reviewed journals and grey literature limited to patient engagement in research (N = 142 studies); they suggest that there is a broad spectrum of engagement which may lead to different types of benefits, including augmenting patient enrollment and reducing attrition, identifying better mechanisms of information dissemination, and reporting more meaningful and understandable results for participants and communities. In that same vein, Brett et al. (2014) conducted a systematic review of 66 published studies in health and social care research and suggested that an enhanced quality and appropriateness of research results when more user-focused applications are applied throughout the research process. However, they also noted several challenges, including ethical conflict in protocol design, tokenism, power struggles, difficulty in recruiting diverse patients, additional time, and cost.
Generally, the body of empirical evidence points to a lack of systematic definition or framework for PER (Carroll et al. 2017), which hampers evaluation of past projects. In other words, conducting empirical research to evaluate and measure the success of PER projects is highly problematic given that they are so diverse. But despite the many challenges, issues, and limitations, studies report the beneficial impacts of PER much more often than the negative aspects (Brett et al. 2014; Domecq et al. 2014). This does not mean that PER should be systematically introduced to all health research, but it does suggest that when responsibly and effectively applied, PER can yield worthwhile results.
While there is growing impetus to recognize the contribution of patients and even to promote equality in patient partner research, the valuation of, and the attribution of merit to, PPRs remain important outstanding challenges. This concern seems to be tangential or even ignored (Abma, Nierse, and Widdershoven 2009). While it may be acknowledged that patients who are considered partners do not have the same needs and expectations as researchers, their experiential knowledge should not be ignored or undervalued. To do so could imply that this knowledge is less important and thus need not be recognised, which may create a paradoxical context in which PER meant to promote experiential knowledge may actually diminish in value.
According to a study by Hamilton et al. (2018), “feeling valued” is essential for positive team interactions and an inclusive research environment. Their participants expressed the desire “to feel equally important on the research team, with appropriate recognition and respect” (Hamilton et al. 2018: 430). Two conditions are essential: 1) acknowledgement of the value of contributions made by PPRs, and 2) sufficient compensation for their contributions. Without both conditions, PPRs may feel as if they have been involved, but not listened to and meaningfully considered (a tokenistic approach); worse, they may feel disposable. In the politics of recognition, the identity of a person or a group can be shaped by the extent and nature of recognition, and misrecognition or non-recognition can actually be seen as oppression (Taylor et al. 1994). Although this may seem somewhat surprising, one may argue that not valuing experiential knowledge is a form of scholarly domination.
Conversely, individuals with insightful experiential knowledge may feel unable to express themselves due to a sense of self-deprecation in the presence of researchers or scholars with explicitly valuable or recognized knowledge. This is counterproductive to effective PER and may lead to some form of tokenism. For individuals to volunteer or share their personal, intimate knowledge of illness–often gained in unpleasant or painful circumstances–both the researcher and the PPRs must recognize its potential use to allow it to be fully expressed, valued, and applied in knowledge production, even if mutual understanding is often difficult (Bélisle-Pipon, Del Grande, and Rouleau 2018). The patient must also be made aware that his or her knowledge is an asset “worth” sharing in knowledge production. As Taylor (1994: 26) mentions, “due recognition is not just a courtesy we owe people. It is a vital need.” Through subconscious moral and psychological processes, an individual may often register the value of an act through its recognition in a social system.
The failure to adequately recognize the value of research subjects has been discussed in recent decades, usually in the context of past patient exploitation (Bentley and Thacker 2004; Grady 2005). Other than limited anecdotal or case-based findings respecting payment, acknowledgement, compensation, participation, and sense of meaning in volunteering or altruism, no concerted debate about adequate valuation of patient participation or contribution in PER has taken place. At this time, recognition is typically limited to some form of reimbursement (e.g., travel, child care, accommodation, food, parking fee reimbursement, and some compensation for loss of salary), although a few centers have raised grants to give prizes, payments, or per diem for time spent working on the project (Bagley et al. 2016; Hewlett et al. 2006). One may consider as an example the “Cost Calculator,” which helps to budget for the costs of including people in studies (INVOLVE 2018). Recognition is seen as mainly pecuniary, and there is little consideration of other forms of appreciation or credit. By compensating “partners” in PER in the same way as the typical research participant, which often seems to be the case, one may conclude that the contributions of each are similar. But this could not be further from the truth–patient partners contribute much more substantially and actively than research subjects.
Pandya-Wood et al. (2017) point out that the display of gratitude in PER is based on principles of respect and integrity. They affirm that valuing patient expertise and involvement reinforces further meaningful, constructive interactions, while also mitigating disempowerment and marginalization. However, the laconic term “reward and recognition” is mentioned without fleshing out or detailing what it implies. While the authors focus primarily on financial implications, they also acknowledge that some patient partners may prefer other types of recognition such as further training and continuing access to health information. This serves to perpetuate financial compensation as the primary means of valuation of patient involvement in PER, and only when that is unacceptable might other types of acknowledgment be considered. Meaningful recognition that goes beyond rhetoric to underscore the full value of the partnership that binds patients and researchers in PER is currently lacking.
Various ways to recognize the mobilization of the experiential knowledge of PPRs are proposed in Table 2. The various categories of recognition–financial, personal, knowledge, academic, and altruistic–are not mutually exclusive. As we consider ways to recognize the contributions of PPRs, knowing what they say they want is also critical. We believe that this paper is only the beginning of a much more significant discussion with PPRs and stakeholders who may influence such projects (e.g., funding agencies, research institutions, or patient advocacy institutions).
It is important to acknowledge at the outset that involvement of PPRs in a research project may be sporadic and limited, interrupt their regular work schedule, and result in wage loss. Organizations such as the Centre of Excellence on Partnership with Patients and the Public (https://ceppp.ca/en/) have set an hourly wage for PPRs at about $50 to recognize that they bring necessary knowledge to research conduct, and also to compensate for any loss of wages or job opportunities as a result of their sporadic involvement (Centre of Excellence on Partnership with Patients and the Public, personal communication). When PPRs occupy more stable and long-term positions within a research team and/or their involvement is related to specific research tasks (e.g., recruitment, data analysis), their salary has to follow established university pay standards (e.g., collective agreement or median salary proportional to the experience, both of which should potentially be revised to include specific provisions for PPRs). Financial compensation has always been a contentious issue in research on human subjects (Grady 2005). Many ethicists, researchers, and oversight committee members are concerned that offering too much money to individuals, especially those who are socioeconomically disadvantaged, could threaten their ability to consent to research by unduly influencing them to participate against their better judgment. However, the PER context is fundamentally different from other types of research in that PPRs are recognized mainly for their contributions as partners, which may extend beyond that of trial subjects. For example, to better understand why some patients comply with cancer treatments while others do not, researchers could work with cancer survivors in remission who are no longer part of such trials. Motivated to share their experiential knowledge, PPRs can contribute to many steps of the research as suggested in Figure 1. Their responsibilities and roles as partners are similar to those of a researcher with an active role, and differ markedly from the role of a passive research participant.
Pay standardization ensures that PPRs are treated similarly to salaried researchers. While this may be highly desirable, it may not be affordable for many research initiatives. Bélisle-Pipon, Rouleau, and Birko (2018) indicate that financial compensation may be more difficult to grant in certain types of projects, particularly those conducted by early career researchers with access to smaller grants than those available to large research teams. In such cases, recognition of patient involvement may take different forms that are personal, involve knowledge sharing, and, when relevant, may be academic. Financial recognition is material and has a measurable “bottom-line” significance, but other forms of recognition also are critical in ensuring that PPRs feel that they are valued, that they have helped their community, and that they identify with the success of the research project and the advancement of scientific knowledge. PPRs engaged in research projects may be amenable to these alternatives; they are more able to understand and recognize the resource limitations of a research project than researchers may think. When recognition of a contribution is part of an ongoing dialogue between researchers and PPRs in a partnership relationship, it is easier to agree upon what constitutes appropriate recognition.
Researchers should not underestimate the value of personal recognition as a means of acknowledging the effort of PPRs to mobilize their experiential knowledge, contribute to research, and ultimately serve their peers and the broader community of patients. These kinds of recognition are typically inexpensive and rather simple, but they are too often overlooked by already busy researchers. A researcher could send a thank-you letter at the end of a study, or, instead of waiting for a project to end, could incorporate recognition during a project or even for some time afterward. For example, the research team’s website can list PPRs as team members, and when the team participates in activities and disseminates the progress of their work in newsletters, PPRs could be informed and acknowledged or mentioned to underscore their relevance to the research. In establishing communication protocols at the outset, PPRs should be consulted to ensure that confidentiality and privacy concerns are addressed. This is particularly significant in sensitive research contexts (e.g., research on communicable diseases and mental illness) where disclosing participation of PPRs could cause them harm.
Because PER is based on mutual learning (i.e., researchers and PPRs learn from each other), knowledge recognition also may be appreciated. The first step is to provide access to research results to which PPRs have contributed, in a form they can readily understand. Additionally, providing access to related scientific literature in a readily available format (e.g., summary of the results) serves to include and recognize PPRs as equal contributors. In addition to disseminating research information, the research team or the institution with which it is affiliated may proactively set up training opportunities that allow PPRs to learn more about health topics and modalities of particular interest to them. Establishing a forum of dialogue and interaction also is effective in maintaining a bond and valuing the patient-researcher partnership after a research project has ended. As previously mentioned, citizens and patient communities often form virtual communities (or e-communities) that undertake citizen-based research. For patients who may feel isolated and alone, online communities can provide a link to share information with others living with similar health issues, and this virtual access allows those with mobility issues to reach out to others. Interested researchers can join such communities to inform patients about ongoing research and relevant findings or breakthroughs. Too often significant research is conducted without affected groups of patients being informed of its existence. The ongoing communication of knowledge extends the patient-researcher relationship beyond one project by keeping patients “in the loop,” and as a result, they may no longer feel “forgotten” or disposable.
The traditional goal of research is the production of new scholarly knowledge. Failure to acknowledge the contribution of PPRs to this goal could be unfair. It has been suggested that in certain circumstances, the contribution of patients in PER warrant recognition through authorship on publications (Bélisle-Pipon, Rouleau, and Birko 2018). Participants in this study explored the ethical ramifications (or implications) of engaging patients in research; they agreed that authorship might foster fair recognition of both patients and researchers in PER. Other researchers have concurred that partners should be granted authorship on papers, abstracts, and conference materials (Hewlett et al. 2006). This approach seems logical and consistent if one accepts that the contribution of patients is in some measure important, and even equal to that of the researcher. In this case both should be recognized and valued in a similar manner. This would have major significance in science, where authorship is the primary means of recognizing individuals who contribute most to knowledge.
While researchers gain credibility from authorship (e.g., earning more grants and scientific recognition), PPRs will probably not be recognized or rewarded in the same way. Authorship is valuable as “social capital” only within the system of science; it is a form of currency that PPRs cannot use. Although authorship may be a form of recognition that leaves PPRs with a greater sense of worth, recognition given to the researcher is not really comparable to what PPRs would receive, so one must question whether authorship is actually an equitable solution. However, there are cases where PPRs have played a role similar to that of a co-investigator and as such, recognition has to be considered within the parameters of authorship; failure to recognize their contribution would be ethically inappropriate and could reduce their sense of public accountability and responsibility for the project. Although authorship is defined in various ways depending on the field of research, most would accept that it represents two important aspects of science: 1) substantial contribution and 2) responsibility and accountability for the work. It is unclear, and perhaps even doubtful, that PPRs who make substantial contributions would agree to be accountable for the body of work within the realm of science. It might be useful to consider a “workaround” that would assign some measure of responsibility and accountability of the PPRs for the quality of experiential knowledge but not for the scientific accuracy of such knowledge. This more focused approach would mitigate worries about possible confusion that might adversely affect the quality or scientificity of research. There still remains fundamental incongruence respecting shared authorship in PER.
Although no authorship norms consider the specificity of PER research, there are many guidelines for authorship, including the International Committee of Medical Journal Editors guidelines and the Committee on Publication Ethics. There is also literature that considers best practices for authorship in multidisciplinary teams, which are somewhat appropriate given the diversity of norms and types of contributions in multidisciplinary research (Smith and Master 2017). We propose the following procedures for further discussion about authorship and expected contributions and responsibilities in research (Table 3). We have adapted the previous best practices to typical PER research tasks. Our process makes it clear that when PPRs do not meet the criteria for being co-authors of a publication, their contribution must at least be recognized in the Acknowledgments and, if the journal permits, in a PPRs involvement statement.
|Step 1: Outline Roles (during initial project planning)||
|Step 2: Determine Authorship Order (at the beginning of conducting research)||
|Step 3: Continuous Dialogue||
|Step 4: Manuscript Writing||
|Step 5: Final Decision on Contributorship, Authorship Order, and Acknowledgement||
|Step 6: Draft a Declaration on Contributorship and Authorship Order and the Acknowledgment||
Fairness is crucial in the allocation of authorship to ensure that individuals are duly recognized for their contributions. This includes recognizing those who have contributed in ways that allow them to be considered co-authors, and also ensuring that others are recognized in a manner that is respectful of authorship criteria. Otherwise, established authorship parameters could be breached, which could foster unethical publication practices, be they gift-authorship (individuals who do not meet authorship criteria are named as authors) or the unfair exclusion of partners (individuals who have contributed to a publication, but are excluded from authorship because their contribution has not been valued fairly). Such practices contradict the aims of inclusivity and fair recognition imbedded in PER’s DNA, and would cast skepticism on the recognition of the contribution of PPRs to research.
Lastly, some individuals will contribute to PER out of altruism. An individual may feel a sense of self-worth from doing something that is generally perceived as being “good.” That self-worth may in itself be an important benefit to an individual. In other words, one may feel morally “good” for helping in the advancement of science. Patients with chronic health conditions may become involved in PER to reduce feelings of distress and powerlessness by actively working to minimize suffering and find solutions to their illness. Some may feel guilty that they have lived through an illness and survived while others have not; as a result they may act in part out of compassion for other sufferers, but also to ease their personal guilt through PER participation. These examples would suggest that altruistic acts are egoistically motivated to some extent. However, research in psychology suggests that empathetic emotions may bring on purely altruistic actions without the intent of helping oneself (Feigin 2014). Although altruistic behavior may be satisfying and rewarding to oneself, this may not necessarily be the primary motivation for PPRs. Empathy-based altruism has been found to reduce stigmatization, promote cooperation, and enhance mutual care (Batson et al. 1997; Oliver et al. 2012; Van Lange 2008); ethically, this would suggest that empathetically motivated altruistic actions that contribute to PER may have a more communal objective to improve the wellbeing of people for whom there is empathy.
There is no clear gold standard for conducting PER (Patrick, Kebbe, and Aubin 2018) or for best practices for recognizing the contributions if PPRs are still lacking. The process laid out in Table 3 for determining contributorship and authorship order in PER does not imply that including PPRs as co-authors must become the norm or a common practice. Authorship is one of the many ways to recognize and value the contribution and involvement of PPRs. But when appropriate, best practices may provide basic guidance on how to determine contributorship and authorship order in PER fairly. We recommend that research teams use a vast array of ways to recognize PPRs and promote more than one type of recognition (financial, personal, knowledge, academic, and altruistic).
The goal of this paper is to develop options for acknowledging and recognizing patients’ experiential knowledge within the scope of PER in health research. It is important to mention that the recognition of citizens within the broader concept of citizen science has created similar discussions (Dickinson et al. 2012). Some citizen science partnerships are very similar to PER in that they use valuable experiential knowledge. For example, researchers have been creating partnerships with culturally specific groups — including First Nation, aboriginal, native, or tribal communities — to promote more efficient health services that are more attuned to cultural specificities (Price-Robertson and McDonald 2011; Rae et al. 2013). These underserved populations may lack trust in researchers, so having someone from the community as a partner collaborator in research may help to address health priorities, ensure bi-directional learning opportunities for the researchers and members of the community, and build research capacity (Woodahl et al. 2014). Cultural context, beliefs, and traditions impact a patient’s lived experience and provide additional insight that informs academic knowledge. Knowledge of a specific culture can be comparable to experiential knowledge of a patient in that their lived experiences provide specific insights that complement academic knowledge.
However, other citizen science projects rely on contributions that are different in nature than experiential knowledge, including scientist-designed projects for which volunteers primarily contribute to data collection or analysis. For example, a non-profit called Lost Bird developed a project called Smartfin (Lost Bird 2018), which analyzes ocean chemistry via sensors on surfers’ boards. This type of contribution to research allows for large-scale data collection in geographically diverse settings, which is significant because of the number of individuals, not because of the type or amount of contribution made by any one individual. These citizens are not co-creators in the project as may be the case of PPRs. The actual contributions of citizen scientist in the Smartfin project should be valued with different justifications and expectations than PPRs. Indeed, stakeholders in citizen science have discussed collective acknowledgment, co-authorship, citizen ownership, increasing public awareness, giving back to the community through research, and providing citizens access to research (e.g., open access publication) (Resnik, Elliott, and Miller 2015; Riesch and Potter 2014). There exists an important diversity of types of citizen science and PER, and it is important to find ways in which contributions are fairly and adequately recognized and acknowledged for each.
Not all contexts are “ideal,” and if experiential knowledge conflicts with knowledge based on evidence or scholarly argument, researchers may find themselves questioning the validity of experiential knowledge, especially because experiential knowledge is not verifiable, replicable, or generalizable in the same way as traditional scientific research. Some have suggested that researchers are now more open to considering other types of knowledge in the belief, and perhaps the hope, that there may be more than one type of absolute knowledge (Caron-Flinterman, Broerse, and Bunders 2005b).
Although one may wish to be inclusive, too many perspectives or types of knowledge creates the potential for “scope creep” (Domecq et al. 2014), which can be defined as the constant reshifting and disruptions of research to the extent that it becomes impossible to meet deadlines and respect resource constraints. Although no one would wish for scope creep, making partial or superficial use of experiential knowledge only when convenient or self-serving could draw aspersions of tokenism (Bélisle-Pipon, Rouleau, and Birko 2018). Research has been labelled as tokenistic PER when patients are said to be included in a very superficial manner; this has been defined as “the practice of making perfunctory or symbolic effects to engage communities or patients” (Hahn et al. 2017). This may occur if patient or citizen preferences or opinions are never duly considered, if they are not empowered to share their experiences, or if their contributions are not valued.
Despite an openness to change, the availability of new funding schemes, and the draw of PER benefits, critics warn that vulnerable citizens may still be instrumentalized. Carman et al. (2013: 226) indicated that the implication of patients is not as authentic and meaningful as hoped and that it is “still rare for patients to have more than a token amount of power and influence.” Ives et al. (2013: 184) are rather skeptical about “the coherence of a fully democratic cooperative model in the context of current arrangements for the conduct of research” and are wary that PER’s “espoused benefits can never be fully realised.” Gallivan et al. (2012) noted that the current lack of consensus and understanding about the expected roles, responsibilities, and contributions of PER stakeholders (including PPRs, citizen scientists, and researchers) may often be the main obstacles for authentic and effective citizen engagement. Hamilton et al. (2018) remarked that the notion of meaningful engagement has not been empirically studied and is often anecdotally invoked. These dimensions are essential to consider, because as Richards and Godlee (2014: 2) indicate, “patient partnership is a lot easier to talk about than to realize.” This can surely be extended to citizen science, which is much more diverse and wide reaching.
While research does have a history of patient inclusion, the institutionalization and normalization of this practice is still in its infancy. Interestingly, early career researchers have estimated that PER may well become mainstream within the next decade (Rouleau et al. 2018). This gives time for the research community, research ethics boards, and funding agencies to develop guidelines and to adopt best practices based on the suggested valuation options for patients’ experiential knowledge. To date, neither PCORI, CIHR, nor NIHR have developed guidance efficiently tackling the issue of fair recognition for contributions to research. While certain research ethics challenges have been discussed, including ethical rationales for patient engagement and justice-related issues, these major institutions in PER have yet to provide guidelines for respecting patient partners’ contributions and labour (Canadian Institutes of Health Research 2011; Ellis and Kass 2017; INVOLVE 2016; Patient-Centered Outcomes Research Institute 2010).
In this paper, we discussed the added value of PER and PPRs; we focused on the lack of valuation of contribution of PPRs, and underscored the inherent inequity which may be seen as tokenism. In so doing, we considered the value of various types of contributions that experiential knowledge brings to research. When the PPRs share significant experiential knowledge, and they have contributed equally or significantly, that contribution must be recognized accordingly. It should be an ethical requirement enshrined within the institutionalization of PER to promote good practices empowering PPRs. Failure to do so only increases the risk of undervaluation and, understandably, intensifies the reluctance of patients to participate in research.
Because patient involvement has emerged as an imperative and is being institutionalized, fair recognition will continue to gain interest; the gap between idealistic rhetoric and reality will likely become more apparent and require remedy. Further discussions with stakeholders are essential to develop ethical, practical, and feasible value systems in PER. Notably, valuation for PPRs (whether it is financial, personal, academic, or altruistic) must be considered within specific contexts where PER is commonly practiced. Different communities may have different needs and value systems that make one type of recognition more important to certain individuals, as such recognition does require some contextual flexibility and sensibility. Effective recognition in PER may help patient partners better understand the importance of their contributions and foster their sense of responsibility to share their experiential knowledge. Future discussion will be necessary as the contributions of citizens to various types of research become commonplace.
Elise Smith is supported by a collaborative fellowship from the US National Institute of Health (NIH) and the Fonds de Recherche du Québec en Santé. Jean-Christophe Bélisle-Pipon is funded by postdoctoral fellowships from the Canadian Institutes of Health Research (CIHR), the Québec Health Research Fund (FRQS), and the Québec SPOR-SUPPORT Unit. David Resnik’s work is supported by the Intramural Program of the US National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). This paper does not represent the views of the NIEHS, NIH, or any governmental institution.
The authors have no competing interests to declare.
Abma, TA, Nierse, CJ and Widdershoven, GAM. 2009. Patients as Partners in Responsive Research: Methodological Notions for Collaborations in Mixed Research Teams. Qualitative Health Research, 19(3): 401–415. DOI: https://doi.org/10.1177/1049732309331869
Aldcroft, A. 2018. New requirements for patient and public involvement statements in BMJ Open. BMJ Open. Available at: http://blogs.bmj.com/bmjopen/2018/03/23/new-requirements-for-patient-and-public-involvement-statements-in-bmj-open/.
Bagley, HJ, Short, H, Harman, NL, Hickey, HR, Gamble, CL, Woolfall, K, Young, B and Williamson, PR. 2016. A patient and public involvement (PPI) toolkit for meaningful and flexible involvement in clinical trials – a work in progress. Research Involvement and Engagement, 2: 15. DOI: https://doi.org/10.1186/s40900-016-0029-8
Batson, CD, Polycarpou, MR, Harmon-Jones, E, Imhoff, HJ, Mitchener, EC, Bednar, LL, Klein, TR and Highberger, L. 1997. Empathy and Attitudes: Can Feeling for a Member of a Stigmatized Group Improve Feelings Toward the Group? 14.
Bélisle-Pipon, J-C, Rouleau, G and Birko, S. 2018. Early-career researchers’ views on ethical dimensions of patient engagement in research. BMC Medical Ethics, 19: 21. DOI: https://doi.org/10.1186/s12910-018-0260-y
Bentley, JP and Thacker, PG. 2004. The influence of risk and monetary payment on the research participation decision making process. Journal of Medical Ethics, 30(3): 293–298. DOI: https://doi.org/10.1136/jme.2002.001594
BMC. 2019. Research Involvement and Engagement. Available at: https://researchinvolvement.biomedcentral.com/ [Last accessed 4 June 2018].
BMJ. 2018. Patient panel members. Available at: https://www.bmj.com/about-bmj/advisory-panels/patient-panel-members [Last accessed 4 June 2018].
Bombak, AE and Hanson, HM. 2017. A Critical Discussion of Patient Engagement in Research. Journal of Patient-Centered Research and Reviews, 4(1): 39–41. DOI: https://doi.org/10.17294/2330-0698.1273
Bottles, K. 2011. The rise of citizen scientists and patient initiated research. KevinMD.com. Available at: https://www.kevinmd.com/blog/2011/12/rise-citizen-scientists-patient-initiated-research.html [Last accessed 30 May 2018].
Brett, J, Staniszewska, S, Mockford, C, Herron-Marx, S, Hughes, J, Tysall, C and Suleman, R. 2014. A Systematic Review of the Impact of Patient and Public Involvement on Service Users, Researchers and Communities. The Patient – Patient-Centered Outcomes Research, 7(4): 387–395. DOI: https://doi.org/10.1007/s40271-014-0065-0
Carman, KL, Dardess, P, Maurer, M, Sofaer, S, Adams, K, Bechtel, C and Sweeney, J. 2013. Patient and family engagement: A framework for understanding the elements and developing Interventions and policies. Health Affairs, 32(2): 223–231. DOI: https://doi.org/10.1377/hlthaff.2012.1133
Caron-Flinterman, JF, Broerse, JEW and Bunders, JFG. 2005a. The experiential knowledge of patients: A new resource for biomedical research? Social Science & Medicine, 60(11): 2575–2584. DOI: https://doi.org/10.1016/j.socscimed.2004.11.023
Caron-Flinterman, JF, Broerse, JEW and Bunders, JFG. 2005b. The experiential knowledge of patients: A new resource for biomedical research? Social Science & Medicine, 60(11): 2575–2584. DOI: https://doi.org/10.1016/j.socscimed.2004.11.023
Carroll, SL, Embuldeniya, G, Abelson, J, McGillion, M, Berkesse, A and Healey, JS. 2017. Questioning patient engagement: Research scientists’ perceptions of the challenges of patient engagement in a cardiovascular research network. Patient preference and adherence, 11: 1573–1583. DOI: https://doi.org/10.2147/PPA.S135457
Charlton, JI. 1998. Google-Books-ID: L_WZaVAoS4EC. Nothing about Us Without Us: Disability Oppression and Empowerment. University of California Press. DOI: https://doi.org/10.1525/california/9780520207950.001.0001
CIHR. 2018. SPOR SUPPORT Units – CIHR. 10 January 2018. Available at: http://www.cihr-irsc.gc.ca/e/45859.html [Last accessed 27 May 2018].
Coulter, A. 1999. Paternalism or partnership?: Patients have grown up—And there’s no going back. BMJ: British Medical Journal, 319(7212): 719. DOI: https://doi.org/10.1136/bmj.319.7212.719
Crocker, JC, Boylan, A-M, Bostock, J and Locock, L. 2016. Is it worth it? Patient and public views on the impact of their involvement in health research and its assessment: A UK-based qualitative interview study. Health Expectations, 20(3): 519–28. DOI: https://doi.org/10.1111/hex.12479
Dickinson, JL, Shirk, J, Bonter, D, Bonney, R, Crain, RL, Martin, J, Phillips, T and Purcell, K. 2012. The current state of citizen science as a tool for ecological research and public engagement. Frontiers in Ecology and the Environment, 10(6): 291–297. DOI: https://doi.org/10.1890/110236
Domecq, JP, Prutsky, G, Elraiyah, T, Wang, Z, Nabhan, M, Shippee, N, Brito, JP, Boehmer, K, Hasan, R, Firwana, B, Erwin, P, Eton, D, Sloan, J, Montori, V, Asi, N, Abu Dabrh, AM and Murad, MH. 2014. Patient engagement in research: A systematic review. BMC Health Services Research, 14: 89. DOI: https://doi.org/10.1186/1472-6963-14-89
Ellis, LE and Kass, NE. 2017. How are PCORI-funded researchers engaging patients in research and what are the ethical implications? AJOB Empirical Bioethics, 8(1): 1–10. DOI: https://doi.org/10.1080/23294515.2016.1206045
Engelen, L. 2012. No-Show patients on conferences: Patients Included™. Lucien Engelen. Available at: https://lucienengelen.com/2012/02/15/no-show-patients-on-conferences/ [Last accessed 26 March 2018].
Etzkowitz, H and Leydesdorff, L. 2000. The dynamics of innovation: From National Systems and ‘“Mode 2”’ to a Triple Helix of university–industry–government relations. Research Policy, 29: 109–123. DOI: https://doi.org/10.1016/S0048-7333(99)00055-4
Fagan, MB, Morrison, CR, Wong, C, Carnie, MB and Gabbai-Saldate, P. 2016. Implementing a pragmatic framework for authentic patient–researcher partnerships in clinical research. Journal of Comparative Effectiveness Research, 5(3): 297–308. DOI: https://doi.org/10.2217/cer-2015-0023
Flaskerud, JH and Winslow, BJ. 1998. Conceptualizing Vulnerable Populations Health-Related Research. Nursing Research, 47(2): 69. DOI: https://doi.org/10.1097/00006199-199803000-00005
Fleurence, RL, Curtis, LH, Califf, RM, Platt, R, Selby, JV and Brown, JS. 2014. Launching PCORnet, a national patient-centered clinical research network. Journal of the American Medical Informatics Association, 21(4): 578–582. DOI: https://doi.org/10.1136/amiajnl-2014-002747
Flickinger, TE, DeBolt, C, Waldman, AL, Reynolds, G, Cohn, WF, Beach, MC, Ingersoll, K and Dillingham, R. 2017. Social Support in a Virtual Community: Analysis of a Clinic-Affiliated Online Support Group for Persons Living with HIV/AIDS. AIDS and Behavior, 21(11): 3087–3099. DOI: https://doi.org/10.1007/s10461-016-1587-3
Grady, C. 2005. Payment of clinical research subjects. The Journal of Clinical Investigation, 115(7): 1681–1687. DOI: https://doi.org/10.1172/JCI25694
Hahn, DL, Hoffmann, AE, Felzien, M, LeMaster, JW, Xu, J and Fagnan, LJ. 2017. Tokenism in patient engagement. Family Practice, 34(3): 290–295. DOI: https://doi.org/10.1093/fampra/cmw097
Hamilton, CB, Hoens, AM, Backman, CL, McKinnon, AM, McQuitty, S, English, K and Li, LC. 2018. An empirically based conceptual framework for fostering meaningful patient engagement in research. Health Expectations, 21(1): 396–406. DOI: https://doi.org/10.1111/hex.12635
Hewlett, S, de Wit, M, Richards, P, Quest, E, Hughes, R, Heiberg, T and Kirwan, J. 2006. Patients and professionals as research partners: Challenges, practicalities, and benefits. Arthritis Care & Research, 55(4): 676–680. DOI: https://doi.org/10.1002/art.22091
Huan Xu, R and Wong, EL-Y. 2017. Citation Classics in Patient Engagement Research: An Analysis of the 50 Most Cited Articles. Open Journal of Nursing, 07(06): 630–644. DOI: https://doi.org/10.4236/ojn.2017.76047
International Society for Pharmacoeconomicas and Outcomes Research. 2018. Value in Health. Available at: https://www.valueinhealthjournal.com/ [Last accessed 4 June 2018].
INVOLVE. 2018. Involvement Cost Calculator. Available at: http://www.invo.org.uk/resource-centre/payment-and-recognition-for-public-involvement/involvement-cost-calculator/ [Last accessed 2 May 2018].
Ives, J, Damery, S and Redwod, S. 2013. PPI, paradoxes and Plato: Who’s sailing the ship? Journal of Medical Ethics, 39(3): 181–185. DOI: https://doi.org/10.1136/medethics-2011-100150
Kaye, J, Curren, L, Anderson, N, Edwards, K, Fullerton, SM, Kanellopoulou, N, Lund, D, MacArthur, DG, Mascalzoni, D, Shepherd, J, Taylor, PL, Terry, SF and Winter, SF. 2012. From patients to partners: Participant-centric initiatives in biomedical research. Nature Reviews Genetics, 13(5): 371–376. DOI: https://doi.org/10.1038/nrg3218
Litva, A, Coast, J, Donovan, J, Eyles, J, Shepherd, M, Tacchi, J, Abelson, J and Morgan, K. 2002. ‘The public is too subjective’: Public involvement at different levels of health-care decision making. Social Science & Medicine, 54(12): 1825–1837. DOI: https://doi.org/10.1016/S0277-9536(01)00151-4
Lost Bird. 2018. The Project; Smartfin. Smartfin. Available at: https://smartfin.org/project/ [Last accessed 3 October 2018].
Magwood, GS, Andrews, JO, Zapka, J, Cox, MJ, Newman, S and Stuart, GW. 2012. Institutionalization of Community Partnerships: The Challenge for Academic Health Centers. Journal of health care for the poor and underserved, 23(4): 1512–1526. DOI: https://doi.org/10.1353/hpu.2012.0161
Mockford, C, Staniszewska, S, Griffiths, F and Herron-Marx, S. 2012. The impact of patient and public involvement on UK NHS health care: A systematic review. International Journal for Quality in Health Care, 24(1): 28–38. DOI: https://doi.org/10.1093/intqhc/mzr066
Moorhead, SA, Hazlett, DE, Harrison, L, Carroll, JK, Irwin, A and Hoving, C. 2013. A New Dimension of Health Care: Systematic Review of the Uses, Benefits, and Limitations of Social Media for Health Communication. Journal of Medical Internet Research, 15(4). DOI: https://doi.org/10.2196/jmir.1933
NIHR. n.d. INVOLVE Supporting public involvement in NHS, public health and social care research. Available at: http://www.invo.org.uk/ [Last accessed 1 June 2018].
Oliver, MB, Dillard, JP, Bae, K and Tamul, DJ. 2012. The Effect of Narrative News Format on Empathy for Stigmatized Groups. Journalism & Mass Communication Quarterly, 89(2): 205–224. DOI: https://doi.org/10.1177/1077699012439020
Pandya-Wood, R, Barron, DS and Elliott, J. 2017. A framework for public involvement at the design stage of NHS health and social care research: Time to develop ethically conscious standards. Research Involvement and Engagement, 3: 6. DOI: https://doi.org/10.1186/s40900-017-0058-y
Patient-Centered Outcomes Research Institute. 2010. Compilation of Patient Protection and Affordable Care Act: Extracted sections concerning patient-centered outcomes research and the authorization of the patient-centered outcomes research institute (PCORI).
Patrick, K, Kebbe, M and Aubin, D. 2018. A home for patient-oriented research. Can Med Assoc. DOI: https://doi.org/10.1503/cmaj.180587
PCORI. 2017. Patient-Centered Outcomes Research Institute – About Us. 21 March 2017. Available at: https://www.pcori.org/about-us [Last accessed 1 June 2018].
Price-Robertson, R and McDonald, M. 2011. Working with Indigenous children, families, and communities: Lessons from practice. Child Family Community Australia, 1 March 2011. Available at: https://aifs.gov.au/cfca/publications/working-indigenous-children-families-and-communities [Last accessed 3 October 2018].
Rae, K, Weatherall, L, Hollebone, K, Apen, K, McLean, M, Blackwell, C, Eades, S, Boulton, J, Lumbers, E and Smith, R. 2013. Developing research in partnership with Aboriginal communities–strategies for improving recruitment and retention. Rural and Remote Health, 13(2): 2255.
Resnik, DB, Elliott, KC and Miller, AK. 2015. A framework for addressing ethical issues in citizen science. Environmental Science & Policy, 54: 475–481. DOI: https://doi.org/10.1016/j.envsci.2015.05.008
Richards, T and Godlee, F. 2014. The BMJ’s own patient journey. BMJ, 348: g3726. DOI: https://doi.org/10.1136/bmj.g3726
Riesch, H and Potter, C. 2014. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public Understanding of Science, 23(1): 107–120. DOI: https://doi.org/10.1177/0963662513497324
Rouleau, G, Bélisle-Pipon, J-C, Birko, S, Karazivan, P, Fernandez, N, Bilodeau, K, Chao, Y-S, de Pokomandy, A, Foley, V, Gagnon, B, Gontijo Guerra, S, Khanji, C, Lamoureux-Lamarche, C, Lebouché, B, Lunghi, C, Menear, M, Riverin, BD and Rodrigue, C. 2018. Early career researchers’ perspectives and roles in patient-oriented research. Research Involvement and Engagement, 4(1): 35. DOI: https://doi.org/10.1186/s40900-018-0117-z
Shippee, ND, Domecq Garces, JP, Prutsky Lopez, GJ, Wang, Z, Elraiyah, TA, Nabhan, M, Brito, JP, Boehmer, K, Hasan, R, Firwana, B, Erwin, PJ, Montori, VM and Murad, MH. 2015. Patient and service user engagement in research: A systematic review and synthesized framework. Health Expectations, 18(5): 1151–1166. DOI: https://doi.org/10.1111/hex.12090
Smith, E and Master, Z. 2017. Best Practice to Order Authors in Multi/Interdisciplinary Health Sciences Research Publications. Accountability in Research, 24(4): 243–267. DOI: https://doi.org/10.1080/08989621.2017.1287567
Society for Participatory Medicine. 2017. Journal of Participatory Medicine. Available at: https://participatorymedicine.org/journal/ [Last accessed 4 June 2018].
SpringerNature. 2018. Patient-Centered Outcomes Research. Available at: https://www.springer.com/adis/journal/40271 [Last accessed 4 June 2018].
Taylor, C, Appiah, KA, Habermas, J, Rockefeller, SC, Walzer, M and Wolf, S. 1994. Multiculturalism: Examining the Politics of Recognition, Gutmann, A (ed.). Expanded Paperback edition. Princeton, N.J.: Princeton University Press.
Van Lange, PAM. 2008. Does empathy trigger only altruistic motivation? How about selflessness or justice? Emotion, 8(6): 766–774. DOI: https://doi.org/10.1037/a0013967
van Uden-Kraan, CF, Drossaert, CHC, Taal, E, Shaw, BR, Seydel, ER and van de Laar, MAFJ. 2008. Empowering Processes and Outcomes of Participation in Online Support Groups for Patients With Breast Cancer, Arthritis, or Fibromyalgia. Qualitative Health Research, 18(3): 405–417. DOI: https://doi.org/10.1177/1049732307313429
Wallerstein, N and Duran, B. 2010. Community-Based Participatory Research Contributions to Intervention Research: The Intersection of Science and Practice to Improve Health Equity. American Journal of Public Health, 100(S1): S40–S46. DOI: https://doi.org/10.2105/AJPH.2009.184036
White, M and Dorman, SM. 2001. Receiving social support online: Implications for health education, Health Education Research, 16(6): 693–707. DOI: https://doi.org/10.1093/her/16.6.693
Wiley Online Library. 2018. Health Expectations. Available at: https://onlinelibrary.wiley.com/journal/13697625 [Last accessed 4 October 2018].
Wilson, P, Mathie, E, Keenan, J, McNeilly, E, Goodman, C, Howe, A, Poland, F, Staniszewska, S, Kendall, S, Munday, D, Cowe, M and Peckham, S. 2015. Research with Patient and Public involvement: A Realist evaluation – the RAPPORT study. Health Services and Delivery Research, 3(38): 1–176. DOI: https://doi.org/10.3310/hsdr03380
Woodahl, EL, Lesko, LJ, Hopkins, S, Robinson, RF, Thummel, KE and Burke, W. 2014. Pharmacogenetic research in partnership with American Indian and Alaska Native communities. Pharmacogenomics, 15(9): 1235–1241. DOI: https://doi.org/10.2217/pgs.14.91