A mixed-methods process evaluation was used to evaluate the implementation of the initiative over its first year (May 2019 to May 2020). The evaluation was designed in line with the Medical Research Council guidance on process evaluation for complex interventions , and reported according to the Standards for Reporting Implementation Studies (StaRI) Statement . Evaluation questions were formulated to address fundamental process evaluation metrics (according to Saunders et al. ): Aim 1 addressed dose delivered and dose received (including satisfaction), Aim 2 addressed fidelity, and Aim 3 addressed context (reach and recruitment were not formally assessed) (Fig. S1). As the focus was on the process of implementation, no effectiveness measures were planned. When complex interventions are implemented in new contexts, it is expected that some elements of the intervention will need to be tailored over time . The evaluation was therefore intended to be flexible and to capture and facilitate any changes . The evaluation design and all iterative changes were discussed, reviewed with, and approved by the stakeholders as the implementation progressed.
The intervention was developed in partnership between the YWCA Elm Centre and Women’s College Hospital (WCH). The YWCA Elm Centre is a not-for-profit housing complex with 300 mixed housing units for women and gender diverse people and their children (85 supportive housing units for individuals with complex mental health needs, 50 Indigenous-specific units, and 165 affordable rental units). Existing Elm Centre services were provided by (1) the YWCA, (2) the Jean Tweed Centre, and (3) WCH. The YWCA provides housing supports (eviction prevention), community engagement, and individualized mental health and substance use supports as needed for tenants of all 300 units. The Jean Tweed Centre, a community based substance use and mental health agency, provides case management and some nurse practitioner-led primary care to tenants of the supportive housing and Indigenous-specific units. WCH, a nearby academic hospital, had formed a primary care partnership with the Elm Centre in 2015 to support tenants of supportive housing and Indigenous-specific units (50). Initially, the WCH family physician provided on-site primary care, however by 2018 she had successfully connected most tenants with off-site primary care providers, so the role had transitioned to one of system navigation, liaison, coordination, and advocacy. Some tenants had access to off-site psychiatric services (e.g. assertive community treatment), but most tenants did not have ongoing psychiatric care. An on-site psychiatrist from WCH previously provided direct consultation and follow-up onsite at the Elm Centre, but uptake and integration were poor, and there was no formal collaboration between the psychiatrist and housing, case management, and primary care providers. In 2018, WCH and YWCA-Elm decided to pursue a new model and WCH committed to 3 years of funding for psychiatric indirect care (CA$25,000 per year for a half-day per week of indirect care), program development (e.g. purchase of reference materials for use with staff and tenants), and evaluation.
Targeted sites and participants
A stakeholder group consisting of managers, nurse practitioners, and physicians from YWCA-Elm, the Jean Tweed Centre, and WCH participated in designing and implementing the initiative. Stakeholders came from diverse professional and personal backgrounds. YWCA-Elm and Jean Tweed Centre staff who worked with tenants (e.g. case managers, community engagement workers) participated in the intervention. Tenants in supportive housing and Indigenous-specific units were the focus of the intervention, however some aspects of the initiative (e.g. psychoeducation sessions) were open to all tenants.
The intervention was designed by stakeholders from the three partner organizations, and desired outcomes included meeting tenants’ mental health care needs, enhanced safety, trauma-informed and culturally-safe care, better sense of agency and support among staff, and sustainability in care (see logic model, Fig. S2). Since most individuals had external primary care providers, a modified shifted collaborative care model  was envisioned in which the psychiatrist would collaborate directly with Elm Centre housing/case management staff in addition to on-site primary care supports. The plan was to iteratively adapt the intervention during the course of the project. The program activities fell into three categories (1) multidisciplinary support for tenants, (2) tenant engagement, and (3) building staff capacity (Fig. 1). Within (1) multidisciplinary support, a rostering system was used to identify tenants who (a) lived in supportive housing or Indigenous-specific units, (b) required enhanced mental health supports, and (c) consented to having YWCA, the Jean Tweed Centre, and WCH collaborate around their care. Rostered tenants were those who met all three criteria. Of note, acceptance of medication was not a condition to being rostered. Initially, the plan was for rostered tenants to be assigned a “mini-team”, consisting of a YWCA community engagement worker and a Jean Tweed Centre case manager. The psychiatrist supported rostered tenants via a mixture of indirect and direct care (Fig. 1) and supporting external referrals to intensive services when necessary (e.g. assertive community treatment ). Psychiatric care included recommendations for medications and psychotherapy, crisis supports, as well as advocacy interventions (e.g. supporting refugee processes). Depending on the circumstance, within direct care tenants could meet one-on-one with the psychiatrist, or have a joint meeting including YWCA and/or Jean Tweed staff. The team recognized that some tenants would not need formal rostering but would still benefit from support, or would decline formal rostering due to a preference to keep housing supports and mental health supports separate, stigma around mental illness, or previous adverse experiences with mental health providers. Staff could still solicit support for these “non-rostered” tenants (e.g. ask deidentified questions, ad hoc supports). Within (2) tenant engagement, the main planned activity was group-based psychoeducation open to all tenants in the building. The WCH family physician had previously successfully engaged tenants in group-based health education so this format was used for psychoeducation. Within (3) staff capacity-building, training sessions for staff were led by the psychiatrist alone or co-led with the WCH family physician, with the plan for topics to be chosen collaboratively with staff. Informal staff capacity building was also anticipated through participating in multidisciplinary tenant support and tenant engagement.
It was a priority that the initiative be flexible and attuned to reflect the local context, including the large proportion of Indigenous tenants . Both Indigenous and non-Indigenous tenants had experienced high rates of trauma, discrimination, and substance use. Trauma-informed [14, 15], culturally safe , and harm-reduction lenses  were expected to enhance the patient-centeredness of the program, and the shared lenses were expected to facilitate collaboration amongst team members. Trauma-informed care was operationalized through having a psychiatrist specialized in trauma, trauma-informed care plans, and education about trauma integrated into both psychoeducation for tenants and training sessions for Elm Centre staff [14, 15]. Culturally safe care was operationalized by reciprocal education and knowledge-sharing with staff (including Indigenous staff), considering cultural factors within care plans, maintaining an awareness of sociopolitical factors (e.g. history of colonization), and considering referral to culturally-specific services as appropriate . Principles of harm reduction were incorporated by meeting clients where they were in their recovery journey, and considering concurrent management of addictions and psychiatric illness within care plans in line with tenant goals (including referral to outside harm reduction-informed services as needed) .
The implementation strategies were multi-pronged, and were identified in accordance with the Expert Recommendations for Implementing Change (ERIC) [23, 24]. We formed an academic partnership, identified and prepared stakeholder champions from all three organizations, and held regular implementation team meetings with identified stakeholders. New funding was accessed for intervention components not covered by existing streams (e.g. activities not billable under provincial health insurance). The intervention and evaluation were designed to be adaptable and tailored to the housing context, and the services were delivered on-site. We used iterative evaluation strategies, specifically purposefully re-examining the implementation (monitoring progress and adjusting practices to improve the intervention), and identifying barriers and facilitators. We conducted educational meetings with the staff prior to implementation, initially created new teams (mini-teams), and created a collaborative learning environment to facilitate implementation among staff. Although we were not able to change the record systems to allow for shared documentation due to privacy/consent and individual organization requirements, we did streamline processes for tenants to provide consent for information-sharing across organizations.
Evaluation data sources
Data sources were designed to be pragmatic, and, whenever possible, use data already collected by the program. Data sources were (a) program documents consisting of stakeholder meeting minutes (~monthly), weekly psychiatrist documentation (detailed de-identified report on all activities, e.g. number of case conferences) and a de-identified care plan log for rostered tenants (components of care plans including medication management, psychosocial support), (b) staff surveys distributed at 3 and 9 months, (c) focus groups with staff and stakeholders at 6 and 12 months (semi-structured), (d) an all-tenant survey delivered at 12 months, and (e) tenant surveys following psychoeducation sessions. Interviews with tenants were planned, but not completed due to a lack of uptake from tenants. Surveys for external primary care providers were initially planned, but were deferred as the initial focus was on internal collaborations.
For Aim 1, the delivery of activities was assessed quantitatively using weekly psychiatrist documentation (number of each program activity delivered each week) and stakeholder meeting minutes (planned changes to activities) (Fig. S1). Staff, tenant, and stakeholder perspectives of activities were assessed quantitatively using staff surveys (Likert 1-5 satisfaction rating) and tenant post-psychoeducation session surveys (Likert ratings of questions related to group utility, feeling respected, and satisfaction with groups, Likert 1-5) and qualitatively using staff and stakeholder focus groups (questions on experiences and opinions of activities) and open-ended feedback on the all-tenant survey. For Aim 2, to assess consistency with team-based patient-centered care, we (a) documented the frequency of team-based activities (e.g. case conferences, capacity-building, co-led tenant groups) and non-team-based activities (e.g. direct consultation with tenants) using weekly psychiatrist documentation, (b) documented the providers involved in care plans using the log of rostered tenants, (c) solicited quantitative ratings of collaboration, engagement, tenant-centeredness, and consistency with each of the shared lenses (trauma-informed, culturally safe, harm reduction) on staff surveys (each Likert scale 1-5), (d) asked questions related to collaboration, tenant-centeredness, and shared lenses during staff/stakeholder focus groups (qualitative), and (e) solicited tenant perspective using open-ended questions on the all-tenant survey. For Aim 3, we used qualitative data from stakeholder meeting minutes, staff/stakeholder focus group data, and free-text tenant survey responses to assess barriers and facilitators to implementation. To capture additional contextual information and information about any changes to the initiative, we used qualitative data from stakeholder meeting minutes (main data source) as well as information from psychiatrist documentation and staff/stakeholder focus group data. Outcomes of the intervention (e.g. change in tenant mental health) and economic evaluation were outside the scope of the study.
Sample size and recruitment
The evaluation was designed to ensure that tenants, staff, and stakeholders felt comfortable sharing information with the evaluation team (e.g. managers not having access to staff feedback, ensuring that tenants knew the evaluation was not linked to housing status). Staff were invited to participate in surveys via email; the goal was for all staff (up to 14 individuals) to participate. Stakeholders and staff were invited to participate in focus groups via email; the aim was to conduct one stakeholder and one to two staff focus groups with 4-10 participants per group at each time point. Rostered tenants were to be invited to participate in interviews by their case managers/community engagement workers. Following psychoeducation groups, post-group surveys were distributed on paper to all interested participants. An all-tenant survey was distributed on paper to 135 supportive housing and Indigenous-specific units under their door (or directly by staff in certain circumstances). Draws for gift card prizes (CA$25) were done for each of the staff surveys and the all-tenant survey.
Analysis was done throughout the evaluation to iteratively refine the implementation process. Quantitative data were analyzed using descriptive statistics in Excel. Likert-type responses were described using medians and interquartile ranges (IQR). Qualitative data were analyzed using thematic analysis, using the framework explication by Braun and Clarke : (i) documents/transcripts were read and re-read for a broad understanding, (ii) documents/transcripts were examined closely and initial codes inserted, (iii) codes were grouped into potential themes, (iv) themes were reviewed and mapped conceptually, and (v) refined and grouped into themes/sub-themes. The aim of the analysis was to deductively identify themes related to participants’ qualitative experiences of the program and provide further contextual information. Documents and transcripts were coded by two team members (LB/AB), and discrepancies were discussed to reach consensus. Qualitative data was analyzed using NVIVO software. The two methods were then triangulated; quantitative and qualitative data were interpreted together to maximize the information and perspectives available for each study aim. No subgroup analyses were planned.
Ethics approval was obtained through Women’s College Hospital Ethics Assessment Process for Quality Improvement Projects (WCH APQIP). All methods were conducted in accordance with relevant guidelines and regulations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.