Sage Journals: Discover world-class research

Abstract

Purpose

This study investigated the extent of and factors influencing implementation of a population health approach within sexual health programming in public health.

Method

This sequential multi-phase mixed methods study combined findings from a quantitative survey assessing the extent that a population health approach was implemented in sexual health programs in Ontario public health units and qualitative interviews with sexual health managers and/or supervisors. Interviews explored factors influencing implementation and were analyzed using directed content analysis.

Results

Staff from fifteen of 34 public health units completed surveys and ten interviews were completed with sexual health managers/supervisors. From the 8 Population Health Key Elements Template, 6 elements were moderately implemented and 2 had low implementation. Qualitative findings focused on enablers and barriers to implementing a population health approach in sexual health programs and services and explained most of the quantitative results. However, some of the quantitative findings were not explained by qualitative data (e.g., low implementation of using the principles of social justice).

Conclusion

Qualitative findings revealed factors influencing the implementation of a population health approach. A lack of resources available to health units, differing priorities between health units and community stakeholders, and access to evidence around population-level interventions influenced implementation.

Keywords

Public health sexual health population health mixed methods

Background and purpose

The delivery of sexual health clinical services in Ontario, Canada, through public health includes the provision of direct client care by public health nurses, nurse practitioners, and physicians. This is similar to other jurisdictions, including some in the United States (US) (Ivankovich et al., 2013). Clinic services often include health education, assessment, diagnosis, treatment, and case management for sexually transmitted and blood-borne infections (STBBIs). Given increased prevalence and incidence of gonorrhea, chlamydia, and syphilis across Canada, and in Ontario specifically (Choudhin et al., 2018a, 2018b; Choudhin et al., 2018c; Public Health Ontario, 2019), public health units (PHUs) have had to prioritize the detection and treatment of STBBIs leaving little opportunity to focus more broadly on sexual health promotion.

Public health and population health

In Ontario, local PHUs are responsible for the delivery of public health services (Lyons, 2016). A board of health governs each of the 34 PHUs, with a Medical Officer of Health (MOH), usually a physician with training in public health who is responsible for day-to-day operations (Lyons). Boards of health are governed by the Health Protection and Promotion Act (HPPA; Health Protection and Promotion Act, O. Reg. 135/18, O. Reg. 569, 2018), with Ontario Public Health Standards (OPHS) guiding the programs and services delivered by PHUs. Costs are shared between the Ministry of Health and local municipalities (Lyons). Population-based interventions in the delivery of public health programs and services are now emphasized in the current OPHS (Ministry of Health and Long-Term Care [MOHLTC], 2018). The OPHS and associated MOHLTC protocols guide PHUs on implementing population health approaches in general (e.g., focusing on social determinants of health) but do not provide specific instructions on how to implement at a local level. The OPHS define a population health approach as –“focused on upstream efforts to promote health and prevent diseases to improve the health of populations and the differences in health among and between groups” (MOHLTC, 2018, p. 6). Implementation of a population health approach depends on factors present within local PHUs and depends on direction from senior leadership (e.g., MOH, Board of Health) to implement a population health approach in programs, including sexual health. As well, health unit priority setting and financial resources, such as public health spending and workforce elements (e.g., staffing), impact the extent that a population health approach will be implemented (Ministry of Health, 2019).

Moving to a population health approach is complex and requires efforts in multiple areas, such as engaging with individuals and communities, using evidence to plan programming, and focusing on determinants of health (Jobse et al., 2014; Olson Keller et al., 2004). These activities, and others, are noted in the Population Health Key Elements Template (Health Canada, 2001, https://www.phac-aspc.gc.ca/ph-sp/pdf/overview_handout_black-eng.pdf), which includes eight elements with associated activities that are required to implement a population health approach. This template assists organizations like public health to identify areas of focus to achieve a population-level approach. This template was chosen for this study because of its relevance to the Canadian context and its comprehensive list of defined elements and accompanying details on activities to address them.

Barriers and facilitators to implementing a population health approach in public health

A critical review of the literature from 2007 to 2020, encompassing North America, Europe, and Australia, was conducted to identify barriers and facilitators to implementing a population health approach within public health in any area of programming. The purpose of the review is to identify the current contribution of existing research on this topic, provide an overview of that research, and appraise the quality (Grant & Booth, 2009). Results of the literature review, identified factors influencing implementation that were categorized under four of the eight key elements of a population health approach including Addressing the Determinants of Health, Collaboration across Sectors, Employ Mechanisms for Public Involvement, and Base Decisions in Evidence (see Table 1) and as discussed below.

Table 1.

Factors influencing implementation of a population health approach.

Population Health Framework Key Element	Barriers/ Facilitators	Reference
Addressing the Determinants of Health	FacilitatorsOrganizational Influences Ensuring effective communication to provide clarity on what should be done by public health to address SdoH. Having individuals within public health providing leadership and direction in addressing SdoH. Supportive political environment within each community serviced by public health units. BarriersKnowledge and Understanding Lack of clarity around the reach, application, and legitimacy of this broader approach. Addressing SdoH is a long-term investment that does not produce immediate results. Organizational Influences Philosophy of leaders regarding the benefits of SdoH. Funding and staffing challenges to maintain the momentum necessary to address SdoH. Bureaucratic nature of public health, with tight procedures and policies; slowness in decision making; and being up-focused (i.e., accountable to the board of health).	Brassolotto et al., 2014.
Collaborate Across Sector and Levels	Facilitators Being in close geographic proximity to stakeholders. Having a previous relationship with stakeholders. Barriers Time to develop partnering relationships. Conflicting ideas that can occur between the partners. Staff turnover in partnering organizations resulting in the loss of knowledge about the purpose of the partnership.	Sibbald et al., 2012
Employ Mechanisms for Public Involvement	Facilitators Understanding from public health leaders about the importance of public participation. Building partnerships with organizations to link clients to. Barriers Shifting from individual focus to a population-focused approach. Lack of understanding from politicians, managers, and citizens about public health nurses’ role in facilitating public involvement Increased workload demands conflicts with doing more activities that foster citizen participation.	Aston et al., 2009.
Base Decisions in Evidence	FacilitatorsOrganizations Support of administration. Positive attitudes from managers. Training of staff in the use of evidence in policymaking. Availability and access to research. Access to Evidence Tools and methodologies that allow for applicable evidence. Access to research that is relevant and reliable, along with researchers that have a solid grasp of the policy process. Networking Collaboration between researchers and policy makers, face-to-face interactions between researchers and policymakers. Links between academia and policymaking organizations, and the interests. Awareness of research and the research skills of policymakers. BarriersOrganizations Lack of simultaneity between research and policymaking. Insufficient resources, financial constraints. Existence of research evidence that does not apply to policies being created. Insufficient access to research. Lack of managerial support. Poor dissemination of research. Access to Evidence Not having access to relevant evidence on local problems or policy context, affects stakeholders being able to make decisions. Networking Values of stakeholders. Lack of opportunity to use research evidence being considered a barrier. Differing priorities between researchers and policymakers. Competing pressures (i.e., economic and political) hindered the development of evidence-based policy.	Van de Goor et al., 2017.Oliver et al., 2014.

Addressing the determinants of health and their interaction

Refocusing the activities of public health programs and services from “lifestyle choices” to addressing underlying causes (i.e., social, economic, and political) that create inequalities is important (Brassolotto et al., 2014). Brassolotto et al. (2014) conducted a study in nine PHUs in Ontario to understand supports and barriers that affected the degree of implementation of social determinants of health (SdoH) within each health unit. They utilized qualitative interviews with MOH, Associate Medical Officers of Health (AMOH), and lead staff who directly address SdoH. Barriers were classified into factors that relate to knowledge and understanding of SdoH and factors that stem from the organizational structure of health units.

Collaborate across sectors and levels

Collaboration among health care providers, government, and other sectors that influence health is an important element in a population health approach. Sibbald et al. (2012) conducted a qualitative study using semi-structured interviews with staff in Ontario PHUs to determine how partnerships are initiated and sustained in public health practice. Most staff were public health nurses, with some health promoters, and public health nutritionists/dietitians. Partners involved in program planning were identified as well as facilitators and barriers that affected their collaboration. Three types of partnerships were found: (1) partnerships internal to the PHU (internal to the health unit); (2) partnerships internal to public health (outside the health unit with public health professionals from other health units); and (3) external partnerships (external to both the unit and the public health system). An enabler in these partnerships was having previous relationships with stakeholders and a barrier was the time required to develop them.

Employ mechanisms for public involvement

Public involvement is a vital element that ensures health care being delivered meets the needs of those intended. However, creating an environment of public participation that is successful, poses challenges. Aston and colleagues (2009) conducted a qualitative study that involved interviewing public health nurses (PHNs) in eastern Canada to understand barriers and supports to achieving public participation. Three main barriers were identified that influenced PHNs ability to engage in public participation such as increased workload demands.

Base decisions in evidence

The use of evidence at all stages of policy development is important in a population health approach. Public health needs to invest in research and disseminate findings to assist with policy uptake. Van de Goor et al. (2017) conducted a qualitative study that utilized semi-structured interviews with policymakers (national, regional, and local), researchers, public sector officers, and other influential stakeholders from six European countries. Three facilitators and barriers to evidence-based practice (EBP) in public health policy across countries were found and organized by domains. They were: 1) organizations, systems, and infrastructure; 2) access and availability of relevant evidence; and 3) networking and collaboration between researchers and policymakers.

As well, a systematic review by Oliver and colleagues (2014) addressed facilitators and barriers to using evidence in state, national, and regional policy. They categorized facilitators and barriers into five themes including: 1) organizations and resources; 2) contact and collaboration; 3) research and researcher; 4) policymaker characteristics; and 5) policy characteristics.

None of the research studies identified related specifically to sexual health programming, but rather pertained to public health activities in general. This leaves a gap in understanding the factors influencing implementation of population health activities in sexual health programming. This study applied a mixed methods (MM) approach to fill this knowledge gap – by measuring the extent that population-level activities are implemented in sexual health programming and exploring factors that influence implementation.

Methods and procedures

Study design

A four-phase explanatory sequential MM design with the use of qualitative and quantitative approaches for data collection and analysis was utilized. This approach was chosen to enable findings that are more complete, comprehensive, and provide us with answers to not only “how much” (e.g., degree of implementation) but also an understanding of “why” (factors that influenced implementation). Each phase built on the results from the subsequent phases to answer the question: In what ways do qualitative interviews with managers, supervisors or staff serve to contribute to the understanding of what has been integrated from a population health approach into sexual health within Ontario PHUs? The first two phases consisted of the creation and validation of items for a survey to identify the extent that a population health approach was implemented in sexual health programs and services in Ontario health units, while phase three consisted of administrating a cross-sectional survey. The fourth and final phase involved completing a qualitative descriptive study that involved interviewing sexual health program managers and supervisors to identify and explain individual, program, and structural factors, that influenced the implementation of a population health approach within sexual health. The synthesis of the two sections of the study was completed during the interpretation of the results. Figure 1 in the appendix, presents an overview of the steps taken in this multi-phase sequential MM design.

Figure 1.

Mixed multi-phase sequential design (qual, qual, quan, QUAL).

Qualitative methods

We began our investigation by conducting semi-structured focus groups (n = 6) with public health professionals engaged in sexual health programming to understand what they perceive are activities that reflect a population health approach within sexual health. Based on conversations with these public health professionals we were able to uncover additional information not discovered by the literature review. By the end of this process, we had narrative data that confirmed literature findings and added new activities for a population health approach.

Quantitative methods

Using data derived from the qualitative methods (literature and interviews), we developed an instrument that gave us quantitative measures of the extent that key elements and activities associated with a population health approach were implemented by health units.

Sample and recruitment

To measure the extent to which population health was implemented in each of Ontario's PHUs, we invited one sexual health manager from each health unit to participate. This was an organizational survey where managers were advised to consult with other individuals in their health units (e.g., epidemiologists, public health nurses) to provide a single view. Managers dedicated solely to STBBI case management were excluded because case management for this study was considered part of infectious disease programming and not sexual health programming.

Measures

The on-line survey consisted of 69 items (Frost et al., 2022). Where appropriate, survey items (activities such as collect information to identify priority populations) were listed under headings representing the eight key elements of the Population Health Template (Health Canada, 2001) and were measured on a 5-point Likert scale (1 = never to 5 = always). The survey included items applicable to both sexual health promotion programs and/or sexual health services (e.g., sexual health clinics). We also included items to capture the geographical area of the health unit (e.g., rural, urban), disciplines consulted with while completing the survey, and an open-ended section at the end of the survey for general feedback. Participants were invited by email to complete the on-line survey, with email reminders sent every two weeks for a total of two reminders.

Data analysis

Categorical variables were used to capture the type of population served (e.g., rural or urban). The 5-point Likert scale used to measure the degree that each of the eight key elements was implemented was considered a continuous variable. Descriptive statistics included the mean and standard deviation (Daniel & Cross, 2013).

Qualitative descriptive study

After the quantitative phase, follow-up qualitative interviews were conducted. Methodological principles drawn from fundamental qualitative description (Sandelowski, 2000) to inform decisions related to sampling, data collection, and analysis in this phase. A naturalistic perspective was used by describing the experiences of sexual health managers in implementing a population health approach to understand why their health unit might have had challenges or successes with implementing this approach (Kim et al., 2017; Neergaard et al., 2009; Sandelowski, 2000). Sexual health managers/supervisors were selected through purposive sampling, because we believed they would be best to articulate and explain individual, organizational, and system factors that influence the extent that a population health approach is implemented in the current and proposed amalgamated public health structure. Data was collected through the application of key informant interviews, using a semi-structured interview guide. Flexibility in the use of two frameworks to guide data collection in the different phases of this study was employed, while the use of constant comparison was utilized to provide a descriptive summary of the data that was organized in a way that best fit the data collected (Kim et al., 2017; Neergaard et al., 2009; Sandelowski, 2000).

Sample and recruitment

All individuals were contacted from the pool of 15 out of 34 sexual health managers who completed the online survey to participate in the interview. To be included, participants must have served in a leadership position within public health and/or sexual health, ensuring they could knowledgeably respond to interview questions. If the manager decided not to participate, they were asked to forward the email invitation to staff who would be knowledgeable about the topic. The first author conducted interviews (telephone and secure online platform) that were 45–60 min in length and audio-recorded.

Data collection

The semi-structured interview guide was designed to prompt participants to explain their experience of implementing a population health approach and discuss their perceptions of potential challenges and factors that might enable implementation. It was based on five domains from the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) and included one overarching question for each domain with prompts. The domains included inner setting, outer setting, characteristics of individuals, characteristics of the intervention, and processes. For example, under inner setting, an interview question was “What resources are being provided for the implementation of the population health approach in your program and ongoing operations? Prompts: training sessions; education; evidence from the literature; physical space; time.”

Data analysis

Survey

The 5-point Likert scale was treated as a continuous variable and mean and standard deviation (SD) were used for this measure (Daniel and Cross; Isotalo; XLSTAT, 2017). A two-way frequency table displayed the extent that each of the eight key elements and their associated activities were implemented. Analyses were performed to describe the degree that health units have implemented each of the key elements of a population health approach and the type of population (e.g., rural, urban. mixed) that it serves. The statistical analyses were performed using SPSS20 (IBM, 2011).

Key informant interviews

Interview recordings were transcribed verbatim. NVivo 12 (QSR international Pty Ltd., 2018) was used for data management and coding. Interview data were analyzed inductively and deductively by the first author (Elo & Kyngäs, 2007; Hsieh & Shannon, 2005). Each interview transcript was read individually and then interviews were coded inductively staying close to the language used by participants. Then codes, which captured participants’ perceptions and personal experiences with implementing a population health approach, were organized under concepts from the CFIR framework domains (Damschroder et al., 2009). As analysis proceeded, multiple coding cycles were performed where emergent themes were iteratively generated using constant comparison, reviewed by the research team, and refined until we felt that we had produced a concise set of themes under each domain that reflected the data.

Mixed methods design analysis

Analysis of data occurred in three phases. There was an initial analysis of quantitative results, followed by the analysis of the follow-up qualitative interviews, and then an explanatory analysis using the MM question to see how qualitative data helped explain quantitative results was completed (Creswell & Plano Clark, 2011). Merging of quantitative and qualitative data gave a voice to study participants and offered depth and breadth of understanding of the factors that influenced implementing a population health approach within sexual health (Creswell & Plano Clark; Tariq & Woodman, 2013). A joint display was chosen to merge and present both data sets (Creswell & Plano Clark). This involved using a table (Supplemental file 1) displaying quantitative and qualitative results under each element of the population health template (Health Canada, 2001).

The extent that health units implemented activities associated with that element was displayed as a proportion of the total number of health units responding, and qualitative results were reviewed to identify meaningful comments applicable to each activity in the table. Activities given a score of 4.0 or 5.0 (5 = always) for the extent of implementation by health units were categorized as highly implemented; activities that were given a score of 3.0 showed evidence of a moderate degree of implementation, and activities that were given a score of 1.0 or 2.0 (1 = never) were categorized as low implementation. For the qualitative data, the term “most” was used when discussing common ideas shared by 7 to 10 participants, whereas “some” was used for 3 to 6 participants, and “few” was used to represent ideas shared by 1 to 2 participants.

Results

The survey was completed by managers from 44% of Ontario PHUs (n = 15 serving n = 2 rural, n = 1 urban, n = 12 rural/urban populations). Follow-up in-depth interviews occurred with 12 sexual health leaders from 10 health units who had completed the survey. The geographical representation of these PHUs included Central East, East, Central West, South West, and North West. Of these leaders, 11 were managers and 1 was a supervisor. There were eight participants with 5 years or less experience in management and ten that had 10 years or more years of experience in public health. Results are presented in the form of a narrative weaving approach, which involves presenting quantitative and qualitative findings after each other; findings are organized according to the eight population health elements, highlighting how qualitative results explained the quantitative findings (Fetters et al., 2013; Health Canada, 2001). Activities are presented in italics, while themes associated with the qualitative findings are bolded and indented. Not all activities for each element are presented, due to journal word limits. In the results below, if a single mean and SD is present than it refers to sexual health services only. If two means and SD are presented, the first mean refers to sexual health programs and the second refers to sexual health services. All quantitative results and related qualitative themes are presented for all activities in supplemental file 1.

Population health element A: focus on the health of populations

This element provides guidance that public health actions should be directed to an entire population, or a specific sub-population, rather than individuals (Public Health Agency of Canada [PHAC], 2013). Within this element, ten activities were listed, focusing on collecting and using data, addressing stigma, and working with priority populations. This element showed a moderate degree of implementation by health units for sexual health programs and sexual health services (M = 3.71, SD = 0.66; M = 3.81, SD = 0.54 respectively). Mean scores for the ten activities ranged from 2.73 to 4.33. Qualitative findings supported the quantitative results for six of ten activities.

Theme: assessing barriers and needs of priority populations in the community

Activities related to working with priority populations were highly implemented by health units, (M = 3.53, SD = 0.91). Participants indicated that at their health unit, they collect information to identify priority populations, collaborate with priority populations, and provide individuals within priority populations with targeted health information to make decisions about their sexual health. Some health unit staff explained that they participate in the planning process by collecting data to assess whether their clients are from priority populations. In addition to creating working groups to lead health promotion campaigns to target priority populations and ensure these populations have access to services, one manager further explained the added value of these collaborations in the delivery of programming:

We have created a whole priority populations working group where we have folks who are in those groups who are assigned to specific populations and part of their work is promoting our services and with an end goal of hopefully being able to deliver services if they're needed in the community (Participant 7).

However, a few participants explained that assessing the barriers and needs of priority populations was difficult to accomplish for health units. One manager described their struggles as follows: “It's difficult to engage the population, the high-risk population that you're trying to reach” (Participant 10). However, participants mentioned that their health units were using population health approaches to identify and target interventions to priority populations.

Population health element B: address the determinants of health and their interactions

For this element, addressing determinants of health means acting based on analyses and understandings of the entire range of determinants of health (PHAC, 2013). See the box below for a list of determinants. However, public health focuses on SDoH, where they are able to influence change. Embedded in this element were eight activities related to access to clinic services and characteristics of sexual health programs and services (e.g., using principles of social justice, using trauma and violence-informed care) that focus on social determinants of health. The quantitative results for five of these activities were explained by qualitative results.

Box 1.

Income and social status

Employment and working conditions

Education and literacy

Childhood experiences

Physical environments

Social supports and coping skills

Healthy behaviours

Access to health services

Biology and genetic endowment

Gender

Culture

Race / Racism

Theme: valuing of clinic work over population health

Some participants noted that the activities offering access for youth 24 years of age or under and offering access to adults over the age of 25 was highly implemented, ensuring that both adults (M = 4.60, SD = 0.91) and youth (M = 4.80, SD = 0.77) could access sexual health services. Participants explained that public health was often the only clinic offering STBBI services in their community. Providing testing and treatment for STBBIs remained a priority, regardless of age: “There weren't necessarily age restrictions on the STBBI testing that was sort of a bit more open because we didn't want any barriers” (Participant 4). The activity ensure accessibility, was moderately implemented (M = 3.73, SD = 0.59). One manager mentioned that the geographical area of the health unit determines how easily changes can be made to programming. Specifically, the participant mentioned that in rural areas, health units need to “Look at each sort of county and their needs” (Participant 3) to tailor sexual health services.

Theme: diminishing resources for sexual health

As well, health units highly implemented ensuring quality in programs are offered (M = 3.80, SD = 0.94) but qualitative data did not support this result. Most participants reported that their health unit was understaffed with capacity issues, which prompted sexual health to review clinic services they were providing, as they were no longer able to keep up with the volume of clients. However, in the face of significant resource issues, a few managers were involved in population health changes. These participants mentioned that their health unit was creative in utilizing nursing staff in advanced roles (e.g., nursing project officer) to support sexual health staff and work on program evaluation pieces, such as developing performance-monitoring indicators.

Population health element C: base decisions in evidence

Using a population health approach requires that quantitative and qualitative evidence be used to identify priorities and strategies to improve health and determine the effectiveness of interventions (PHAC, 2013). This element included three activities that focused on using evidence/guidelines for program planning and being involved in research. Qualitative findings supported all of the quantitative results in this element.

Theme: gaining access to data to inform population health

The use of best evidence/guidelines to inform decision-making in practice and/or program development was highly implemented among health units (M = 4.40, SD = 0.63; M = 4.27, SD = 0.70). Most participants brought up that they need evidence of population health interventions to decide what interventions are effective in sexual health. These managers spoke of the value of reviewing evidence to guide their decision making around the allocation of resources to ensure they invest services in the right place.

However, despite the need for evidence to make decisions, the activity get involved in primary research, had low implementation (M = 2.33, SD = 1.11). Participants mentioned that within health departments the value of research was viewed negatively, which influenced their organizations’ participation in research. Some participants consistently cited the importance of support from senior leaders within their organization to participate in primary research. As one manager explained:

I think there is great opportunity there, but I just don't feel that a lot of the health unit buys into the value of research, but I think it's more. I’m not talking about managers and frontline staff; I’m talking more of the MOH (Participant 10).

Reduced funding, capacity, and time affected the ability of sexual health programs and services to participate in research.

Population heath element D: increase upstream investment

Increasing upstream investment means directing investments to areas that have the greatest potential to influence population health status (PHAC, 2013). In this element, ten activities related to the provision of information and education to the public, primary care, and within public health programs on sexual health, and policymaking. Generally, qualitative results supported quantitative findings, but there was variation among health units in the implementation of activities associated with this element. For most activities, about 60% of health units implemented them to a high degree, but qualitative findings did not support this. Rather, qualitative findings supported low to moderate implementation, with only a few to some participants discussing these activities.

Theme: tailoring sexual programming to local needs and populations

One upstream activity - provide information to the public on healthy sexuality and sexual health - was highly implemented in sexual health services but had much more variation in the implementation within sexual health programs. Only a few participants described this activity as something they were doing. However, there was work being done to get messaging out to individuals at high risk for STBBIs, which was highlighted by one participant, “We would be advertising on Grindr and different things and looking at how to reach priority populations and where do they meet, and we’ve got to get to them where they’re at” (Participant 10).

However, an exception within this element was the activity- engage in health public policy to create supportive environments that enable people to lead healthy sexual lives. This activity was implemented to a low extent by health units (M = 2.60, SD = 1.29). Participating in the planning process to implement changes supported the quantitative result since only a few participants indicated that they engaged in this type of activity. These participants stated that their health unit was engaged in policy work through the development of working groups that focused on different areas such as school, community, and health care providers. The goal of this work would be: “influencing implementation of policy changes” (Participant 7).

Population health element E: apply multiple strategies

The health of populations is associated with factors that fall outside the health system or established health sector that requires multiple strategies applied within multiple settings (PHAC, 2013). For this element, there were thirteen activities, many of which were only relevant for sexual health services and not sexual health programs. Activities for this element were related to STBBI screening, testing, and treatment; referrals for victims of violence; provision of multiple intervention components (e.g., primary, secondary and, tertiary care); and community development work involving the public and primary care.

Most of the qualitative results supported the quantitative findings for this element, but there was variation among health units in the implementation of activities within this element. However, when it came to health units employing multiple strategies, such as offering multiple services within sexual health clinics or providing multiple intervention components within sexual health programming, qualitative results helped to explain some of the strategies or limitations that some health units experienced in conducting these activities.

Theme: diminishing resources available to sexual health

Offering multiple services (e.g., sexually transmitted infection testing and naloxone) was highly implemented (M = 4.67, SD = 0.62), but only a few participants mentioned that this activity was happening in their health unit. They noted that being networked with other relevant programs in the health unit was one way that sexual health was able to offer multiple services to clients given the diminishing resources available to sexual health: “Vaccine-Preventable Disease (VPD) and harm reduction, we have worked out a thing where every person who comes in for needle exchange will be offered a flu shot” (Participant 5).

As well, the activity - sexual health programming in your health unit provides multiple components in interventions (e.g., primary, secondary, and/or tertiary) - was moderately implemented (M = 3.27, SD = 1.16). Qualitative findings helped explain some limitations related to this activity. Although some interview participants indicated that their health unit offered multiple components, the modernized standards and limited resources available to sexual health influenced the extent that this activity was happening. Health units pointed out that they engage in multiple strategies like working with school boards to support policy to provide sexual health education, but these strategies were implemented on a much smaller scale than they had been previously. As noted by one participant:

It's just more than we don't just do one strategy, it's reporting a multitude of strategies from case management to working with the school board to supporting policy to putting education out there but it's just so much more in a smaller scale than we used to (Participant 1).

Population health element F: collaborate across sectors and levels

Improving health outcomes is a shared responsibility with “intersectoral collaboration” being the joint action among health and other groups (PHAC, 2013). There were seven activities in this element that focused on intersectoral partnerships with community stakeholders and local/provincial government. There was variation in the implementation of four activities, and qualitative findings helped to explain their low implementation. The activity- work with other local municipal government services (e.g., housing, police, paramedics) to address issues related to sexual health in your community - was not discussed by participants.

Theme: working and collaborating with multiple local and regional partners

Engaging in intersectoral partnerships to address health promotion and prevention was an activity that had variation in implementation among health units (M = 3.13, SD = 0.74; M = 2.87, SD = 0.83). Some participants indicated that they participate in community engagement but that aligning the health department's priority with those of community partners was challenging and influenced collaboration. One manager stated that: “…. they all certainly have their own strategic priorities” (Participant 4). In addition, working with the provincial government to address issues related to sexual health and work with provincial government to provide updates on sexual health issues in your community, were ways that health units were networked with other sexual health programs by holding provincially run meetings to support these programs (M = 2.87, SD = 0.99). However, these meetings were not viewed as a venue for sharing sexual health issues: “I have never really found that folks typically use that as an opportunity for information sharing” (Participant 4). A few participants mentioned that this meeting had changed over the years and is now a combined infectious disease meeting, “95% of it is all around flu and other outbreaks” (Participant 6), leaving little opportunity to discuss sexual health.

Population health element G: employ mechanisms of public involvement

Employing mechanisms of public involvement promotes the participation of all Canadians in developing strategies to improve health and in determining health priorities (PHAC 2013). In this element, four activities looked at involving clients and the community in planning. Qualitative results helped to explain two activities.

Theme: networking with the community

First, involve local community partners in planning showed variation in implementation (M = 2.87, SD = 1.30; M = 2.73, SD = 1.33). Some participants mentioned that sexual health programming is tailored to local needs and population. This is one way they connected with external partners and how they included them in planning. These partnerships ensured that the needs of priority populations like LGBTQ2S living in a rural setting were assessed. However, working and collaborating with community partners presented challenges for health units. Differing priorities between health units and community partners influenced the partnering relationship and was a barrier to developing strategies that everyone could agree on. One manager said:

It is very difficult to bring community partners together, like the school board and some of the youth-serving agencies and try and come up with strategies that everyone can agree on at the table. So, community engagement is a barrier (Participant 4).

Second, involve clients in planning was an activity that had low implementation among health units (M = 2.40, SD = 1.12; M = 2.33, SD = 1.11), with a few participants mentioning that they involve clients in planning using client surveys. These surveys were done to ensure that hours of operation for sexual health clinic services were meeting their population's needs. Capacity within sexual health allowed health units to survey existing clients but did not allow expansion to include other community groups that might not be utilizing sexual health programming:

We just don't have the capacity to be able to get input from clients. We may do a quick little survey for those that are coming to our clinic but that's a very select group, those that are already coming as opposed to the population itself (Participant 2).

Population health element H: demonstrate accountability for health outcomes

Demonstrating accountability for health outcomes requires an increased focus on determining the degree of change that can be attributed to an intervention (PHAC, 2013). There were fourteen activities listed in this element that centered around using and collecting information on a variety of sexual health topics, such as incidence of STBBIs, risk factors for STBBIs, or incidence of sexual violence. However, only two of these activities were explained through qualitative findings.

Theme: providing information and data to the community

Both activities including Report the incidence of STIs of your community to health care providers and make epidemiological data available on your website were moderately implemented among health units [(M = 3.93, SD = 1.03) and (M = 3.80, SD = 1.57; M = 3.67, SD = 1.23) respectively] but only some participants indicated that their health units make data accessible. Those who did share this information used multiple ways (e.g., website, mail) to provide updates to community health care providers on the incidence of STBBIs in their community. Using the health department website was the most common way to convey STBBI information to health professionals in the community: “As most health units do, we post a lot of information on our website” (Participant 3).

Discussion

One characteristic of mixed methods research is the articulation of a MM research question that ultimately drives analysis and integration of quantitative and qualitative data sets. The integrated approach to analysis expanded our understanding of the factors that influenced implementation by explaining: 1) factors that influenced why certain key elements were implemented to a low extent; 2) factors that influenced a high level of implementation of key elements; and 3) why there is variation in implementing key elements across health units. The consistent message from people that participated in the study was that competing priorities like clinical work and the values of senior leadership were influential factors in implementing a population health approach, regardless of whether the health unit was rural or urban.

We found the lowest implemented activities in sexual health programming among health units - collaborate across sectors and levels (element F) and employ mechanisms of public involvement (element G) were reported by others. We know from the literature that multi-organizational partnerships can be impactful in influencing population health but are the most difficult and least prevalent forms of collaboration (Mays & Scutchfield, 2010). As well, excluding public participation by organizations can often be due to a lack of understanding in how to implement this approach (Azizan, 2015). Concepts from these elements are requirements in the OPHS:

“The board of health shall collaborate with health care providers and other relevant partners to ensure access to….clinical services (e.g., sexual health, STI clinics [MOHLTC, 2018, pg. 44]).

However, participants explained that differing agency priorities and a lack of resources influenced developing and maintaining partnerships with community stakeholders and the public. Sibbald et al. (2012) also found that conflicting ideas and priorities between community partners and public health were barriers for health units in maintaining external partnerships. With disinvestment in public health, resources available to health units have decreased over time, which has been felt in programs like sexual health, leaving less time for engagement activities (e.g., community development, advocacy) in favor of individually-focused sexual health clinic services (Guyon et al., 2017). Opportunities for intersectoral collaboration and public participation can be leveraged to implement a population health approach, especially when it comes to underserviced people who are at increased risk for health inequities. This allows for the pooling of resources and building bridges between different actors to ensure programming is targeted appropriately to reduce health inequity.

Second, over half of the activities listed under the highly implemented elements - Focus on the health of populations (element A) and Address the determinants of health and their interaction (element B) were supported by other researchers. Literature indicates that leaders in Canadian health care recognize the need to focus on social determinants and population health approach to relieve disparity and improve health equity (Andermann & CLEAR Collaboration, 2016; Henderson & King, 2018). Qualitative findings indicated that engaging priority populations was a difficult task for health units because of limited resources, although participants discussed some work being done to reach priority populations (e.g., health promotion campaigns). Public health interventions that utilize community partnerships have been found to be more successful in reaching these populations for STBBIs (O’Mara-Eves et al., 2015). Unfortunately, low implementation of activities related to building community partnerships [such as engage in intersectoral partnerships, work with community stakeholders (e.g., teachers, community leaders) to assist them with identifying their attitudes, beliefs, and values related to sexual health] was a barrier to reaching priority populations. Health units need to devote more resources and develop better strategies for engaging priority populations in planning to ensure sexual health programming is meeting their needs. To increase engagement, strategies such as crowdsourcing can be utilized for difficult-to-reach populations (Tang et al., 2019).

Crowdsourcing has the advantage of bringing non-experts and experts together in a bottom-up approach to share ideas and develop solutions to problems within their communities (Tang et al.). This approach can assist with development of sexual health interventions to improve the uptake of STI testing (Tang et al.). Engaging members of the community in public health interventions is a way to reduce health inequities by ensuring that programs and services offered are appropriate, accessible, and utilized by end-users (Moloughney, 2012, & O’Mara-Eves et al., 2015). For the element - Address the determinants of health -some activities had variation in implementation, and some had no qualitative data to explain the results such as use trauma and violence-informed care, use principles of social justice to address the disadvantage of certain groups, and provide supports to address broader social needs. Although these activities are important in public health practice, they are not referenced in the OPHS. Barriers to implementing population health activities such as limited resources, lack of sexual health-specific evidence on population-level interventions, or challenges engaging some populations related to being a government agency may explain this finding. As well, some activities may be too abstract and difficult to operationalize. Social justice principles are seldom accompanied by an explanation of what is required for implementation, making it challenging for health units to apply (Buyx et al., 2016). In addition, nurses working in public health have a better understanding of individual approaches than population-level ones, highlighting a gap requiring further exploration (Mabhala, 2015).

Third, qualitative data helped explain the elements- base decisions in evidence (element C) and increase upstream investment (element D) which had variation in the extent that they were implemented across health units. Regarding base decisions in evidence, qualitative findings highlighted that health units need evidence of effective population health interventions for sexual health to assist with making decisions on where investments need to be made. However, within health units, the value of research from upper leadership and reduced funding influenced a health unit's ability to implement this activity. Other studies have found that there are several barriers to the adoption and implementation of sexual health EBP. Lack of knowledge about where to find evudence, lack of guidance regarding the adoption process, and lack of specialized training on sexual health were some of the cited reasons (Peskin et al., 2017). In addition, competing demands, inadequate funding, and the absence of organizational support were noted to be barriers for health departments in the utilization of evidence-based decision-making (Brownson et al., 2012). There is evidence to suggest that public health needs a greater focus on evidence-informed decision-making (EIDM) but requires resource allocation (e.g., workforce, financial) and an organizational culture that supports this type of investment (Brownson et al., 2012). As well, developing a workforce that has skills to support EIDM and the formation of partnerships with external organizations (e.g., universities), can assist with the development and use of research that is relevant to public health (Brownson et al.). For the element - increase upstream investment, quantitative findings demonstrated that policy work was not highly implemented among health units. However, qualitative findings provided insight into the fact that a few health units engaged in policy work with schools and the community but the exact work being done was not identified. Creating environments that promote health and prevent disease is the goal of policy work, with the core of this work centered on engaging with decision-makers and transferring evidence to them to influence public policy (Moloughney, 2012). However, using an evidence-informed approach is a critical way that health units can guide policy change but as described above, a lack of value in research by senior leaders hinders this process and influences policy work done within sexual health. It is important that policy supports access to complete and accurate sexual health information, which government-funded health care services have an obligation to provide to ensure informed decision making (Santelli et al., 2017).

Strengths and limitations

For qualitative and quantitative data, utilizing the experiences of sexual health nursing managers and supervisors within PHUs in Ontario, Canada, was a strength and limitation. First, managers/supervisors are in a unique position to offer insight into what activities are occurring within their health unit as they have knowledge of front-line work and senior-level decisions. However, this means that only one group of nurses were captured at a specific point and did not consider the experiences of front-line nurses who deliver sexual health programs and services, or senior leadership who are responsible for program decision making. As well, data collection,. occurred only one year after the modernized standards were released. However, this is not viewed as a limitation given that population health is not a new concept for public health in Ontario and has been part of the OPHS previously (MOHLTC, 2014).

Different frameworks supported the quantitative and qualitative studies. CFIR (Damschroder et al., 2009) was limited since community participation/engagement is a key activity embedded within public health, the OPHS, and a population health approach (Moloughney, 2012) but is missing as a concept from CFIR. Each framework was useful for the individual studies, but integrating results was challenging and MM researchers need to acknowledge that combining different data types does not always integrate easily. The focus should be on whether our understanding has been enriched by the combination of different types of data (Uprichard & Dawney). In the future, using one framework would make merging results easier, however, the framework/s must be relevant to the research questions being studied.

Conclusion

Overall, this study contributes insights into population health activities on which health units should focus and assists sexual health managers, senior management, and policymakers with a better understanding of where resources, both financial and human need to be invested. We found that broader upstream activities like social justice and engaging in healthy public policy were not highly implemented by health units with little to no qualitative findings to explain why. This inconsistency points to important areas for future research. Given that social justice is a foundational component of community health, it would be important to understand barriers to implementing interventions related to health equity.

Supplemental Material

sj-docx-1-cjn-10.1177_08445621221143019 - Supplemental material for A Multiphase Mixed Methods Study on the Integration of a Population Health Approach in Sexual Health Programs and Services in Ontario Public Health Units

Supplemental material, sj-docx-1-cjn-10.1177_08445621221143019 for A Multiphase Mixed Methods Study on the Integration of a Population Health Approach in Sexual Health Programs and Services in Ontario Public Health Units by Linda Frost, Ruta Valaitis, Susan M. Jack, Michelle Butt and Noori Akhtar-Danesh in Canadian Journal of Nursing Research

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical approval

Verbal consent was acquired, and confidentiality was assured at all stages of the research. Ethics approval was obtained from Hamilton Integrated Research Ethics Board (HiREB #5692) for this study.

ORCID iD

Linda Brousseau

Supplemental material

Supplemental material for this article is available online.

Author Biography

Linda Frost, NP, MN, PhD, School of Nursing, McMaster University, Hamilton, Ontario.

References

Andermann

, & CLEAR Collaboration. (2016). Taking action on the social determinants of health in clinical practice: A framework for health professionals. CMAJ: Canadian Medical Association Journal, 188, E474–E483. https://doi.org/10.1503/cmaj.160177

Aston

Meagher-Stewart

Edwards

Young

L. M.

(2009). Public health nurses’ primary health care practice: Strategies for fostering citizen participation. Journal of Community Health Nursing, 26, 24–34. https://doi.org/10.1080/07370010802605762

Azizan, M. (2015). Challenges in the public participation and the decision making process. Sociologija i prostor/Sociology & Space, 53(1). https://doi.org/10.5673/sip.53.1.2

Brassolotto

Raphael

Baldeo

(2014). Epistemological barriers to addressing the social determinants of health among public health professionals in Ontario, Canada: A qualitative inquiry. Critical Public Health, 24(3), 321–336. https://doi.org/10.1080/09581596.2013.820256

Brownson

R. C.

Allen

Duggan

Stamatakis

K. A.

Erwin

P. C.

(2012). Fostering more-effective public health by identifying administrative evidence-based practices: A review of the literature. American Journal of Preventative Medicine, 43(3), 309–319. https://doi.org/10.1016/j.amepre.2012.06.006

Buyx

Killar

Laukötter

(2016). Sridhar Venkatapuram's health justice: A collection of critical essays and a response from the author. Bioethics, 30(1), 2–4. https://doi.org/10.1111/bioe.12220

Chadwick

R. C.

McGregor

Sneath

Rempel

Li Qun He

Brown

Seifred

McAuley

Kamatovic

R. J.

Al-Husari

Ahmed

Bertolo

Munkley

Luterman

(2018). STI Initiative: Improving testing for sexually transmitted infections in women. BMJ Open Quality, 7, 1–7. https://doi.org/10.1136/bmjoq-2018-000461

Choudhin

Miller

Sandhu

Leon

Aho

(2018a). Chlamydia in Canada, 2010–2015. Communicable Disease Report, 44(2), 49–54. https://doi.org/10.14745/ccdr.v44i02a03

Choudhin

Miller

Sandhu

Leon

Aho

(2018b). Gonorrhea in Canada, 2010–2015. Communicable Disease Report, 44(2), 49–54. https://doi.org/10.14745/ccdr.v44i02a01

10.

Choudhin

Miller

Sandhu

Leon

Aho

(2018c). Infectious and congenital syphilis in Canada, 2010–2015. Communicable Disease Report, 44(2), 49–54. https://doi.org/10.14745/ccdr.v44i02a02

11.

Cliente

Brownson

R. C.

Umble

Campbell Erwin

Summers

(2012). Information-seeking behaviors and other factors contributing to successful implementation of evidence-based practices in local health department. Journal of Public Health Management and Practice, 18(6), 571–576. https://doi.org/10.1097/PHH.0b013e31825ce8e2

12.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Sage Publications.

13.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(50), 1–15. https://doi.org/10.1186/1748-5908-4-50

14.

Daniel

W. W.

Cross

C. L.

(2013). Biostatistics. A foundation for analysis in the health sciences (10th ed.). John Wiley & Sons.

15.

Elo

Kyngäs

(2007). The qualitative content analysis process. Journal of Advanced Nursing, 22, 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x

16.

Fetters

M. D.

Curry

L. A.

Creswell

J. W.

(2013). Achieving integration in mixed methods designs-principles and practices. Health Services Research, 48(6 Pt 2), 2134–2156. https://doi.org/10.1111/1475-6773.12117

17.

Frost

Valaitis

Butt

Akhtar-Danesh

Jack

S. M.

(2022). Preliminary instrument development to measure implementing a population health approach in sexual health. Public Health Nursing. https://doi.org/10.1111/phn.13106

18.

Grant

M. J.

Booth

(2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26, 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

19.

Guyon, A. I., Hancock, T., Kirk, M., MacDonald, M., Neudorf, C., Sutcliffe, P., Talbot, J., & Watson-Creed, G. (2017). The weakening of public health: A threat to population health and health care system sustainability/L'affaiblissement de la santé publique: une menace pour la santé des populations et la viabilité du système de soins de santé. Canadian Journal of Public Health/Revue Canadienne de Santé Publique, 108(1), e1–e6. https://doi.org/10.17269/cjph.108.6143

20.

Health Canada. (2001). The Population Health Template: Key elements and actions that define a population health approach . https://www.phac-aspc.gc.ca/ph-sp/pdf/discussion-eng.pdf

21.

Henderson

A. B.

King

(2018). Social determinants and teen pregnancy prevention: Exploring the role of nontraditional partnerships. Health Promotion Practice, 19(1), 23–30. https://doi.org/10.1177/1524839916680797

22.

Hsieh

H. F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. https://doi.org/10.1177%2F1049732305276687

23.

IBM Corp. Released (2011). IBM SPSS statistics for windows, Version 20.0. IBM Corp.

24.

Ivankovich

M. B.

Fenton

K. A.

Douglas

J. M.

Jr. (2013). Considerations for national public health leadership in advancing sexual health. Public Health Reports, 128(Suppl 1), 102–110. https://doi.org/10.1177/00333549131282S112

25.

Izenberg

(2019). Public health in Ontario: What it does, and what might change . https://healthydebate.ca/2019/05/topic/public-health-in-ontario

26.

Jobse

B. N.

Adams

Levy

(2014). Population health management a Canadian perspective on the future of health systems. Canadian Journal of Physician Leadership, 4(1), 8–15.

27.

Kim

Sefcik

J. S.

Bradway

(2017). Characteristics of qualitative descriptive studies: A systematic review. Research in Nursing & Health, 40(1), 23–42. https://doi.org/10.1002/nur.21768

28.

Kirk

M. A.

Kelley

Birken

S. A.

Abadie

Damschroder

(2016). A systematic review of the use of the Consolidated Framework for Implementation Research. Implementation Science, 11(72). https://doi.org/10.1186/s13012-016-0437-z

29.

Lyons, J. (2016). The Independence of Ontario's public health units: Does governing structure matter? Healthcare Policy, 12(1), 71–83.

30.

Mabhala

(2015). Public health nurse educators’ conceptualisation of public health as a strategy to reduce health inequities: A qualitative study. International Journal of Health Equity, 14(14). https://doi.org/10.1186/s12939-015-0146-2

31.

Marzuki

(2015). Challenges in the public participation and the decision making process. Sociologija I Prostor, 53, 21–39. http://dx.https://doi.org/10.5673/sip.53.1.2

32.

Mays

G. P.

Scutchfield

F. D.

(2010). Improving public health system performance through multiorganizational partnerships. Preventing Chronic Disease, 7(6), A116.

33.

Ministry of Health. (2019). Discussion paper: Public health modernization . file:///M:/PhD%20Study/Qualitative%20paper/dp_public_health_modernization.pdf

34.

Ministry of Health and Long-Term Care. (2014). Ontario public health standards . https://neltoolkit.rnao.ca/sites/default/files/Ontario_Public_Health_Standards_2008%20.pdf

35.

Ministry of Health and Long-Term Care. (2018). Ontario public health standards: requirements for programs, services, and accountability . http://www.health.gov.on.ca/en/pro/programs/publichealth/oph_standards/docs/protocols_guidelines/Ontario_Public_Health_Standards_2018_en.pdf

36.

Moloughney

(2012). Community engagement as a public health approach: A targeted Literature review. Peel Public Health . https://www.peelregion.ca/health/library/pdf/Community_Engagement.pdf

37.

Neergaard, M. A., Olesen, F., Andersen, R. S., & Sondergaard, J. (2009). Qualitative description–the poor cousin of health research? BMC Medical Research Methodology, 9(1), 1–5. https://doi.org/10.1186/1471-2288-9-52

38.

Olson Keller

Strohschein

Lia-Hoagberg

Schaffer

M. A.

(2004). Population-based public health interventions: Practice-based and evidence supported. Part I. Public Health Nursing, 21(5), 453–468. https://doi.org/10.1111/j.0737-1209.2004.21509.x

39.

O’Mara-Eves

Brunton

Oliver

Kavanagh

Jamal

Thomas

(2015). The effectiveness of community engagement in public health interventions for disadvantaged groups: A meta-analysis. BMC Public Health, 15(129). https://doi.org/10.1186/s12889-015-1352-y

40.

Peskin

M. F.

Hernandez

B. F.

Gabay

E. K.

Cuccaro

D. H.

Ratliff

Reed-Hirsch

Rivera

Johnson-Baker

Emery

S. T.

Shegog

(2017). Using intervention mapping for program design and production of iCHAMPSS: An online decision support system to increase adoption, implementation, and maintenance of evidence-based sexual health programs. Frontiers in Public Health, 5, 203. https://doi.org/10.3389/fpubh.2017.00203

41.

Public Health Agency of Canada. (2013). Population health approach: The organizing framework . https://cbpp-pcpe.phac-aspc.gc.ca/population-health-approach-organizing-framework/

42.

Public Health Ontario. (2019). Monthly infectious diseases surveillance reports . https://www.publichealthontario.ca/-/media/Documents/Surveillance-Reports/Infectious/2019/surveillance-report-infectious-diseases-2019.pdf?sc_lang=en

43.

QSR International Pty Ltd (2018). NVivo qualitative data analysis Software. Version 12 2018.

44.

Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nursing & Health, 23(4), 334–340. https://doi.org/10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g

45.

Santelli

J. S.

Kantor

L. M.

Grilo

S. A.

Speizer

I. S.

Lindberg

L. D.

Heitel

Schalet

A. T.

Lyon

M. E.

Mason-Jones

A. J.

McGovern

Heck

C. J.

Rogers

Ott

M. A.

(2017). Abstinence-only-until-marriage: An updated review of US policies and programs and their impact. Journal of Adolescent Health, 61(3), 273–280. https://doi.org/10.1016/j.jadohealth.2017.05.031

46.

Sibbald

Kothari

Rudman

Dobbins

Rouse

Edwards

Gore

(2012). Partnerships in public health: Lessons from knowledge translation and program planning. Canadian Journal of Nursing Research, 44(1), 94–119.

47.

Tang

Ritchwood

T. D.

Ong

J. J.

Wei

Iwelunmor

Tucker

J. D.

(2019). Crowdsourcing to improve HIV and sexual health outcomes: A scoping review. Current HIV/AIDS Reports, 16(4), 270–278. https://doi.org/10.1007/s11904-019-00448-3

48.

Tariq

Woodman

(2013). Using mixed methods in health research. JRSM Short Reports, 4(6), 2042533313479197. https://doi.org/10.1177%2F2042533313479197

49.

Uprichard

Dawney

(2019). Data diffraction: Challenging data integration in mixed methods research. Journal of Mixed Methods Research, 13(1), 19–32. https://doi.org/10.1177%2F1558689816674650

50.

Van de Goor

Hamalainen

R. M.

Syed

Juel Lau

Sandu

Spitters

Eklund Karlsson

Dulf

Valente

Castellani

Aro

A. R.

(2017). Determinants of evidence use in public health policy making: Results from a study across six EU countries. Health Policy, 121(3), 273–281. https://doi.org/10.1016/j.healthpol.2017.01.003/

51.

XLSTAT. (2017). Which descriptive statistics tool should you choose? https://help.xlstat.com/customer/en/portal/articles/2800352-which-descriptive-statisticstool-should-you-choose-?bid=9283

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB