On the same page: Co-designing the logic model of a telehealth service for children in rural and remote Australia

Abstract

The value of programme logic models as a tool for planning, evaluation, and communication is well recognised. However, the value of its development process is less discussed. In this paper, we describe how we used a combination of literature review and organisational stakeholder consultations to develop a logic model for a telehealth programme for children in rural and remote Australia. Our aim was to use this process to further embed the programme within its implementing organisation, and by so doing to promote its sustainability and scale-up; a major challenge of telehealth programmes, especially those involving reorganisation of processes. Our efforts to describe the components of this complex intervention on the one-page logic model allowed for debates and discussions within the implementing organisation which then facilitated an improved cross-organisational understanding of the telehealth programme; a real time face-to-face (video-link) service which requires the reorganisation of existing service delivery platforms. The process helped to embed the telehealth programme within existing services. We conclude that stakeholder engagement in developing logic models can transform them from being only a tool that provides the picture of why and how a programme works, to one that plays a role in embedding programmes within implementing organisations.

Keywords

Telehealth Telecare embed sustain scale logic model process

Introduction

Telehealth is a potentially powerful strategy for addressing health system challenges, for example, health workforce shortages and inequities in access to health services in remote and rural communities. But despite great expectations of telehealth as part of the digital health revolution, it is still not considered a routine option for service delivery, even in high-income countries. The experience globally is that telehealth projects begin as pilots which are rarely embedded to the point of sustainability and scale-up, due, in part, to the many challenges involved in re-organising service delivery processes within health care organisations. So far, there has been a wider adoption of the ‘store and forward’ provider-to-provider interaction mode of telehealth (as is often used in tele-radiology and tele-dermatology), likely because this involves less significant re-organisation of processes. But the adoption of the real-time patient–provider interaction mode of telehealth has been more limited. This mode often requires more resources and commitment to set up the new patient–provider interface, ensure that processes and equipment are well coordinated on the side of the provider and of the patient in real time,¹ and address issues that may limit acceptability among providers, patients or third-party payers.²

Insights to facilitate the adoption, scale-up and sustainability of real time patient–provider telehealth are scant in the literature.² This paper reports the process of co-designing a programme logic model³ for a real-time telehealth programme to reach children with developmental, behavioural or psychological care needs in rural and remote Australia with inadequate access to these health care services. We posit that the process itself – for example, of co-designing a programme logic model for telehealth – can assist with embedding telehealth into ‘systems as usual’ within an existing health service, thus increasing the likelihood of sustainability. Telehealth offers a real solution for tackling developmental, behavioural and mental health concerns as early as possible for rural and remote families. In Australia, 22% of children are developmentally vulnerable in at least one domain; a proportion that rises to 33% in the most socio-economically disadvantaged communities, and even higher, at 47%, in the most remote communities.⁴ This higher prevalence of developmental vulnerabilities for children who live in rural areas also exists in the United States.⁵ This high need for health services in rural areas is unfortunately met with a lack of health professionals to treat them. Given the benefits of early intervention for many childhood conditions,^6–11 and the significant long-term costs of not intervening, having access to real-time patient–provider mode of telehealth to address developmental vulnerabilities for children in rural and remote communities is imperative.

The Royal Far West national ‘Telecare for Kids’ programme

Royal Far West (RFW) is a non-profit organisation that provides health services to children living in rural and remote Australia. Founded in 1924, RFW uses an integrated health, education and social model of care to reach children with mental, behavioural and developmental disorders with limited access to local services, and delivers services to these children through three channels: 1) in-person care for children at RFW’s centre in Sydney; 2) in community, for example, through outreach screening, and 3) paediatric and allied health screening, assessment and therapy, by video-link (RFW Telecare) from the RFW centre into local schools, homes or health clinics. Since 2015, RFW has delivered up to 10,000 occasions of allied health services annually to children in schools in rural and remote communities through their Telecare programmes. In this paper, the word ‘Telecare’ refers to the name of the specific organisational – that is, the RFW programme – under consideration, which is a telehealth service. The majority of RFW Telecare works primarily through partnerships with rural and remote schools. Relationships with local schools are established through existing partner schools and through new relationships, for example, contacts established in the process of delivering outreach services. Partner schools provide people support in terms of a therapy aide or teaching assistant (if a parent cannot attend) as well as infrastructure for service delivery, that is, private space for student, internet access, computers, webcam and microphone.

In practice, a dedicated ‘Telecare Coordinator’ makes contact to coordinate the initial referral process, any ongoing screening and assessment and any subsequent therapy with clinicians. A therapy aide at the school helps to engage and support the child and troubleshoot technological issues during therapy sessions. RFW provides capacity building programmes for school staff on identified mental, behavioural and developmental disorders through webinars, feedback sessions and consultations. Parents are engaged throughout the RFW ‘Telecare for Kids’ process – they are provided with updates (when not present at sessions) and resources to support care at home. RFW relies on teachers, parents/carers and local health professionals to identify children who are at risk and then to refer them to RFW Telecare. Children are initially screened using an assessment form completed by teachers and parents, consent is sought, and there is subsequent follow-up by an allied health professional to determine suitability for therapy via Telecare. Therapies are delivered via technology by paediatric health professionals working from RFW ‘Telehubs’ in Sydney to rural and remote child clients, with the support of a local teacher, therapy aide and/or parent. Each Telecare programme runs for a set number of sessions, during which there are regular feedback sessions with teachers and parents with guidance on how to further support the child at school and at home. A report is delivered to parents and the child’s local health care provider at assessment and discharge points. RFW works to develop the capability and confidence of teachers to identify at-risk children, and support them during class.

Developing the programme logic model

The data collection process to inform the programme logic model was completed in two stages. The first stage comprised a literature review and the second stage involved in-person meetings with RFW staff. The initial draft logic model was developed based on the literature review, and then tested during in-person meetings to ensure its contents were accurate, its structure was clear, easy to follow and acceptable and reflects both the current practice and future aspirations of RFW. The current iteration of the logic model, titled ‘RFW National Telecare for Kids –Allied Health Service Logic Model’, emerged over a 12-month period (see Table 1) during which external researchers and the internal RFW research team conducted a series of consultations with RFW internal stakeholders, beginning with clinical and non-clinical Telecare programme implementers, to executive staff, including the organisation’s Chief Executive Officer. Each group of stakeholders was systematically engaged to first review and then meet in-person to provide comments, suggestions, edits and input.

Table 1.

Steps taken in developing the programme logic model.

Step 1:10 September 2016	In an inductive approach to identify the potential components of the logic model, a review of the literature (both peer-reviewed and the grey literature, including existing RFW internal documents) was conducted to answer the question ‘what is important to include in a logic model for in-school telehealth delivered to children with mental, behavioural and developmental disorders?’
Step 2: 20 September 2016and11 October 2016	The literature review was discussed in two workshop-style meetings with the RFW Telecare management team and was used to draw out a draft framework chart. Gaps were identified in the emerging logic model to be filled with information from the RFW Telecare plans and by the RFW personnel team’s context of service delivery in rural and remote Australia. These workshop meetings provided opportunity to incorporate information from RFW internal documents (standard operating procedures, guidelines and strategic plans), which were used to revise the draft logic model.
Step 3: 17 January 2017	RFW’s internal research team summarised the five A3 page long-form framework into a one A3 page logic model, bringing all the essential information in the logic model on the same page.
Step 4: 21 February 2017	The next step was a meeting held on 21 February 2017 to discuss the one-page logic model with the RFW Telecare management team and the internal RFW research team, with whom the draft logic model had been shared one week earlier. This meeting was convened to assess the extent to which the overall logic model fitted current practice and assumptions within RFW. The logic model was adapted in-line with the feedback, and was subsequently shared with the RFW Telecare management team for further feedback.
Step 5: 27 February 2017	The group highlighted the need to review the logic model against two frameworks: the RFW Strategic Plan to 2020, and MAST (Model for the Assessment of Telemedicine),¹² which was highlighted by the embedded university researcher and adopted by RFW and includes considerations of safety, clinical effectiveness and patient perspectives, and also economic and organisational aspects of telehealth, and the socio-cultural, legal and ethical aspects of the telehealth. The logic model was adapted in-line with these frameworks.
Step 6: 28 February 2017and22 March 2017	The external and RFW research team had two rounds of meetings with the Telecare management team and ‘Telecare Coordinators’ to review the logic model. Between the two meetings, the revised logic model was sent to the RFW Telecare management team and Telecare Coordinators for checks and feedback.
Step 7: 22 March 2017	Clinicians working in the tele-psychology, occupational therapy and speech therapy services (clinical team leaders and at least one colleague) reviewed the logic model. Subsequently, representatives of clinicians from each discipline met separately with the research team to provide feedback on the logic model.
Step 8:16 May 2017	The next draft was shared with the Telecare management team and RFW Executive, that is, the Chief Executive Officer and RFW Health Director, in preparation for a meeting with the research team, after which the logic model was revised into the final version: the RFW ‘National Telecare for Kids – Allied Health Service Logic Model’

RFW: Royal Far West

The RFW Telecare logic model includes three major components: inputs, outputs, and outcomes. Inputs refer to the resources that go into implementing the Telecare programme. Outputs include activities involved in telehealth service delivery and the participants involved in delivering and using those services. Outcomes are defined in three stages: short-term, defined by change in service delivery, participant awareness of the programme, knowledge of needs and improvement in skills to help children at risk; medium-term, defined by change in participant behaviour after improving their awareness, knowledge and skill; long-term, defined by long-term change in quality of life, financial impact and change in industry culture or practice with increasing acceptance of the RFW Telecare model of care, increased capacity building in community, and securing of funding to ensure the care model’s long term sustainability.

Insights from the consultation process

Rather than describe each component and item in the RFW Telecare logic model, this article focuses on the insights gained from the consultations held during the process of developing the logic model. By defining what should constitute inputs, outputs and outcomes, the logic model development process provided the RFW Telecare management team with an opportunity to participate in conversations, clearly articulating what the RFW Telecare service model entails.

Defining input – the logic model as an input

For the Telecare management team, clearly defining input was necessary to facilitate the development of systems and processes that support telehealth rather than existing systems and processes for in-person service ‘dictating’ Telecare input. The logic model also helped to facilitate impulse control and to avoid ‘scope creep’; for example, defining the age range for children in the priority target group for Telecare. Further to this, rather than a stand-alone document, the logic model was viewed by the management team as reflecting the content of two inter-linking and dependent documents: the ‘Telecare Guidelines’, which capture policies, and the ‘Telecare Standard Operating Procedures’, a task-level analysis document.

But the logic model became an input in itself – it was identified as a tool to develop a common language across service delivery, research, business development, marketing, partnerships and advocacy teams, and to promote understanding of the purpose, breadth and complexity of RFW Telecare services. By coherently capturing such complexity on one page, the logic model was seen as having much value for communicating to government, funders and other key partners in a way that, in the words of one RFW executive, ‘Makes this [Telecare] real…so other people will believe in it and invest in it.’ In addition, the logic model served as an inward facing document as it captures knowledge which was hitherto ‘socialised’ through conversations in hallways and boardrooms. It therefore has the potential to enhance efficiency (by removing the need to ‘have the same conversations over and over again’) and consistency (by ‘making sure the same story is being clearly told each time’). Hence its use in the induction of new Telecare Coordinators.

Clinicians saw it as a tool to support their understanding of the ‘magnitude’ of RFW Telecare for Kids, as ‘understanding the bigger picture can be tricky for new clinicians’, who often arrive and need to learn the technology and equipment, familiarise themselves with cases and, in some situations, learn how to adapt their practice for a telehealth setting. But some clinicians spoke of how the entire multi-disciplinary logic model was overwhelming and that new staff may need someone to talk them through it. Hence, they requested additional discipline-specific logic models that ‘cut out the bits we don’t have to worry about’.

Defining output – debates on language

The first debate was on whether the RFW Telecare service was an ‘intervention’ or a ‘therapy’. Initially the word ‘therapy’ was used throughout the document. The management and implementing teams questioned this, because RFW Telecare goes beyond delivering one-to-one therapy and includes building capacity of children, therapy aides, teachers and schools. The team considered using the word ‘intervention’, but the implications of the word for Aboriginal and Torres Strait Islander communities in Australia, given the history of forced government ‘intervention’ selectively targeted at them, made the word unsuitable. The term ‘Telecare Delivery’ was adopted to capture the broad range of the RFW Telecare services.

The second debate was between the management and the clinical teams, on whether the term ‘evidence-based’ was appropriate, given the evolving evidence around gold standard treatment for conditions for which RFW Telecare for Kids provides a broad service, for example, Autism Spectrum Disorder, Foetal Alcohol Syndrome Disorder and/or Complex Trauma. The team debated whether instead the use of the term ‘evidence-informed’ would be more appropriate. But the decision was made to use the term ‘evidence-based’, with the understanding that RFW will work in an ‘evidence-informed’ way, and respond to emerging science.

The third discussion, again between the management and the clinical teams, was on the use of the phrase ‘helping inform diagnosis’ to describe a goal of clinicians’ activities. RFW clinicians debated whether the phrase accurately described their practice given that they currently provide only a small number of comprehensive assessments, and because they were concerned about the accuracy of remote diagnosis and how a child’s behaviour may be influenced using technology. But the management team asked clinicians to consider how their activities (e.g. capacity building for parents and school principals, teachers and therapy aides) and the information they share with their clients’ local service providers may contribute to ‘helping inform diagnosis’. This explanation made the clinicians more comfortable with the phrase.

These discussions helped to bring internal RFW Telecare stakeholders onto the same page, and led to a greater understanding of one another’s roles and their role relationships with Telecare. For one of the Telecare managers, the ‘process has put a shining light on how complex our world is and how many stakeholders it takes to get it right.’ The debates helped clinicians, management and Telecare coordinating staff better understand the range of activities being fulfilled by each role, and how the role tasks relate to one another. In particular, the debates helped clinicians understand the extent of activities of the Telecare Coordinators beyond sending through clients, which made the Telecare Coordinators, whose role is often invisible, feel better appreciated.

Defining outcomes – what is the programme set up to achieve?

The process allowed internal RFW Telecare stakeholders to better understand one another’s expectations of outcomes. While the management team was focused on being able to report tangible outcome measures, clinicians were more focused on the journey of individual clients. For example, there was a debate on the short-term ‘improvement’ for children participating in RFW Telecare for Kids. One of the earlier versions listed only a demonstrated improvement in Goal Attainment Scaling (GAS) goals in the outcomes section.^13,14 But RFW clinicians felt that GAS was not a sufficient measure of improvement on its own, especially in the short-term, and that there was the need to include discipline-specific measures and suggested these should be reported in the programme logic model as well. On the other hand, the management team saw the value of listing a single, quantifiable and reportable evidence-based outcome measure that is comparable across services. However, through these discussions the management team agreed to include discipline-specific measures, alongside GAS goals in the model, and, in practice, for completeness.

In an earlier version of the logic model, ‘demonstrated improvement in academic assessment’ was listed as a medium-term outcome. But clinicians raised concerns about aiming to improve ‘academic assessment’ and suggested a change to ‘academic achievement’, which shifts the focus to progress measured against an individual child’s previous performance, rather than assessment that ranks children against an external objective measure of performance. The management team accepted this as a more meaningful measure of progress for children who may be dealing with developmental or behavioural challenges.

In addition, clinicians queried the inclusion of improvements in school literacy and numeracy scores as a long-term outcome for schools, as they do not have this outcome in mind when delivering Telecare to clients. However, clinicians were better able to accept this measure when it was explained to them in terms of the broader intended outcomes of RFW Telecare for Kids through long-term engagement and capacity building with schools (and for school staff to recognise, understand and support children in need), which may lead to improved attendance, class behaviour and academic performance of individual children, and a positive impact for the overall school including literacy and numeracy scores. In earlier versions of the logic model, long-term outcomes focused on improving outcomes for individual children, but did not explicitly mention that communities may be better off because of Telecare for Kids. This highlighted the need to deliver the full package of services (especially capacity building) to achieve community-level outcomes.

Uses of the programme logic model

The value of the process shows in two forms: first, the use of the logic model to facilitate communication within the implementing organisation and with external stakeholders; second, the process of working to develop the logic model led to important debates and discussions within the organisation which facilitated common understating of the Telecare programme, its services and priorities. The process helped to synthesise understanding of internal stakeholder groups; notably by highlighting that the programme is targeted not only at children, but also their families, schools and local services providers; by clarifying language and outcomes; and by demonstrating the complexity of the RFW Telecare for Kids programme to the multitude of its stakeholders. It moved internal stakeholder perceptions on logic models away from being seen as an essentially ‘academic’ tool and exercise to a more ‘practical’ and therefore a more valuable process and product, grounded in reality, which can help to embed and support scale-up of this telehealth programme. Table 2 shows additional uses to which the programme logic model was put within the implementing organisation.

Table 2.

Uses of the logic model identified in the process of development.

Induction with new staff and training

Improvement of service delivery model and processes

Refinement of evaluation methods and aims

Communications tool

Marketing and advocacy

Work planning and resourcing

Prioritisation and focus

The value of logic models as a tool for programme planning, evaluation and communication is well recognised, but the value of the logic model development process itself is less discussed or understood.³ Beyond use in communication, the RFW logic model development process facilitated consensus among stakeholders, in line with previous experience which suggests that using a participatory approach to develop programme logic can strengthen the relationship of the research team with the community, help to build trust based on transparency and foster sustainability, and also showed that the value of logic models in planning, evaluation and communication is limited if stakeholder input is not part of the development.^15–17 Indeed stakeholder engagement can play a significant role in embedding and scaling up programmes within their implementing organisations, a major challenge for the sustainability of real time telehealth services which require much reorganisation of existing service delivery platforms.

Of course, the final logic model is not a magic wand in itself. It is only as good as the logic and thinking that went into it. The logic model process and product, however, appear to have been an unexpectedly useful tool to support systematic thinking of the complex systems of telehealth itself and the complex systems within which the telehealth services weave. Through co-design and engagement, we initially started with the end in mind – asking the big questions – what are the ultimate goals of this telehealth programme? Why? And How? What will demonstrate these are being met? We discussed the inputs, outputs and outcomes; we moved them around in columns and boxes; and brainstormed the validity and strength of links in the model. We assessed the logic of our process by verifying within and across teams and through the literature and also the external researchers facilitating the process. This process exposed flawed or incomplete assumptions, and assumptions that were not acceptable to all levels of staff and management. The logic model is also a framework for evaluation as the features are explicit on how to measure success.

Securing long-term ongoing funding for telehealth is a major threat to sustainability. Although neither the programme logic model nor the process itself can solve funding challenges, the process of homing in on inputs, activities, stakeholders and outputs provided insights into potential ways to alleviate costs, and placed in context potential funding avenues. This is a pathway through which a programme logic model can be a tool in sustaining telehealth. There may now be more buy-in to provide resources (e.g. marketing and funding), as the logic model makes it clearer what is needed, for instance, having a clearer idea of where new funding opportunities align with existing and planned Telecare services. This has occurred, for example, in 2017; Medicare began to cover psychology services provided for using telehealth for rural and remote individuals; as well as the newly implemented National Disability Insurance Scheme. Since the logic model displays clearly if you have access to these inputs (resources) then you can accomplish outputs, we observed that the logic model assisted staff in advocating for resources from their managers.

As a busy, multi-modal service organisation with multiple stakeholders and programmes, it took a lot of trust for the RFW executive and Telecare management team to agree to invest the significant time necessary to complete the multi-layered consultative development process of the logic model. In addition, there were concerns about divulging proprietary information to competitors. To conduct a similar process, researchers need to be aware of the possibility of these challenges, and plan for possible obstacles. And there is also the challenge that research funders do not typically support efforts at stakeholder engagement.¹⁵

Conclusion

Our experience demonstrates the potential of an evidence-informed and consultative process to facilitate cross-organisational understanding and agreement on the delivery of a complex intervention that requires significant reorganisation of the process of service delivery. The logic model process increased the visibility of the complexity of the service, and the service re-organisation. The process and the model itself enabled staff to provide greater focus to areas needing attention. However, the sustainability of such a telehealth programme requires much more than internal stakeholders being on the same page. It requires, for example, getting external stakeholders (e.g. governments, non-government organisations, philanthropists, and local health care providers who are interested in using telehealth to deliver services to rural and remote communities) on the same page with the organisation as well; a process in which the logic model can also play a role. The logic model can facilitate efforts to lobby for change to improve funding for telehealth services nationally. We recommend that organisations aiming to embed, sustain and scale telehealth services consider using a similar multi-stakeholder approach to develop a logic model for their telehealth programmes.

Footnotes

Contributorship

SA, CL and AM researched literature, consulted with the partner organisation and conceived the study. SA, CL, PC and AM were involved in the collection of data. All authors contributed to analysis. SA wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved of the final version.

Conflict of interest

Royal Far West delivers specialist paediatric and allied health care for children in rural and remote Australia in-person and through real-time telehealth. Royal Far West provided in-kind resources and time towards this project.

Ethical approval

Not applicable.

Funding

This work was supported by a National Health and Medical Research Council (NHMRC) Translating Research into Practice (TRIP) Fellowship (MARTINIUK APP1112387, 2016 and 2017) and a University of Sydney Industry Engagement Commercial Development Industrial Partnerships (CDIP) Grant (K2901 2017). SA is funded by a NHMRC Early Career Fellowship (APP1139631).

Guarantor

AM.

Peer review

This manuscript was reviewed by two individuals who have chosen to remain anonymous.

References

Rada

Telemedicine: are we advancing the science?

Cochrane Database Syst Rev 2015; 9: ED000105. doi:10.1002/14651858.ED000105.

Whitten

Allen

ACE.

Analysis of telemedicine from an organizational perspective. Telemed J 1995; 1: 203–213.

Taylor-Powell

Henert

Developing a logic model: Teaching and training guide, https://fyi.uwex.edu/programdevelopment/files/2016/03/lmguidecomplete.pdf (accessed 12 December 2018).

Australia Department of Education and Training (issuing body.) Australian Early Development Census national report 2015 : a snapshot of early childhood development in Australia. Canberra, A.C.T. Department of Education and Training, 2016. https://www.aedc.gov.au/Websilk/Handlers/ResourceDocument.ashx?id=45cf2664-db9a-6d2b-9fad-ff0000a141dd (accessed 12 December 2018).

Robinson

Holbrook

Bitsko

et al.

Differences in health care, family, and community factors associated with mental, behavioral, and developmental disorders among children aged 2–8 years in rural and urban areas – United States, 2011–2012.

MMWR Surveill Summ 2017; 66: 1–11.

Chan

Lake

Hansen

The early years: Silent emergency or unique opportunity?

Lancet 2018 ; 389: 11–3.

Daelmans

Darmstadt

Lombardi

et al.

Early childhood development: The foundation of sustainable development.

Lancet 2018 ; 389: 9–11.

Richter

Daelmans

Lombardi

et al.

Investing in the foundation of sustainable development: Pathways to scale up for early childhood development.

Lancet 2018; 389: 103–118.

Rogers

Vismara

LA.

Evidence-based comprehensive treatments for early autism.

J Clin Child Adolesc Psychol 2008; 37: 8–38.

10.

Prior

Roberts

Rodger

et al. A review of the research to identify the most effective models of practice in early intervention for children with autism spectrum disorders, https://www.dss.gov.au/sites/default/files/documents/10_2014/review_of_the_research_report_2011_0.pdf (accessed 12 December 2018).

11.

Kelleher

Gardner

Out of sight, out of mind — behavioral and developmental care for rural children.

N Engl J Med 2017; 376: 1301–1303.

12.

Kidholm

Clemensen

Caffery

et al.

The Model for Assessment of Telemedicine (MAST): A scoping review of empirical studies.

J Telemed Telecare 2017; 23: 803–813.

13.

Cytrynbaum

Ginath

Birdwell

et al.

Goal attainment scaling: A critical review.

Eval Q 1979; 3: 5–40.

14.

Ruble

McGrew

Toland

MD.

Goal attainment scaling as an outcome measure in randomized controlled trials of psychosocial interventions in autism.

J Autism Dev Disord 2012; 42: 1974–1983.

15.

Afifi

Makhoul

El Hajj

et al.

Developing a logic model for youth mental health: Participatory research with a refugee community in Beirut.

Health Policy Plan 2011; 26: 508–517.

16.

Maini

Mounier-Jack

Borghi

How to and how not to develop a theory of change to evaluate a complex intervention: Reflections on an experience in the Democratic Republic of Congo.

BMJ Glob Health 2018; 3: e000617.: https://doi.org/10.1136/bmjgh2017-000617 (2018, accessed).

17.

Bagot

Cadilhac

Kim

et al.

Transitioning from a single-site pilot project to a state-wide regional telehealth service: The experience from the Victorian Stroke Telemedicine programme.

J Telemed Telecare 2017; 23: 850–855.