Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic

Abstract

The COVID-19 pandemic required large-scale service delivery changes for government, and provided the opportunity for evaluators to step up and support decision makers to understand the impact of these changes. Rapid evaluation methods (REM) provide a pragmatic approach for generating timely information for evidence-based policy and decision-making. Grounded in developmental and utilisation-focused evaluation theory, REM incorporates a team-based, mixed methods design, executed over a 6–8-week period. Customised rubrics were used to rigorously assess effectiveness and scalability of practice changes to inform COVID-19 response planning. REM is an alternative approach to full-scale evaluation models frequently implemented to assess policies and programs. Adapted use of REM suggests that meaningful insights can be gained through use of smaller scale evaluations. This article shares lessons learned from a novel rapid evaluation method applied in the context of the COVID-19 pandemic. The rapid evaluation approach was implemented to provide real-time insights and evaluative conclusions for 15 program and practice adaptations across Victorian health and human service settings. The article shares insights about the practical applicability of balancing rigour and timeliness when implementing a rapid evaluation, and strengths and limitations of working within a fast-paced evaluation framework. Findings can inform evaluative practice in resource and time-limited settings.

Keywords

rapid evaluation evaluation methods real-time evaluation team-based analysis COVID-19 pandemic

Introduction

The COVID-19 pandemic brought about inevitable changes to the way in which healthcare and human services care are delivered worldwide. The pandemic disrupted or halted critical health services, and in many places led to increased demand for health care (World Health Organization, 2020). Within Victoria, Australia, service changes were shaped by policy direction from State Government which aimed to limit physical interaction in the community and ensure infection control within high-risk settings while maintaining service continuity. The unfolding crisis led to increased demand by government policy makers for real-time evidence to optimise the development, implementation, and adaptation of the health system, and related areas of human services, in response to the COVID-19 outbreak.

In evolving public health contexts, such as the COVID-19 pandemic, traditional comprehensive evaluation approaches often lag behind fast-paced innovation (Laura et al., 2020). Faster techniques are required to understand effective service delivery changes within a public health emergency context and evaluation must balance the competing priorities of rigour and utility of evaluation findings (Raine et al., 2016). This requires novel approaches to feed into rapid evaluation processes and provide timely insights to inform effective response planning. The utilisation of ethnographic methods makes rapid evaluation methods (REM) valuable for providing contextually relevant evaluation findings in a timely and cost-effective manner.

The authors are staff at the Centre for Evaluation and Research Evidence (CERE), an internal evaluation unit within the Victorian Department of Health. The purpose of this article is to share insights drawn from CERE’s adapted rapid evaluative practice during the pandemic between March 2020 and June 2021. This article also responds to growing interest in the Australian evaluation community around the question of how should rapid evaluation methods be deployed and whether they can be adapted or improved? (Australian Evaluation Society, 2020). We share experiences on how CERE deployed the REM, incorporating a standard methodological evaluation framework and customised rubric to increase rigour. This article highlights the appropriateness of rapid evaluations in supporting innovation, adaptation and systems change in a dynamic and complex health crisis response. Findings are significant to inform practice for evaluators who intend to apply rapid evaluation techniques in resource and time-limited settings.

Rapid evaluation methods

While there is no agreed definition or consensus of whether an evaluation method is rapid, for the purpose of this article a rapid evaluation is defined as a rigorous, team-based evidence generation approach that utilises multiple data collection and analytical methods to identify operational problems and inform decision-making quickly (Anker et al., 1993; McNall & Foster-Fishman, 2007). Initially developed and promoted by the World Health Organization, iterative rapid evaluation techniques have been increasingly used in a broader range of settings including health care (Laura et al., 2020) and social care innovations (McCarthy et al., 2019).

Variations of REM include rapid appraisal (Beebe, 1995), rapid feedback evaluations (McNall et al., 2004), rapid participatory appraisals (Annett et al., 1995), rapid ethnographic assessments (Bentley et al., 1988), and rapid assessment, research, and evaluation (Trotter et al., 2001). Despite the variance in methodology and context of application, rapid evaluations have distinctive core elements including iterative data collection and analysis process, and short implementation timelines ranging from 4 days to 6 months. The evaluation process is often team-based involving stakeholder participation and triangulation of data collected using complementary qualitative and quantitative methods (McNall & Foster-Fishman, 2007). These features are consistent with perspectives highlighted by on rapid evaluations.

Delivering rapid evaluations requires intensive planning prior to fieldwork as a critical element to ensure prompt findings within the short timelines (Skillman et al., 2019). However, whilst speed for obtaining findings is a prominent feature of REM and other rapid methods, this rapidity creates potential challenges for credibility given the short implementation timeframes (Vindrola-Padros et al., 2020). Despite these limitations, creating a balance between speed and trustworthiness of evaluation findings remains paramount. Researchers have developed techniques to accelerate time-efficient collection and analysis of data including audio coding (Neal et al., 2015) and rigorous and accelerated data reduction (RADaR) technique (Watkins, 2017). More recently, other iterations of rapid approaches have been deployed including rapid impact evaluations (Rowe, 2019) which fosters utility for developmental and formative evaluation in ex ante and ex post settings. Such fit-for-purpose methods suggest that rapid evaluations can be rigorous and deliver legitimacy and credibility with decision makers and key stakeholders (Rowe, 2019). Within the evaluation field, however, there is still limited information about real world experience implementing rapid evaluations within large government institutions and utility of findings.

Use of rapid approaches in public health emergency contexts

Global public health emergencies are complex and multifaceted with wide-ranging contextual challenges. During outbreaks such as COVID-19, research and innovation have been progressively shown to play an important role during and after disease outbreaks. Scarcity of evidence in such time-limited settings can pose specific challenges for evidence-informed decision-making. Whilst emergency situations require timely intervention, the high volatility of such situations makes it difficult to plan for ongoing development. Within such dynamic settings, the use of rigorous and comprehensive evaluation designs to assess the impact of adaptive development programs and innovation is hampered by the complexity of contextual factors. Lessons can be drawn from previous infectious disease outbreaks such as Ebola in West Africa in 2014–2015 which necessitated rapid responses to facilitate control, treatment and prevention. An evaluation of UNICEF’s Response to the Ebola Outbreak highlighted that rapid response strategies in health emergency contexts should be evidence-based and utilise robust data on the evolution of practice over time and on community knowledge, attitudes and perceptions (UNICEF, 2016)

Recent literature also highlights increased attention to rapid research and evaluations in emergency contexts (Rodriguez et al., 2021). Rapid approaches can provide contextually rich information on behaviours influencing service use. Johnson and Vindrola-Padros (2017) provides further guidelines for undertaking rapid research in emergency contexts such as early engagement of community leaders, timely and ongoing information sharing and collaborative development of recommendations with local policy makers and practitioners. However, conducting rapid research may pose challenges given the limited time for triangulation and validation of findings with stakeholders. Additionally, a limited evidence base also exists around the ethics of conducting context-appropriate research during outbreaks.

Using rapid evaluation methods to support adaptation and systems change for the COVID-19 response

The COVID-19 pandemic brought forward innovative service and practice changes, some of which could have benefits for health and human services moving forward (for example greater service access due to digital delivery). In order to assess these service and practice changes, government policy decision makers rely on research evidence to inform decision-making. During the pandemic, rapid assessment approaches have been used to examine psychological and epidemiological correlates of COVID-19 concern (Nelson et al., 2020). Some studies have highlighted the value of rapid analytic approaches on gathering evidence to support behavioural change in emergency contexts such as COVID-19 pandemic (UNICEF,2020). However, there is still very limited evidence on specific rapid evaluation approaches and lessons from deployment during COVID-19.

Centre for evaluation and research evidence conducted 15 rapid evaluations from May 2020 to March 2021 over four tranches (Figure 1). Rapid evaluations spanned across multiple health (N = 9) and human service (N = 6) portfolios including perinatal services, specialist services, acute services, paediatric services, maternal and child health, family violence, mental health, homelessness, hospital in the home services and children and family services (see Supplementary Table 1).

Figure 1.

Overview of centre for evaluation and research evidence rapid evaluations implemented during COVID-19.

The aim of CERE implementing the rapid evaluation approach by the then Department of Health and Human Services was to provide decision makers with real-time insights in a 6–8 week period about aspects of program adaptation that were working well. This enabled continuous improvement of program and policy design in the emergency context. CERE explicitly sought to answer three key evaluation questions:

1. What are health and human services doing differently as a result of the COVID-19 response?

2. What is the impact of these changes in service delivery and practice?

3. What aspects of the changes should the department seek to keep or extend?

Rapid evaluation findings informed COVID-19 recovery and planning for the department and provided information to guide responses to potential future waves of COVID-19 infection or other pandemics. The lessons learned from implementing the rapid evaluation approach can be applied to evaluations in similar contexts.

Methods

The REM approach adopted by CERE draws on established rapid evaluation approaches to assess effectiveness and sustainability of practice changes implemented in response to the health crisis (McNall & Foster-Fishman, 2007). CERE applied an adapted version of the REM combined with principles from developmental evaluation (Patton, 2010) tailored to emergency management and recovery planning. The methodology examined the nature and implications of the innovative and adaptive processes implemented as part of the COVID-19 response. Our rapid evaluations had a strong utilisation-focus (Patton, 2008) and used a combination of features from existing rapid evaluation models.

This article shares our experience delivering two rapid mixed methods evaluations that aimed to assess service delivery changes in the COVID-19 context: the real-time evaluation of the High Risk Accommodation Response and the rapid evaluation of Telehealth in Perinatal Services.

Study 1: Evaluation of the high-risk accommodation response

The High Risk Accommodation Response (HRAR) was a revised service model to provide safe accommodation to homeless people and rough sleepers to minimise COVID-19 infection risk. The intervention aimed to: (1) ensure appropriate public health measures were in place to protect the health and wellbeing of residents across public housing and other high-risk accommodation settings with shared facilities; (2) ensure culturally appropriate and accessible service and community engagement strategies were in place to respond to COVID-19 outbreaks; and (3) establish a clear operating model to deliver these services.

The HRAR rapid evaluation was co-designed by CERE in collaboration with the department’s Human Services COVID-19 Readiness and Response Centre. The evaluation was conducted in early stages of the HRAR response to assess effectiveness and support ‘real-time’ corrective action for implementation (Rogers, 2020). At inception, stakeholders were engaged to provide in-depth information about service design which was critical to evaluation planning. This enabled evaluators to clarify the purpose and objectives of the evaluation, intended use of findings and preferred format of the end-user products (Bamberger et al., 2006). Being clear about the intended use of real-time findings guided pragmatic choices on the magnitude of data collection. An Evaluation Advisory Group was established with eight key stakeholders (including authorising stakeholders, policy/program content advisors and end users/beneficiaries) that met on a weekly basis for an hour. This governance structure and meeting frequency was essential in the rapid evaluation process to build ownership but would not necessarily be acceptable for longer-term evaluations. Progressive meetings were used to access program documentation, refine key evaluation questions, submit data requests and align validation and reporting to ensure fit-for-purpose evaluation products.

A tailored mixed methods design supported efficient rapid evaluation delivery within an 8 week period. CERE developed standardised templates a priori’ to fast-track evaluation design and delivery, including an evaluation plan, data matrix, interview and focus group discussion guide, as well as final reporting templates. These templates were stored on a secure online database with shared access for the evaluation team. CERE drew on existing state-wide administrative data and program documentation supplemented by survey data, key informant interviews and focus group discussions to provide insights and explanatory information on the program. Figure 2 outlines an example Gantt chart used by CERE to document the steps involved in implementing the rapid evaluation.

Figure 2.

Evaluation steps implementing centre for evaluation and research evidence rapid evaluations.

Document Review: We undertook a rapid literature review supplemented by analysis of program documentation to provide deeper understanding of critical jurisdictional, national and international learnings and a baseline record for changes in service delivery compared to pre-COVID context.

Site visits: CERE collaborated with lead providers who visited the high-risk accommodation sites to provide information and support for infection control resourcing. In total, 3874 of 4013 (97%) of high-risk accommodation properties were visited. Data collection in excel spreadsheets was simultaneously completed during site visits for real-time reporting on quality issues that were subsequently discussed at regular evaluation group advisory meetings. Through site visits, catchment management plans could be developed in response to real-time findings.

Survey: The evaluation team developed a brief survey (5–10 minutes) to examine the effectiveness of the HRAR response within shared facilities. The survey included both open ended and close-ended questions to assess the extent to which public health measures were appropriate to minimise community transmission. Purposive sampling (Palinkas et al., 2015; Robinson, 2014) was used to seek perspectives from a range of service providers. Snowballing techniques were applied where some service providers distributed the SurveyMonkey link to others. Consistent with Rubin and Rubin’s (2005) guidelines, participants were selected based on their knowledge of the service, their diverse location and perspectives and willingness to participate in the evaluation. Whilst this sampling method may risk researcher bias, the evaluators sought to minimise threats to internal validity by defining and sourcing the target population in consultation with stakeholder advisory groups.

In-depth Interviews: Semi-structured interviews were carried out with HRAR participants to elicit their experiences of the impact of public health measures within the high-risk setting. A purposive sample of 15 lead providers was selected for inclusion in the study. Consistent with the overall rapid approach, tightly scoped 15–30 minute interviews were conducted face to face, via telephone, or video conference with the providers. Some interviews were transcribed by an external provider service with shorter turn-around time of 1 day. In other instances, evaluators developed interview summaries for collation into a single reporting document.

Centre for evaluation and research evidence conducted regular sense making workshops to triangulate and converge information from multiple sources. Despite truncated timelines, all rapid evaluations used an ethical framework and involved voluntary participation with informed consent and data privacy. Culturally and Linguistically Diverse and Aboriginal and Torres Strait Islander participants were provided an opportunity to have an interpreter if required. Participants were contacted prior to the interview to obtain written consent which they emailed to one senior evaluator. During this initial contact participants were also informed that they could find a private space to have the interview, could turn on virtual backgrounds for privacy, and provided choice of interview time and place.

Virtual Focus Group Discussions (FGDs): Three FGDs were conducted via Microsoft Teams with lead providers to better understand their perspectives about how HRAR innovations were impacting infection control and barriers to minimising transmission within the catchment areas. Our team emailed to participants instructions on online discussion etiquette and material for use during discussions ahead of time. FGDs comprised about 6–8 participants who reflected on the response strategies they considered to be most effective. Commensurate with the arguments of Dos Santos Marques et al., 2021, the virtual focus groups accelerated data collection and differed from in-person FGDs given that the co-moderator played a dual role in managing chat discussions and note taking.

Study 2: Assessing changes in telehealth delivery in pregnancy and afterbirth during COVID-19

Due to COVID-19 social distancing restrictions, telehealth became a service offering by maternity services to support pregnancy and home-based care after birth (Department Health and Human Services, 2020). In this study, telehealth described the use of information and telecommunication technologies, including telephone and or video conferencing calls through Skype, Zoom and Health Direct (Australian government funded service providing quality, approved health information and advice). Telehealth played a critical role in the COVID-19 response because it could be used by the wider workforce, facilitated quick triage; and limited infectious disease transmission through eliminating person-to-person contact (Smith et al., 2020).

This evaluation was deployed between May and June 2020 and aimed to assess the impact of the shift from face-to-face delivery (where appropriate) to telehealth service delivery. Evaluation scope included public hospitals and dedicated pregnancy and postnatal services.

Implementation of the rapid evaluation involved establishment of an evaluation advisory group. A theory of change and program logic model were developed in consultation with key stakeholders at a workshop to contextualise COVID-19 service adaptations and desired outcomes. This facilitated a shared understanding of how and why desired changes were expected to occur (Figure 3). Equally, negotiating evaluation scope early on assisted managing stakeholder expectations of evaluation delivery and fit-for-purpose end evaluation products. Evaluation products included short 10-page reports, interview summaries and publications (Ore, 2021).

Figure 3.

Theory of change for use of telehealth in pregnancy and childbirth.

The rapid evaluation team drew on existing data from the Victorian Integrated Non-Admitted Health Data (VINAH) (N = 13,207 telehealth contacts). Descriptive analysis of this data was completed using Microsoft Excel to summarize data and highlight emergent patterns and trends. The evaluation team described measures of frequency including mean differences in service usage patterns.

Our approach utilised parallel data collection and analysis. Two surveys were distributed including a maternity workforce survey (N = 87) and a consumer survey (N = 28). The workforce survey provided insight into how telehealth was being delivered by the maternity workforce during COVID-19. Nine health services were recruited to assist with the distribution of the SurveyMonkey link for the consumer survey to women. A privacy notice was included with the consumer survey to acknowledge compliance to privacy and data security standards.

This study also utilised semi-structured telephone interviews (N = 15) to collect qualitative data from participants. Interviews were audio-recorded, and the evaluation team independently undertook extensive review of the audio transcripts using the RADaR technique for analysis (Watkins, 2017). Evaluators initially organised the data in consistent format then iteratively analysed and reduced data in Microsoft Excel tables, drawing out high impact quotes and themes for subsequent comparative team-based analysis. Given the time limitations, emergent themes were validated with the evaluation working group instead of transcript review by participants where they may not significantly add to the accuracy (Hagens et al., 2009). This decision was based on the premise that audio-recording ensured accuracy of transcripts and researchers could confirm meaning of data due to familiarity.

A purposive sample of Koori Maternity Service workforce were selected to take part in a focus group discussion. These participants were specifically selected to gather information on whether the intervention had facilitated increased empowerment for partners and the community (including Aboriginal communities) to drive reform and service improvements. The interviews and focus group collected data on women’s experiences with using telehealth services, and improvements required to support timely service access and clinical quality. To increase trustworthiness and rigour of our findings, we undertook data triangulation and tested our findings with stakeholders.

Value judgements in this study were based on discussion with the evaluation advisory group about the strength of evidence of service and practice changes, assessed against eight evaluative criteria in a customised rubric (Figure 4). The rubric outlined specific criteria and standards for judging success and bridging the knowledge gap between stakeholders and evaluators (Dickinson & Adams, 2017). Qualitative descriptors for each criteria provided shared understanding of outcome measures such as reduced risk of infection and improved client experience. The development of the criteria of merit (Davidson, 2005; Davidson et al., 2011) on what constituted negative or positive change was informed by literature, practice and inputs from program stakeholders and consumer representatives. CERE found that use of rubrics was valuable in facilitating transparent and consistent evidence into evaluative conclusions based on breadth of data sources and evidence. However, it was difficult to complete rubrics in instances where other rapid evaluations lacked data. Both positive impacts and some unintended outcomes of service and practice changes were associated with the expedited introduction of telehealth in this study.

Figure 4.

Perception of change rubric – Telehealth delivery in pregnancy and afterbirth during COVID-19.

The rapid evaluation insights were valuable to inform health services, workforce and families about the relevant benefits and challenges of using telehealth. Findings facilitated triple-loop learning to inform areas for further investigation. Establishing evidence-informed practices in this emerging dimension of health care delivery during COVID-19 was important to respond to and mitigate potential health risks, unintended consequences, including clinical outcomes and experience or costs that could be associated with rapid adoption of new technologies.

Findings

The move to rapid methods represented a significant shift for CERE’s evaluation delivery practice which generally involved tailored mixed methods formative and summative evaluations conducted over 6–12 months. Through rapid evaluation implementation, CERE drew on established techniques from literature and practice to successfully deliver evaluation products within shorter periods. CERE’s experience of implementing rapid evaluations acknowledges the reality for many government and non-government audiences that decisions about program funding and continuation are often made within weeks and months, rather than years. As the pandemic progressed with subsequent delivery, CERE found that rapid evaluations can produce meaningful and valued insights in a timely and cost-effective manner.

Overall, rigorous planning is essential to ensuring timely delivery of rapid evaluations. Throughout the design and delivery of the rapid evaluations, there was regular communication and meetings to capture key findings in real-time. These were vital to flag any issues identified and reflect on the effectiveness of evaluation delivery. In addition, principal evaluators of rapid evaluations compiled one-page summaries and presentations that were shared in real time with program stakeholders. However, CERE found that timing for when to conduct regular feedback meetings was significantly influenced by executive sponsor availability and data availability.

The subsequent section discusses our experiences of the key strengths in application of the REM, and challenges faced during evaluation implementation. Drawing on our experience and literature (Rogers, 2020), distinctions between traditional evaluation methods and REM are presented in Table 1.

Table 1.

Comparison of conventional evaluation approaches and rapid evaluation methods.

Conventional evaluation methods	Rapid evaluation methods
Conventional evaluation methods	Strengths	Weaknesses
For rapidly changing interventions, limited documentation can impose difficulty to inform changes or document lessons for future interventions	Real-time data collection to provide evidence that can be used immediately and in future for other interventions	Managing a potentially emerging set of evaluation questions and purposes can be difficult
Delay between data collection and evaluation reporting	Rapid feedback in real time reporting of evaluation data with rapid analysis and tailored end products such as 1-page summaries shared with stakeholders	Risk of providing a false view/inaccurate findings based on interim evidence or limitation of data collection within tight timeliness (e.g., small and limited samples)
Project reporting without systematic use of evaluative evidence in between	Multiple timings of evaluative activity completed at a number of points throughout evaluation implementation	Addressing casual inference in real time, often without credible counterfactuals
Mid-term evaluations support single loop learning and end of project evaluations support double-loop learning where modifications can be made to theories of change	Includes 3 looped processes for sensemaking and learning. Can identify inconsistencies, support modification of assumptions/theory of change and review of evidence being used to support decision-making	Requires adequate content knowledge to support triple-loop learning informed by evidence rather than simply documenting assumptions
Evaluators drawing conclusions and making recommendations that are reported back for action	Explicitly bringing a range of key stakeholders together to validate data, and jointly develop recommendations for action	Managing different perspectives of evaluative criteria and what are appropriate ways to synthesise evidence about performance across different dimensions – such as trade-offs between conflicting objectives

Delivering a credible evaluation using a team-based approach (Fernald & Duclos, 2005) is dependent on several factors. CERE found that having two experienced full-time evaluators allocated for each rapid evaluation cycle, supplemented by optional one or two part time less experienced evaluators was vital to fast-track delivery. Although each rapid evaluation was resource intensive, the team-based approach enabled multiple streams of work to be delivered concurrently and within a short timeframe. The HRAR response site visits were conducted at multiple time points to allow for evaluative evidence to support improvements of the COVID-19 response in real time. A key enabler for fast tracking evidence generation and validation was the utilisation of rapid analysis techniques for efficient validation processes.

Keeping the evaluation team consistent, was a vital aspect to timely completion of the evaluation and strengthened accuracy of findings. Having multiple researchers present whilst conducting interviews allowed for differential perspectives in data interpretation and reduced the potential for interviewer bias (Frey, 2018). However, as the number of rapid evaluations progressed through each tranche, it became increasingly challenging for CERE to conduct multiple rapid evaluations in parallel due to competing priorities with evaluators within the team and also maintaining the intensity required for fast-paced evaluation delivery. Communication modes were also adapted to shorten real-time feedback meetings using online platforms and provide snapshots of key insights to diverse stakeholders. A positive outcome of the department completing a considerable number of rapid evaluations during COVID-19, were cross project findings to inform decision-making.

Although CERE initially set out to conduct evaluations over a 4–6-week period, to sustain many rapid evaluations, it was more feasible to work across a 6–8-week period. Given the resource intensity required to deliver rapid evaluations, a 2-week break period between commencing another rapid evaluation was also found to be a necessary feature of rapid evaluation planning to avoid staff burnout.

Discussion

This article demonstrates an alternative approach to full-scale evaluation models that are frequently used to support assessment and continuous improvement for government services. Although comprehensive evaluations have a wider scope, purpose and longer time periods for in-depth assessment, this research suggests that rapid evaluations can produce time-efficient evaluative conclusions to inform recovery and planning during a health crisis.

Centre for evaluation and research evidence used a team-based approach to facilitate accelerated analysis and synthesis of data gathered from the field. This approach was essential to reduce travel, minimise transcription costs and ensure that analysis was timely. Using mixed methods ensured that the qualitative evidence collected offered explanatory insights that would not have been available if only quantitative methods were used.

The REM approach was effective in delivering fit-for-purpose findings to support decision-making within the dynamic COVID-19 context. The rapid evaluation approach produced findings about the service delivery changes due to COVID-19 response, the outcomes and potential for sustenance of the practice changes that were valued by end users. During field work, the rapid evaluations provided immediate feedback for the strategic planning team for the HRAR response which enabled them to make improvements based on the evidence instead of later on in the pandemic. The approach delivered early implementation insights within a much shorter period than traditional evaluation which can take much longer periods (12 months to 5 years) depending on the design. CERE observed that timely delivery of evaluation outputs supported a high level of utilisation for decision-making and influenced program adaptation and improvements. Learnings from the evaluation have been incorporated into commissioning the next phase of HRAR delivery, including a strengthened data reporting and evaluation framework.

The telehealth rapid evaluation case study led to the development of guidance to the sector reflecting the strengths and weaknesses of the service model. Findings from this evaluation also informed decisions on extending the use of telehealth to some regions and identified priority cohorts requiring greater support to maintain connections and increase telehealth uptake during the COVID-19 crisis. The evaluation findings informed real-time departmental decisions on next steps in the dynamic environment of a health emergency.

From CERE’s experience, the efficacy of REM is in part determined by whether some critical pre-conditions are in place. The REM approach works well within settings that have a highly engaged project sponsor, clear program documentation, pre-existing administrative data and access to key stakeholders including service users. In the case studies presented, these pre-conditions were present. Use of de-identified administrative VINAH data facilitated a purposive sample of pregnant women and mothers from both metropolitan and rural regions. Inclusion of diverse groups of participants was vital for comparative analysis of service usage patterns and identification of access gaps among both vulnerable cohorts and those with less risk. Data triangulation was conducted iteratively at the end of each day, to provide 1-page summaries to the stakeholders.

Although rapid evaluation processes facilitate time-efficient findings, we acknowledge there could have been potential loss of data through our accelerated analytical processes. Similarly, working in a dynamic context means that data is sometimes incomplete. Therefore, systems data should be interpreted with caution if the data is inconsistent and/or may not be representative of service use due to the evaluation being conducted in the early stage of implementation during the pandemic. CERE’s reflections on deployment of REM within the COVID-19 context acknowledge that findings were time and place-specific providing a snapshot of a point in time. Samples achieved for surveys and interviews may not be comprehensive, although this was counterbalanced through triangulation with population-level administrative data. Similarly, there is a possibility of overrepresentation of more technologically literate service beneficiaries through use of online surveys.

Resourcing of experienced evaluators with expertise to interpret large data sets remains a major challenge to successful implementation of rapid evaluations. Budget and staffing availability should at a minimum ensure resourcing for two full-time experienced evaluators, optional part time evaluators and resourcing to support fast production of transcription for interviews and or surveys.

Lastly, delivering evaluations within an 8-week period requires continuous collaboration with policy or program content experts to acquire meaningful stakeholder input throughout the rapid evaluation cycle. Stakeholder engagement in the first weeks of project delivery is critical to REM success.

Conclusion

Centre for evaluation and research evidence delivered evaluations to produce results that could feed into policy and practice quickly, but yet were sufficiently robust to provide good guidance for decision-making. Evaluation activities were delivered in parallel to ensure timely completion, an aspect that requires intensive resourcing. While it proved possible to deliver meaningful and valued insights quickly, the process is more resource intensive than longer-term evaluations.

By sharing the steps and rapid techniques applied in the evaluation process, this article highlights the factors and pre-conditions that need to be in place for efficient rapid evaluations. Building on this momentum and practice experience, key insights of situations where rapid evaluations might be most effectively deployed are offered. This article adds to the body of knowledge around evaluative practice and methods particularly for more responsive, fast turn-around options for conducting evaluations in an emergency context. Findings are relevant for professional evaluators and evaluation end-users seeking to understand the basis of rapid evaluation processes, and what they can be expected to deliver.

Supplemental Material

sj-pdf-1-evj-10.1177_1035719X211057630 – Supplemental Material for Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic

Supplemental Material, sj-pdf-1-evj-10.1177_1035719X211057630 for Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic by Milbert Gawaya, Desiree Terrill and Eleanor Williams in Evaluation Journal of Australasia

Footnotes

Acknowledgments

We would like to acknowledge the contribution of key stakeholders contributing to the rapid evaluations including department staff, workforce, consumers, and key experts. We would also like to thank and acknowledge the CERE team who also led and supported the 15 rapid evaluations: Dr Lisa Thomson, Dr Jan Browne, Bernadette Pound, Wendy Wade, Dr Clare Carberry, Dr Timothy Ore, Nikki Condon, Diz McKinnon, Abirami Jayawardena, Meredith Jones, Tom Hannon, Kathryn Logan, Jane Howard, Delyth Lloyd, Emma Braithwaite and Nerida Rixon.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Milbert Gawaya

Desiree Terrill

Supplementary material

Supplementary material for this article is available online

Author Biographies

Milbert Gawaya is a Senior Evaluation Specialist at the Centre for Evaluation and Research Evidence in the Department of Health. Milbert holds a Masters of Public health from the University of Melbourne and Masters Degree in Demography from the North West University. Throughout her diverse career, Milbert has executed diverse strategic leadership roles within government, academia, and as an international development consultant. Previously, Milbert was the Deputy Director responsible for Population Policy monitoring in the National Department of Social Development in South Africa. Milbert is passionate about evidence based policy reforms and knowledge translation into practice for optimum service delivery.

Desiree Terrill is the Principal Evaluation Officer at the Centre for Evaluation and Research Evidence in the Department of Health. Desiree possesses a Masters in Public Health and is affiliated with Deakin University as an Honorary Fellow since 2014. Desiree has accomplished and executed senior roles in health promotion, health policy and led various reforms within the Department of Health. She has also led and delivered several strategic evaluations for the Victorian Government over the course of her career.

Eleanor Williams is a public policy and evaluation professional and was previously the Director of the Centre for Evaluation and Research Evidence at the Department of Health. She holds a Masters of Public Policy and Management and Masters of Evaluation from the University of Melbourne and has worked in a variety of senior management and executive roles in Government and as a management consultant. Eleanor sits on the board of the Australian Evaluation Society and co-founded the Australian Public Sector Evaluation Network.

References

Anker

Guidotti

R. J.

Orzeszyna

Sapirie

S. A.

Thuriaux

M. C.

(1993). Rapid evaluation methods (REM) of health services performance: Methodological observations. Bulletin of the World Health Organization, 71(1): 15–21.

Annett

Rifkin

S. N.

& World Health Organization . (1995). Guidelines for rapid participatory appraisals to assess community health needs: A focus on health improvements for low-income urban and rural areas. https://apps.who.int/iris/handle/10665/59366

Australian Evaluation Society . (2020, 15 July). Webinar: Rapid evaluation during the COVID-19 response. https://www.aes.asn.au/evaluation-learning/professional-learning-events/pl-events-vic/1001922-old-seminar-rapid-evaluation-during-the-covid-19-response

Bamberger

Rugh

Mabry

(2006). RealWorld evaluation: Working under budget, time, data and political constraints. SAGE.

Beebe

(1995). Basic concepts and techniques of rapid appraisal. Human Organization, 54(1), 42–51. http://www.jstor.org/stable/44126571

Bentley

M. E.

Pelto

G. H.

Straus

W. L.

Schumann

D. A.

Adegbola

De La Pena

Huffman

S. L.

(1988). Rapid ethnographic assessment: Applications in a diarrhea management program. Social Science & Medicine, 27(1), 107–116. https://doi:10.1016/0277-9536(88)90168-2

Better Evaluation .(2020). Analyse data. https://www.betterevaluation.org/en/rainbow_framework/describe/analyse_data

Davidson

E. J.

(2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. SAGE Publications, Inc. https://www.doi.org/10.4135/9781452230115

Davidson

Wehipeihana

McKegg

(2011, 31 August–2 September). The rubric revolution. Paper presented at Australian Evaluation Society conference, Sydney, Australia, September 1, 2011.

10.

Department Health and Human Services . (2020). Maternity and neonatal care during COVID-19. https://www.dhhs.vic.gov.au/health-services-and-general-practitioners-coronavirus-disease-covid-19

11.

Dickinson

Adams

(2017). Values in evaluation – The use of rubrics. Evaluation and Program Planning, 65, 113–116. https://doi.org/10.1016/j.evalprogplan.2017.07.005

12.

Dos Santos Marques

I. C.

Theiss

L. M.

Johnson

C. Y.

McLin

Ruf

B. A.

Vickers

S. M.

Fouad

M. N.

Scarinci

I. C.

Chu

D. I.

(2021). Implementation of virtual focus groups for qualitative data collection in a global pandemic. American Journal of Surgery, 221(5), 918–922. https://doi.org/10.1016/j.amjsurg.2020.10.009

13.

Fernald

D. H.

Duclos

C. W.

(2005). Enhance your team-based qualitative research. Annals of Family Medicine, 3(4), 360–364. https://doi.org/10.1370/afm.290

14.

Frey

(2018). The SAGE encyclopedia of educational research, measurement, and evaluation (Vol. 1–4). SAGE. https://doi.org/10.4135/9781506326139

15.

Hagens

Dobrow

M. J.

Chafe

(2009). Interviewee transcript review: Assessing the impact on qualitative research. BMC Medical Research Methodology, 9(1), 47. https://doi.org/10.1186/1471-2288-9-47

16.

Johnson

G. A.

Vindrola-Padros

(2017). Rapid qualitative research methods during complex health emergencies: A systematic review of the literature. Social Science & Medicine, 189, 63–75. https://doi.org/10.1016/j.socscimed.2017.07.029

17.

Laura

M. H.

Nadia

Marcy

Karl

A. L.

Mary

Steve

Elizabeth

(2020). Adapting rapid assessment procedures for implementation research using a team-based approach to analysis: A case example of patient quality and safety interventions in the ICU. Implementation Science, 15(1), 1–12. https://doi.org/10.1186/s13012-020-0972-5

18.

McCarthy

McMeekin

Haining

Bainbridge

Laing

Gray

(2019). Rapid evaluation for health and social care innovations: Challenges for “quick wins” using interrupted time series. BMC Health Services Research, 19(1), 964. https://doi.org/10.1186/s12913-019-4821-7

19.

McNall

Foster-Fishman

P. G.

(2007). Methods of rapid evaluation, assessment, and appraisal. American Journal of Evaluation, 28 (2), 151–168. https://doi.org/10.1177/1098214007300895

20.

McNall

M. A.

Welch

V. E.

Ruh

K. L.

Mildner

C. A.

Soto

(2004). The use of rapid-feedback evaluation methods to improve the retention rates of an HIV/AIDS healthcare intervention. Evaluation and Program Planning, 27(3), 287. https://doi.org/10.1016/j.evalprogplan.2004.04.003

21.

Neal

J. W.

Neal

Z. P.

VanDyke

Kornbluh

(2015). Expediting the analysis of qualitative data in evaluation: A procedure for the rapid identification of themes from audio recordings (RITA). American Journal of Evaluation, 36(1), 118–132. https://doi.org/10.1177/1098214014536601

22.

Nelson

B.W

Pettitt

Flannery

J.E

Allen

N.B.

(2020). Rapid assessment of psychological and epidemiological correlates of COVID-19 concern, financial strain, and health-related behavior change in a large online sample. PLoS ONE, 15(11), e0241990. https://doi.org/10.1371/journal.pone.0241990

23.

Ore

(2021). How effective is the use of telehealth for children with autism spectrum disorders? International Journal of Psychiatry Research, 4(1), 1–4.

24.

Palinkas

L. A.

Horwitz

S. M.

Green

C. A.

Wisdom

J. P.

Duan

Hoagwood

(2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y

25.

Patton

M.Q.

(2008). Utilization-focused evaluation. (4th ed.). SAGE.

26.

Patton

(2010). Developmental evaluation applying complexity concepts to enhance innovation and use. Guilford Press.

27.

Raine

Fitzpatrick

de Pury

(2016). Challenges, solutions and future directions in evaluative research. Journal of Health Services Research & Policy, 21(4), 215–216. https://doi.org/10.1177/1355819616664495

28.

Robinson

R.S.

(2014). Purposive sampling. In: Michalos

A. C.

(Ed.), Encyclopedia of quality of life and well-being research (pp XX-YY if avaialble). Springer. https://doi.org/10.1007/978-94-007-0753-5_2337

29.

Rodriguez

R. M.

Torres

J. R.

Chang

A. M.

Haggins

A. N.

Eucker

S. A.

O’Laughlin

K. N.

Anderson

Miller

D. G.

Wilkerson

R. G.

Caldwell

Lim

S. C.

Raja

A. S.

Baumann

B. M.

Graterol

Eswaran

Chinnock

& REVVED UP Investigators (2021). The Rapid Evaluation of COVID-19 Vaccination in Emergency Departments for Underserved Patients Study. Annals of Emergency Medicine, 78(4), 502–510. https://doi.org/10.1016/j.annemergmed.2021.05.026

30.

Rogers

(2020). Real-time evaluation. Monitoring and evaluation for adaptive management Working Paper Series, Number 4. www.betterevaluation.org/en/monitoring-and-evaluation-adaptive-management-working-paper-series

31.

Rowe

(2019). Rapid impact evaluation. Evaluation, 25(4), 496–513. https://doi.org/10.1177/1356389019870213

32.

Rubin

H. J.

Rubin

I. S.

(2005). Qualitative interviewing (2nd ed.): The art of hearing data (2nd ed.). SAGE Publications, Inc. https://www.doi.org/10.4135/9781452226651

33.

Skillman

Cross-Barnet

Friedman Singer

Rotondo

Ruiz

Moiduddin

(2019). A framework for rigorous qualitative research as a component of mixed method rapid-cycle evaluation. Qualitative Health Research, 29(2), 279–289. https://doi.org/10.1177/1049732318795675

34.

Smith

A. C.

Thomas

Snoswell

C. L.

Haydon

Mehrotra

Clemensen

Caffery

L. J.

(2020). Telehealth for global emergencies: Implications for coronavirus disease 2019 (COVID-19). Journal of Telemedicine and Telecare, 26(5), 309–313. https://doi.org/10.1177/1357633X20916567

35.

Trotter

R. T.

Needle

R. H.

Goosby

Bates

Singer

(2001). A methodological model for rapid assessment, response, and evaluation: The RARE program in public health. Field Methods, 13(2), 137–159. https://doi.org/10.1177/1525822X0101300202

36.

UNICEF . (2020). Community rapid assessment on COVID-19: Behavioural findings and insights from Round 1 in Kenya, Madagascar and South Sudan”. UNICEF Evaluation Office.

37.

United Nations Children’s Fund . (2016). Evaluation of UNICEF’s response to the Ebola outbreak in West Africa, 2014-2015, UNICEF.

38.

Vindrola-Padros

Chisnall

Cooper

Dowrick

Djellouli

Symmons

S. M.

Martin

Singleton

Vanderslott

Vera

Johnson

G. A.

(2020). Carrying out rapid qualitative research during a pandemic: Emerging lessons from COVID-19. Qualitative Health Research, 30(14), 2192–2204. https://doi.org/10.1177/1049732320951526

39.

Watkins

D. C.

(2017). Rapid and rigorous qualitative data analysis: The “RADaR” technique for applied research. International Journal of Qualitative Methods, 16(1), 1-9. https://doi.org/10.1177/1609406917712131.

40.

World Health Organization . (2020). The impact of COVID-19 on mental, neurological and substance use services: Results of a rapid assessment. World Health Organization.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.24 MB