The working alliance may be relevant in unguided smartphone-based interventions, but no validated measure exists. We evaluated the psychometric properties of the six-item Digital Working Alliance Inventory (DWAI) using a cross-sectional survey of meditation app users (n = 290) and the intervention arm of a randomized trial testing a smartphone-based meditation app (n = 314). Exploratory factor analysis suggested a single-factor solution which was replicated using longitudinal confirmatory factor analysis. The DWAI showed adequate internal consistency and test–retest reliability. Discriminant validity was supported by a lack of association with social desirability, psychological distress, and preference for a waitlist condition. Convergent validity was supported by positive associations with perceived app effectiveness and preference for an app condition. Supporting predictive validity, DWAI scores positively predicted self-reported and objective app utilization. When assessed at Weeks 3 or 4 of the intervention, but not earlier, DWAI scores predicted pre–post reductions in psychological distress.
AboujaoudeE.SalameW.NaimL. (2015). Telemental health: A status update. World Psychiatry, 14(2), 223-230. https://doi.org/10.1002/wps.20218
2.
AlegríaM.ChatterjiP.WellsK.CaoZ.ChenC.TakeuchiD.JacksonJ.MengX.-L. (2008). Disparity in depression among racial and ethnic minority populations in the United States. Psychiatric Services, 59(11), 1264-1272. https://doi.org/10.1176/ps.2008.59.11.1264
3.
Anderson-LewisC.DarvilleG.MercadoR. E.HowellS.Di MaggioS. (2018). mHealth technology use and implications in historically underserved and minority populations in the United States: Systematic literature review. JMIR mHealth and uHealth, 6(6), Article e128. https://doi.org/10.2196/mhealth.8383
4.
BaumelA.KaneJ. M. (2018). Examining predictors of real-world user engagement with self-guided eHealth interventions: Analysis of mobile apps and websites using a novel dataset. Journal of Medical Internet Research, 20(12), Article e11491. https://doi.org/10.2196/11491
5.
BaumelA.MuenchF.EdanS.KaneJ. M. (2019). Objective user engagement with mental health apps: Systematic search and panel-based usage analysis. Journal of Medical Internet Research, 21(9), Article e14567. https://doi.org/10.2196/14567
6.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57(1), 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
7.
BergerT. (2017). The therapeutic alliance in internet interventions: A narrative review and suggestions for future research. Psychotherapy Research, 27(5), 511-524. https://doi.org/10.1080/10503307.2015.1119908
8.
BerryK.SalterA.MorrisR.JamesS.BucciS. (2018). Assessing therapeutic alliance in the context of mHealth interventions for mental health problems: Development of the mobile Agnew relationship measure (mARM) questionnaire. Journal of Medical Internet Research, 20(4), Article e90. https://doi.org/10.2196/jmir.8252
9.
BordinE. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research & Practice, 16(3), 252-260. https://doi.org/10.1037/h0085885
10.
ChoiS. W.SchaletB.CookK. F.CellaD. (2014). Establishing a common metric for depressive symptoms: Linking the BDI-II, CES-D, and PHQ-9 to PROMIS depression. Psychological Assessment, 26(2), 513-527. https://doi.org/10.1037/a0035768
11.
ChristensenH.GriffithsK. M.FarrerL. (2009). Adherence in internet interventions for anxiety and depression: Systematic review. Journal of Medical Internet Research, 11(2), Article e13. https://doi.org/10.2196/jmir.1194
12.
CohenJ.CohenP.WestS. G.AikenL. S. (2003). Applied multiple regression / correlation analysis for the behavioral sciences (3rd ed.). Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.
13.
CohenJ.TorousJ. (2019). The potential of object-relations theory for improving engagement with health apps. Journal of the American Medical Association, 322(22), 2169-2170. https://doi.org/10.1001/jama.2019.17141
14.
CohenS.WilliamsonG. (1988). Perceived stress in a probability sample of the United States. In SpacapanS.OskampS. (Eds.), The social psychology of health: Claremont Symposium on Applied Social Psychology (pp. 31-67). Sage.
15.
CollinsL. M.MurphyS. A.StrecherV. (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent eHealth interventions. American Journal of Preventive Medicine, 32(5), S112-S118. https://doi.org/10.1016/j.amepre.2007.01.022
16.
CookB. L.ZuvekasS. H.CarsonN.WayneG. F.VesperA.McGuireT. G. (2014). Assessing racial/ethnic disparities in treatment across episodes of mental health care. Health Services Research, 49(1), 206-229. https://doi.org/10.1111/1475-6773.12095
17.
CurranP. J.WestS. G.FinchJ. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16-29.
18.
DahlC. J.Wilson-MendenhallC. D.DavidsonR. J. (2020). The plasticity of well-being: A training-based framework for the cultivation of human flourishing. Proceedings of the National Academy of Sciences, 117(51), 32197-32206. https://doi.org/10.1073/pnas.2014859117
19.
DeRubeisR. J.FeeleyM. (1990). Determinants of change in cognitive therapy for depression. Cognitive Therapy and Research, 14(5), 469-482. https://doi.org/10.1007/BF01172968
20.
EubanksC. F.MuranJ. C.SafranJ. D. (2018). Alliance rupture repair: A meta-analysis. Psychotherapy, 55(4), 508-519. https://doi.org/10.1037/pst0000185
FalkenströmF.GranströmF.HolmqvistR. (2013). Therapeutic alliance predicts symptomatic improvement session by session. Journal of Counseling Psychology, 60(3), 317-328. https://doi.org/10.1037/a0032258
23.
FalkenströmF.HatcherR. L.HolmqvistR. (2015). Confirmatory factor analysis of the patient version of the Working Alliance Inventory–Short form revised. Assessment, 22(5), 581-593. https://doi.org/10.1177/1073191114552472
24.
FalkenströmF.HatcherR. L.SkjulsvikT.LarssonM. H.HolmqvistR. (2015). Development and validation of a 6-item working alliance questionnaire for repeated administrations during psychotherapy. Psychological Assessment, 27(1), 169-183. https://doi.org/10.1037/pas0000038
25.
FirthJ.TorousJ.NicholasJ.CarneyR.PratapA.RosenbaumS.SarrisJ. (2017). The efficacy of smartphone-based mental health interventions for depressive symptoms: A meta-analysis of randomized controlled trials. World Psychiatry, 16(3), 287-298. https://doi.org/10.1002/wps.20472
26.
FirthJ.TorousJ.NicholasJ.CarneyR.RosenbaumS.SarrisJ. (2017). Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. Journal of Affective Disorders, 218(August), 15-22. https://doi.org/10.1016/j.jad.2017.04.046
27.
FlückigerC.Del ReA. C.WampoldB. E.HorvathA.O. (2018). The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy, 55(4), 316-340. https://doi.org/10.1037/pst0000172
28.
FlückigerC.Del ReA. C.WlodaschD.HorvathA. O.SolomonovN.WampoldB. E. (2020). Assessing the alliance–outcome association adjusted for patient characteristics and treatment processes: A meta-analytic summary of direct comparisons. Journal of Counseling Psychology, 67(6), 706-711. https://doi.org/10.1037/cou0000424
29.
GilbodyS.LittlewoodE.HewittC.BrierleyG.TharmanathanP.ArayaR.BarkhamM.BowerP.CooperC.GaskL.KesslerD.LesterH.LovellK.ParryG.RichardsD. A.AndersenP.BrabynS.KnowlesS.ShepherdC.. . . WhiteD. (2015). Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): Large scale pragmatic randomised controlled trial. British Medical Journal, 351, Article h5627. https://doi.org/10.1136/bmj.h5627
30.
GoldbergS. B.FortneyJ. C.ChenJ. A.YoungB. A.LehavotK.SimpsonT. L. (2020). Military service and military health care coverage are associated with reduced racial disparities in time to mental health treatment initiation. Administration and Policy in Mental Health and Mental Health Services Research, 47(4), 555-568. https://doi.org/10.1007/s10488-020-01017-2
HallquistM. N.WileyJ. F. (2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling, 25(4), 621-638. https://doi.org/10.1080/10705511.2017.1402334
33.
HaysR. D.HayashiT.StewartA. L. (1989). A five-item measure of socially desirable response set. Educational and Psychological Measurement, 49(3), 629-636. https://doi.org/10.1177/001316448904900315
HensonP.WisniewskiH.HollisC.KeshavanM.TorousJ. (2019). Digital mental health apps and the therapeutic alliance: Initial review. BJPsych Open, 5(1), Article e15. https://doi.org/10.1192/bjo.2018.86
36.
HerreroR.VaraM.MiragallM.BotellaC.García-PalaciosA.RiperH.KleiboerA.BañosR. M. (2020). Working Alliance Inventory for online interventions-short form (WAI-TECH-SF): The role of the therapeutic alliance between patient and online program in therapeutic outcomes. International Journal of Environmental Research and Public Health, 17(17), Article 6169. https://doi.org/10.3390/ijerph17176169
HorvathA. OGreenbergL. (1989). Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36(2), 223-233. https://doi.org/10.1037/0022-0167.36.2.223
39.
HorvathA. OSymondsB. (1991). Relation between working alliance and outcome in psychotherapy: A meta-analysis. Journal of Counseling Psychology, 38(2), 139-149. https://doi.org/10.1037/0022-0167.38.2.139
40.
KayeL. K.OrbenA.EllisD. A.HunterS. C.HoughtonS. (2020). The conceptual and methodological mayhem of “screen time.”International Journal of Environmental Research and Public Health, 17(10), Article 3661. https://doi.org/10.3390/ijerph17103661
41.
KilukB. D.SerafiniK.FrankforterT.NichC.CarrollK. M. (2014). Only connect: The working alliance in computer-based cognitive behavioral therapy. Behaviour Research and Therapy, 63(December), 139-146. https://doi.org/10.1016/j.brat.2014.10.003
42.
KirschV.KellerF.TutusD.GoldbeckL. (2018). Treatment expectancy, working alliance, and outcome of trauma-focused cognitive behavioral therapy with children and adolescents. Child and Adolescent Psychiatry and Mental Health, 12(1), Article 16. https://doi.org/10.1186/s13034-018-0223-6
43.
LinardonJ.CuijpersP.CarlbringP.MesserM.Fuller-TyszkiewiczM. (2019). The efficacy of app-supported smartphone interventions for mental health problems: A meta-analysis of randomized controlled trials. World Psychiatry, 18(3), 325-336. https://doi.org/10.1002/wps.20673
44.
LinardonJ.Fuller-TyszkiewiczM. (2020). Attrition and adherence in smartphone-delivered interventions for mental health problems: A systematic and meta-analytic review. Journal of Consulting and Clinical Psychology, 88(1), 1-13. https://doi.org/10.1037/ccp0000459
45.
LiuS.YangL.ZhangC.XiangY. T.LiuZ.HuS.ZhangB. (2020). Online mental health services in China during the COVID-19 outbreak. Lancet Psychiatry, 7(4), e17-e18. https://doi.org/10.1016/S2215-0366(20)30077-8
46.
MiloffA.CarlbringP.HamiltonW.AnderssonG.ReuterskiöldL.LindnerP. (2020). Measuring alliance toward embodied virtual therapists in the era of automated treatments with the Virtual Therapist Alliance Scale (VTAS): Development and psychometric evaluation. Journal of Medical Internet Research, 22(3), Article e16660. https://doi.org/10.2196/16660
47.
MiragallM.BañosR. M.CebollaA.BotellaC. (2015). Working Alliance Inventory applied to virtual and augmented reality (WAI-VAR): Psychometrics and therapeutic outcomes. Frontiers in Psychology, 6, Article 1531. https://doi.org/10.3389/fpsyg.2015.01531
OsenbachJ. E.O’BrienK. M.MishkindM.SmolenskiD. J. (2013). Synchronous telehealth technologies in psychotherapy for depression: A meta-analysis. Depression & Anxiety, 30(11), 1058-1067. https://doi.org/10.1002/da.22165
50.
PechorroP.Ayala-NunesL.OliveiraJ. P.NunesC.GoncalvesR. A. (2016). Psychometric properties of the Socially Desirable Response Set-5 among incarcerated male and female juvenile offenders. International Journal of Law and Psychiatry, 49(November–December), 17-21. https://doi.org/10.1016/j.ijlp.2016.05.003
51.
PeerE.BrandimarteL.SamatS.AcquistiA. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70(May), 153-163. https://doi.org/10.1016/j.jesp.2017.01.006
52.
PilkonisP. A.ChoiS. W.ReiseS. P.StoverA. M.RileyW. T.CellaD., & PROMIS Cooperative Group. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS): Depression, anxiety, and anger. Assessment, 18(3), 263-283. https://doi.org/10.1177/1073191111411667
53.
PratapA.NetoE. C.SnyderP.StepnowskyC.ElhadadN.GrantD.MohebbiM. H.MooneyS.SuverC.WilbanksJ.MangraviteL.HeagertyP. J.AreánP.OmbergL. (2020). Indicators of retention in remote digital health studies: A cross-study evaluation of 100,000 participants. npj Digital Medicine, 3(1), 1-10. https://doi.org/10.1038/s41746-020-0224-8
54.
R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
55.
RevelleW. (2020). psych: Procedures for personality and psychological research (R package version 2.0.8) [Computer software]. http://CRAN.R-project.org/package=psych
56.
RobertiJ. W.HarringtonL. N.StorchE. A. (2006). Further psychometric support for the 10-item version of the perceived stress scale. Journal of College Counseling, 9(2), 135-147. https://doi.org/10.1002/j.2161-1882.2006.tb00100.x
57.
SchaletB. D.CookK. F.ChoiS. W.CellaD. (2014). Establishing a common metric for self-reported anxiety: Linking the MASQ, PANAS, and GAD-7 to PROMIS Anxiety. Journal of Anxiety Disorders, 28(1), 88-96. https://doi.org/10.1016/j.janxdis.2013.11.006
58.
TangT. Z.DeRubeisR. J. (1999). Sudden gains and critical sessions in cognitive-behavioral therapy for depression. Journal of Consulting and Clinical Psychology, 67(6), 894-904. https://doi.org/10.1037/0022-006X.67.6.894
59.
TetzlaffB. T.KahnJ. H.GodleyS. H.GodleyM. D.DiamondG. S.FunkR. R. (2005). Working alliance, treatment satisfaction, and patterns of posttreatment use among adolescent substance users. Psychology of Addictive Behaviors, 19(2), 199-207. https://doi.org/10.1037/0893-164X.19.2.199
60.
ThompsonB. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association. https://doi.org/10.1037/10694-000
61.
TorousJ.MyrickK. J.Rauseo-RicuperoN.FirthJ. (2020). Digital mental health and COVID-19: Using technology today to accelerate the curve on access and quality tomorrow. JMIR Mental Health, 7(3), Article e18848. https://doi.org/10.2196/18848
62.
TorousJ.StaplesP.OnnelaJ. P. (2015). Realizing the potential of mobile mental health: New methods for new data in psychiatry. Current Psychiatry Reports, 17(8), Article 61. https://doi.org/10.1007/s11920-015-0602-0
TryonG. S.BlackwellS. C.HammelE. F. (2008). The magnitude of client and therapist working alliance ratings. Psychotherapy: Theory, Research, Practice, Training, 45(4), 546-551.
65.
WasilA. R.GillespieS.PatelR.PetreA.Venturo-ConerlyK. E.ShingletonR. M.WeiszJ. R.DeRubeisR. J. (2020). Reassessing evidence-based content in popular smartphone apps for depression and anxiety: Developing and applying user-adjusted analyses. Journal of Consulting and Clinical Psychology, 88(11), 983-993. https://doi.org/10.1037/ccp0000604
66.
WehmannE.KöhnenM.HärterM.LiebherzS. (2020). Therapeutic alliance in technology-based interventions for the treatment of depression: Systematic review. Journal of Medical Internet Research, 22(6), Article e17195. https://doi.org/10.2196/17195
67.
WeiselK. K.FuhrmannL. M.BerkingM.BaumeisterH.CuijpersP.EbertD. D. (2019). Standalone smartphone apps for mental health: A systematic review and meta-analysis. npj Digital Medicine, 2(1), 1-10. https://doi.org/10.1038/s41746-019-0188-8
68.
ZhouX.SnoswellC. L.HardingL. E.BamblingM.EdirippuligeS.BaiX.SmithA. C. (2020). The role of telehealth in reducing the mental health burden from COVID-19. Telemedicine and e-Health, 26(4), 377-379. https://doi.org/10.1089/tmj.2020.0068
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
