Therapist Expertise in Psychotherapy Revisited ψ

Abstract

The thesis of this article is that the lack of evidence related to the identification and development of therapist expertise is due to the inadequate definition and operationalization of the concept. We propose a definition of expertise that is restricted to performance in the conduct of psychotherapy: the manifestation of the highest levels of ability, skill, professional competence, and effectiveness. In addition, we offer several criteria that may be used to assess expertise: performance (including relational and technical expertise), cognitive processing, client outcomes, experience, personal and relational qualities, credentials, reputation, and self-assessment. We then review research related to the development of expertise, highlighting the role of experience with clients, personal therapy, supervision, deliberate practice, and feedback. Finally, we conclude with recommendations for conducting research on therapist expertise.

Keywords

psychotherapy professional issues training

Get full access to this article

View all access options for this article.

References

Ablon

J. S.

Jones

E. E.

(1998). How expert clinicians’ prototypes of an ideal treatment correlate with outcome in psychodynamic and cognitive-behavior therapy. Psychotherapy Research, 8, 71-83. doi:10.1093/ptr/8.1.71

American Psychological Association. (2012). Revised competency benchmarks in professional psychology. Washington, DC: Author. Retrieved from http://www.apa.org/coa

Anderson

Crowley

M. J.

Himawan

Holmberg

Uhlin

(2015). Therapist facilitative interpersonal skills and training status: A randomized clinical trial on alliance and outcome. Psychotherapy Research. Advance online publication. doi:10.1080/10503307.2015.1049671

Anderson

McClintock

A. S.

Himawan

Song

Patterson

C. L.

(2016). A prospective study of therapist facilitative interpersonal skills as a predictor of treatment outcome. Journal of Consulting and Clinical Psychology, 84, 57-66. doi:10.1037/ccp0000060

Anderson

Ogles

B. M.

Patterson

C. L.

Lambert

M. J.

Vermeersch

D. A.

(2009). Therapist effects: Facilitative interpersonal skills as a predictor of therapist success. Journal of Clinical Psychology, 65, 755-768. doi:10.1002/jclp.20583

Austin

E. J.

(2004). An investigation of the relationship between trait emotional intelligence and emotional task performance. Personality and Individual Differences, 36, 1855-1864. doi:10.1016/j.paid.2003.07.006

Bachelor

(1995). Clients’ perception of the therapeutic alliance: A qualitative analysis. Journal of Counseling Psychology, 42, 323-337. doi:10.1037/0022-0167.42.3.323

Baer

R. A.

Smith

G. T.

Hopkins

Krietemeyer

Toney

(2006). Using self-report assessment methods to explore facets of mindfulness. Assessment, 13, 27-45. doi:10.1177/1073191105283504

Baldwin

S. A.

Imel

Z. E.

(2013). Therapist effects: Findings and methods. In Lambert

M. J.

(Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 258-297). Mahwah, NJ: John Wiley.

10.

Baldwin

S. A.

Wampold

B. E.

Imel

Z. E.

(2007). Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75, 842-852. doi:10.1037/0022-006X.75.6.842

11.

Bandura

(1969). Principles of behavior modification. New York, NY: Holt, Rinehart & Winston.

12.

Barber

J. P.

Barrett

M. S.

Gallop

Rynn

M. A.

Rickels

(2012). Short-term dynamic psychotherapy versus pharmacotherapy for major depressive disorder: A randomized, placebo-controlled trial. Journal of Clinical Psychiatry, 73, 66-73. doi:10.4088/JCP.11m06831

13.

Baron-Cohen

Wheelwright

Hill

Raste

Plumb

(2001). The “Reading the Mind in the Eyes” test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry and Allied Disciplines, 42, 241-251. doi:10.1111/1469-7610.00715

14.

Battle

C. G.

Imber

S. D.

Hoehn-Saric

Stone

A. R.

Nash

E. R.

Frank

J. D.

(1965). Target complaints as criterion of improvement. American Journal of Psychotherapy, 20, 184-192.

15.

Beutler

L. E.

Malik

Alimohamed

Harwood

T. M.

Talebi

Noble

Wong

(2004). Therapist variables. In Lambert

M. J.

(Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 227-306). New York, NY: John Wiley.

16.

Boekaerts

(1991). Subjective competence, appraisals, and self-assessment. Elsevier Science, 1, 1-17. doi:10.1016/0959-4752(91)90016-2

17.

Bonanno

G. A.

Burton

C. L.

(2013). Regulatory flexibility: An individual differences perspective on coping and emotion regulation. Perspectives on Psychological Science, 8, 591-612. doi:10.1177/1745691613504116

18.

Bradley

Drapeau

DeStefano

(2012). The relationship between continuing education and perceived competence, professional support, and professional value among clinical psychologists. Journal of Continuing Education in the Health Professions, 32, 31-38. doi:10.1002/chp.21120

19.

Callahan

J. L.

Almstrom

C. M.

Swift

J. K.

Borja

S. E.

Heath

C. J.

(2009). Exploring the contribution of supervisors to intervention outcomes. Training and Education in Professional Psychology, 3, 72-77. doi:10.1037/a0014294

20.

Chi

M. T. H.

Glaser

Farr

M. J.

(1988). The nature of expertise. Hillsdale, NJ: Lawrence Erlbaum.

21.

Chow

D. L.

Miller

S. D.

Seidel

J. A.

Kane

R. T.

Thornton

J. A.

Andrews

W. P.

(2015). The role of deliberate practice in the development of highly effective psychotherapists. Psychotherapy, 52, 337-345. doi:10.1037/pst0000015

22.

Chui

Hill

C. E.

Ain

Ericson

Del Pino

H. G.

Hummel

. . . Spangler

P. T.

(2014). Training undergraduate students to use challenges. The Counseling Psychologist, 42, 758-777. doi:10.1177/0011000014542599

23.

Connolly

Strupp

H. H.

(1996). Cluster analysis of patient reported psychotherapy outcomes. Psychotherapy Research, 6, 30-42. doi:10.1080/10503309612331331558

24.

Cook

C. M.

Saucier

D. M.

(2010). Mental rotation, targeting ability and Baron-Cohen’s empathizing systemizing theory of sex differences. Personality and Individual Differences, 49, 712-716. doi:10.1016/j.paid.2010.06.010

25.

Crits-Christoph

Gibbons

M. B. C.

Mukherjee

(2013). Psychotherapy process-outcome research. In Lambert

M. J.

(Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 298-340). New York, NY: John Wiley.

26.

Crits-Christoph

P. L.

Gibbons

M. B. C.

Ring-Kurtz

Gallop

Stirman

Present

. . . Goldstein

(2008). Changes in positive quality of life over the course of psychotherapy. Psychotherapy: Theory, Research, Practice, Training, 45, 419-430. doi:10.1037/a001440

27.

Crits-Christoph

P. L.

Mintz

(1991). Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 59, 20-26. doi:10.1037/0022-006X59.1.20

28.

Davidson

K. W.

Gidron

Mostofsky

Trudeau

K. J.

(2007). Hospitalization cost offset of a hostility intervention for coronary heart disease patients. Journal of Consulting and Clinical Psychology, 75, 657-662. doi:10.1037/0022-006X.75.4.657

29.

Davis

M. H.

(1983). Measuring individual differences in empathy: Evidence for a multidimensional approach. Journal of Personality and Social Psychology, 44, 113-126. doi:10.1037/0022-3514.44.1.113

30.

Dennhag

Gibbons

M. C.

Barber

J. P.

Gallop

Crits-Christoph

(2012a). Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22, 720-730. doi:10.1080/10503307.2012.716528

31.

Dennhag

Gibbons

M. C.

Barber

J. P.

Gallop

Crits-Christoph

(2012b). How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22, 475-488. doi:10.1080/10503307.2012.674790

32.

Dobson

Shaw

Vallis

(1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24, 295-300. doi:10.1111/j.2044-8260.1985.tb00662.x

33.

Dunning

Health

Suls

J. M.

(2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, 69-106. doi:10.1111/j.1529-1006.2004.00018x

34.

Dunning

Johnson

Ehrlinger

Kruger

(2003). Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 12, 83-87. doi:10.1111/1467-8721.01235

35.

Eells

T. D.

Kendjelic

E. M.

Lucas

C. P.

(1998). What’s in a case formulation? Development and use of a content coding manual. Journal of Psychotherapy Practice and Research, 7, 144-153. doi:10.1080/10503300500268540

36.

Eells

T. D.

Lombart

K. G.

Salsman

Kendjelic

E. M.

Schneiderman

C. T.

Lucas

C. P.

(2011). Expert reasoning in psychotherapy case formulation. Psychotherapy Research, 21, 385-399. doi:10.1080/10503307.2010.539284

37.

Elkin

Falconnier

Martinovich

Mahoney

(2006). Therapist effects in the NIMH Treatment of Depression Collaborative Research Program. Psychotherapy Research, 16, 144-160. doi:10.1177/001100001454.2600

38.

Elkin

Falconnier

Smith

Canada

K. E.

Henderson

Brown

E. R.

McKay

B. M.

(2014). Therapist responsiveness and patient engagement in therapy. Psychotherapy Research, 24, 52-66. doi:10.1080/10503307.2013.820855

39.

Elks

M. A.

Kirkhart

K. E.

(1993). Evaluating effectiveness from the practitioner perspective. Social Work, 38, 554-563.

40.

Elliott

Bohart

A. C.

Watson

J. C.

Greenberg

L. S.

(2011). Empathy. In Norcross

J. C.

(Ed.), Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed., pp. 132-152). New York, NY: Oxford University Press.

41.

Elliott

Hill

C. E.

Stiles

W. B.

Friedlander

M. L.

Mahrer

A. R.

Margison

F. R.

(1987). Primary therapist response modes: Comparison of six rating systems. Journal of Consulting and Clinical Psychology, 55, 218-223. doi:10.1037/0022-006X.55.2.218

42.

Ericsson

K. A.

(2009). Enhancing the development of professional performance: Implications from the study of deliberate practice. In Ericsson

K. A.

(Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 405-431). New York, NY: Cambridge University Press.

43.

Ericsson

K. A.

Charness

Feltovich

P. J.

Hoffman

R. R.

(Eds.). (2006). The Cambridge handbook of expertise and expert development. New York, NY: Cambridge University Press.

44.

Ericsson

K. A.

Krampe

R. T.

Tesch-Romer

(1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406. doi:10.1037/0033-295X.100.3.363

45.

Ericsson

K. A.

Lehmann

A. C.

(1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305. doi:10.1146/annurev.psych.47.1.273

46.

Farber

B. A.

Doolin

E. M.

(2011). Positive regard and affirmation. In Norcross

J. C.

(Ed.), Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed., pp. 168-186). New York, NY: Oxford University Press.

47.

Fatter

D. M.

Hayes

J. A.

(2013). What facilitates countertransference management? The roles of therapist meditation, mindfulness, and self-differentiation. Psychotherapy Research, 23, 502-513. doi:10.1080/10503307.2013.797124

48.

Flowers

J. V.

Booarem

C. D.

(1989). Four studies toward an empirical foundation for group therapy. Journal of Social Service Research, 13, 105-121. doi:10.1300/J079v13n02_07

49.

Flückiger

Del Re

A. C.

Wampold

B. E.

Symonds

Horvath

A. O.

(2012). How central is the alliance in psychotherapy? A multilevel longitudinal meta-analysis. Journal of Counseling Psychology, 59, 10-17. doi:10.1037/a0025749

50.

Freitas

G. J.

(2002). The impact of psychotherapy supervision on client outcome: A critical examination of two decades of research. Psychotherapy: Theory, Research, Practice, Training, 39, 354-367. doi:10.1037/0033-3204.39.4.354

51.

Fuller

Hill

C. E.

(1985). Counselor and helpee perceptions of counselor intentions in relationship to outcome in a single counseling session. Journal of Counseling Psychology, 32, 329-338. doi:10.1037/0022-0167.32.3.329

52.

Gelso

C. J.

(2011). The real relationship in psychotherapy: The hidden foundation of change. Washington, DC: American Psychological Association.

53.

Gelso

C. J.

(2014). A tripartite model of the therapeutic relationship: Theory, research, and practice. Psychotherapy Research, 24, 117-131. doi:10.1080/10503307.2013.845920

54.

Gelso

C. J.

Hayes

J. A.

(1998). The psychotherapy relationship: Theory, research, and practice. New York, NY: John Wiley.

55.

Gelso

C. J.

Hayes

J. A.

(2007). Countertransference and the therapist’s inner experience: Perils and possibilities. Mahwah, NJ: Lawrence Erlbaum.

56.

Goldberg

(1992). The seasoned psychotherapist. New York, NY: Norton.

57.

Goldberg

S. B.

Rousmaniere

Miller

S. D.

Whipple

Nielson

S. L.

Hoyt

W. T.

Wampold

B. E.

(2016). Do psychotherapists improve with time and experience? A longitudinal analysis of outcomes in a clinical setting. Journal of Counseling Psychology, 63, 1-11. doi:10.1037/cou0000131

58.

Green

Barkham

Kellett

Saxon

(2014). Therapist effects and IAPT psychological wellbeing practitioners (PWPs): A multilevel modelling and mixed methods analysis. Behaviour Research and Therapy, 63, 43-54. doi:10.1016/j.brat.2014.08.009

59.

Hamilton

(1967). Development of a rating scale for primary depressive illness. British Journal of Social & Clinical Psychology, 6, 278-296. doi:10.1111/j.2044-8260.1967.tb00530.x

60.

Hannan

Lambert

M. J.

Harmon

Nielsen

S. L.

Smart

D. W.

Shimokawa

Sutton

S. W.

(2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61, 155-163. doi:10.1002/jclp.20108

61.

Hatcher

R. L.

(2015). Interpersonal competencies: Responsiveness, technique, and training in psychotherapy. American Psychologist, 70, 747-757. doi:10.1037/a0039802

62.

Hayes

J. A.

Owen

Bieschke

K. J.

(2015). Therapist differences in symptom change with racial/ethnic minority clients. Psychotherapy, 52, 308-314. doi:10.1037/a0037957

63.

Hill

C. E.

(2005). Therapist techniques, client involvement, and the therapeutic relationship: Inextricably intertwined in the therapy process. Psychotherapy: Theory, Research, Practice, Training, 42, 431-442. doi:10.1037/0033-3204.42.4.431

64.

Hill

C. E.

(2014). Helping skills: Facilitating exploration, insight, and action (4th ed.). Washington, DC: American Psychological Association.

65.

Hill

C. E.

Anderson

Kline

McClintock

Cranston

McCarrick

. . . Gupta

(2016). Helping skills training for undergraduate students: Who should we select and train? The Counseling Psychologist, 44, 55-77. doi:10.1177/0011000015613142

66.

Hill

C. E.

Baumann

Shafran

Gupta

Morrison

Rojas

A. E.

. . . Gelso

C. J.

(2015). Is training effective? A study of counseling psychology doctoral trainees in a psychodynamic/interpersonal training clinic. Journal of Counseling Psychology, 62, 184-201. doi:10.1037/cou0000053

67.

Hill

C. E.

Charles

Reed

K. G.

(1981). A longitudinal analysis of changes in counseling skills during doctoral training in counseling psychology. Journal of Counseling Psychology, 28, 428-436. doi:10.1037/0022-0167.28.5.428

68.

Hill

C. E.

Chui

Baumann

(2013). Revisiting and re-envisioning the outcome problem in psychotherapy: An argument to include individualized and qualitative measurement. Psychotherapy, 50, 68-76. doi:10.1037/a0030571

69.

Hill

C. E.

Gelso

C. J.

Chui

Spangler

Hummel

Huang

. . . Miles

J. R.

(2014). To be or not to be immediate with clients: The use and perceived effects of immediacy in psychodynamic/interpersonal psychotherapy. Psychotherapy Research, 24, 299-315. doi:10.1080/10503307.2013.812262

70.

Hill

C. E.

Knox

(2009). Processing the therapeutic relationship. Psychotherapy Research, 19, 13-29. doi:10.1080/10503300802621206

71.

Hill

C. E.

Knox

(2013). Training and supervision in psychotherapy: Evidence for effective practice. In Lambert

M. J.

(Ed.), Handbook of psychotherapy and behavior change (6th ed., pp. 775-811). New York, NY: John Wiley.

72.

Hill

C. E.

Lambert

M. J.

(2004). Methodological issues in studying psychotherapy processes and outcomes. In Lambert

M. J.

(Ed.), Handbook of psychotherapy and behavior change (5th ed., pp. 84-136). New York, NY: John Wiley.

73.

Hill

C. E.

O’Grady

K. E.

(1985). List of therapist intentions illustrated in a case study and with therapists of varying theoretical orientations. Journal of Counseling Psychology, 32, 3-22. doi:10.1037/0022-0167.32.1.3

74.

Hill

C. E.

Roffman

Stahl

Friedman

Hummel

Wallace

(2008). Helping skills training for undergraduates: Outcomes and predictors of outcomes. Journal of Counseling Psychology, 55, 359-370. doi:10.1037/0022-0167.55.3.359

75.

Hillerbrand

(1989). Cognitive differences between expert and novices: Implications for group supervision. Journal of Counseling and Development, 67, 293-296. doi:10.1002/j.1556-6676.1989.tb02605

76.

Hilsenroth

M. J.

Kivlighan

D. M.

Jr. Slavin-Mulford

(2015). Structured supervision of graduate clinicians in psychodynamic psychotherapy: Alliance and technique. Journal of Counseling Psychology, 62, 173-183. doi:10.1037/cou0000058

77.

Horvath

A. O.

Del Re

A. C.

Flückiger

Symonds

(2011). Alliance in individual psychotherapy. In Norcross

J. C.

(Ed.), Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed., pp. 25-69). New York, NY: Oxford University Press.

78.

Huppert

J. D.

Bufka

L. F.

Barlow

D. H.

Gorman

J. M.

Sear

M. K.

Woods

S. W.

(2001). Therapists, therapist variables, and cognitive behavioral therapy outcomes in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69, 747-755. doi:10.1037/0022-006X.69.5.747

79.

Imel

Z. E.

Baldwin

Atkins

D. C.

Owen

Baardseth

Wampold

B. E.

(2011). Racial/ethnic disparities in therapist effectiveness: A conceptualization and initial study of cultural competence. Journal of Counseling Psychology, 58, 290-298. doi:10.1037/10023284

80.

Jackson

Hill

C. E.

Spangler

P. T.

Ericson

Merson

Liu

. . . Reen

(2014). Training undergraduate students to use interpretation. The Counseling Psychologist, 42, 778-799. doi:10.1177/001100001454.2600

81.

Jackson

Thompson

C. L.

(1971). Effective counselors: Characteristics and attitudes. Journal of Counseling Psychology, 18, 249-254. doi:10.1037/h0030862

82.

Jennings

Skovholt

T. M.

(1999). The cognitive, emotional, and relational characteristics of master therapists. Journal of Counseling Psychology, 46, 3-11. doi:10.1037/0022-167.46.1.3

83.

Jennings

Skovholt

T. M.

(2016). Expertise in counseling & psychotherapy: Master therapist studies from around the world. New York, NY: Oxford University Press.

84.

Kaslow

N. J.

Grus

C. L.

Campbell

L. F.

Fouad

N. A.

Hatcher

R. L.

Rodolfa

E. R.

(2009). Competency assessment toolkit for professional psychology. Training and Education in Professional Psychology, 3(4, suppl.), S27-S45. doi:10.1037/a0015833

85.

Kendjalic

E. M.

Eells

T. D.

(2007). Generic psychotherapy case formulation training improves formulation quality. Psychotherapy: Theory, Research, Training, Practice, 44, 66-77.

86.

Kim

D.-M.

Wampold

B. E.

Bolt

D. M.

(2006). Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Depression Collaborative Research Program data. Psychotherapy Research, 16, 161-172. doi:10.1080/10503300500264911

87.

Kiresuk

T. J.

Smith

Cardillo

J. E.

(1994). Goal attainment scaling: Applications, theory, and measurement. Hillsdale, NJ: Lawrence Erlbaum.

88.

Kivlighan

Hill

C. E.

Baumann

Gelso

C. J.

(2016). Working alliance, real relationship, session quality, and client improvement in psychodynamic psychotherapy: A longitudinal actor partner interdependence model. Journal of Counseling Psychology, 63, 149-161.

89.

Kivlighan

D. M.

Jr. (2008). Structural changes in counselor trainee intention use and clients’ session evaluation. Psychotherapy Research, 18, 560-572. doi:10.1080/10503300802010830

90.

Kivlighan

D. M.

Jr. (2010). Changes in trainees’ intention use and volunteer clients’ evaluations of sessions during early skills training. Psychotherapy: Theory, Research, Practice, Training, 47, 198-210. doi:10.1037/a0019760

91.

Kivlighan

D. M.

Jr. Gelso

C. J.

Ain

Hummel

A. M.

Markin

R. D.

(2014). The therapist, the client, and the real relationship: An actor-partner interdependence analysis of treatment outcome. Journal of Counseling Psychology. Advance online publication. doi:10.1037/cou0000012

92.

Kivlighan

D. M.

Jr. Gelso

C. J.

Ain

Hummel

A. M.

Markin

R. D.

(2015). The therapist, the client, and the real relationship: An actor-partner interdependence analysis of treatment outcome. Journal of Counseling Psychology, 62, 314-320. doi:10.1037/cou0000012

93.

Kivlighan

D. M.

Jr. Kivlighan

D. M.

III . (2009). Training related changes in the ways that group trainees structure their knowledge of group counseling leader interventions. Group Dynamics: Theory, Research, and Practice, 13, 190-204. doi:10.1037/a0015357

94.

Kivlighan

D. M.

Jr. Kivlighan

D. M.

III . (2010). Are group leader knowledge structures related to member satisfaction with the leader? Small Group Research, 41, 175-197. doi:10.1177/1046496409356480

95.

Kivlighan

D. M.

Jr. Markin

R. D.

Stahl

J. V.

Salahuddin

N. M.

(2007). Changes in the ways that group trainees structure their knowledge of group members with training. Group Dynamics: Theory, Research, and Practice, 11, 176-186. doi:10.1037/1089-2699.11.3.176

96.

Kivlighan

D. M.

Jr. Patton

M. J.

Foote

(1998). Moderating effects of client attachment on the counselor experience-working alliance relationship. Journal of Counseling Psychology, 45, 274-278. doi:10.1037/0022-0167.45.3.274

97.

Kivlighan

D. M.

Jr. Quigley

S. T.

(1991). Dimensions used by experienced and novice group therapists to conceptualize group process. Journal of Counseling Psychology, 38, 415-423. doi:10.1037/0022-0167.38.4.415

98.

Knapp

Youngren

J. N.

Van

Creek

Harris

Martin

J. N.

(2013). Assessing and managing risk in psychological practice: An individualized approach (2nd ed.). Rockville, MD: The Trust.

99.

Kolden

G. G.

Klein

M. H.

Wang

C.-C.

Austin

S. B.

(2011). Congruence/genuineness. In Norcross

(Ed.), Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed., pp. 187-202). New York, NY: Oxford University Press.

100.

Kraus

D. R.

Castonguay

Boswell

J. F.

Nordberg

S. S.

Hayes

J. A.

(2011). Therapist effectiveness: Implications for accountability and patient care. Psychotherapy Research, 21, 267-276. doi:10.1080/10503307.2011.563249

101.

Kruger

Dunning

(1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121-1134. doi:10.1037/0022-3514.77.6.1121

102.

Kuprian

Chui

Barber

J. P.

(in press). Effective therapists in psychodynamic therapy for depression: What interventions are used and how? In Castonguay

L. G.

Hill

C. E.

(Eds.), Therapist effects: Toward understanding how and why some therapists are better therapists than others. Washington, DC: American Psychological Association.

103.

Ladany

Inman

A. G.

(2012). Training and supervision. In Altmaier

E. M.

Hansen

J. C.

(Eds.), The Oxford handbook of counseling psychology (pp. 179-207). New York, NY: Oxford University Press.

104.

LaFromboise

T. D.

Coleman

H. L.

Hernandez

(1991). Development and factor structure of the Cross-Cultural Counseling Inventory–Revised. Professional Psychology: Research and Practice, 22, 380-388. doi:10.1037/0735-7028.22.5.380

105.

Lambert

M. J.

(1986). Implications of psychotherapy outcome research for eclectic psychotherapy. In Norcross

J. C.

(Ed.), Handbook of eclectic psychotherapy (pp. 436-462). New York, NY: Brunner/Mazel.

106.

Lambert

M. J.

(1992). Implications of psychotherapy outcome research for psychotherapy integration. In Norcross

J. C.

Goldfried

M. R.

(Eds.), Handbook of psychotherapy integration (pp. 94-129). New York, NY: Basic Books.

107.

Lambert

M. J.

Burlingame

G. L.

Umphress

V. J.

Hansen

N. B.

Vermeersch

Clouse

Yanchar

S. C.

(1996). The reliability and validity of the Outcome Questionnaire. Clinical Psychology and Psychotherapy, 3, 249-258. doi:10.1002/(SICI)1099-0879(199612)3:4<249::AID-CPP106>3.0.CO;2-S

108.

Lambert

M. J.

Shimokawa

(2011). Collecting client feedback. Psychotherapy, 48, 72-79. doi:10.1037/a0022238

109.

Laska

K. M.

Smith

T. L.

Wislocki

A. P.

Minami

Wampold

B. E.

(2013). Uniformity of evidence-based treatments in practice? Therapist effects in the delivery of cognitive processing therapy for PTSD. Journal of Counseling Psychology, 60, 31-41. doi:10.1037/a0031294

110.

Lawrence

E. J.

Shaw

P. P.

Baker

D. D.

Baron-Cohen

S. S.

David

A. S.

(2004). Measuring empathy: Reliability and validity of the Empathy Quotient. Psychological Medicine, 34, 911-919. doi:10.1017/S0033291703001624

111.

Lent

R. W.

Hill

C. E.

Hoffman

M. A.

(2003). Development and validation of the Counselor Activity Self-Efficacy Scales. Journal of Counseling Psychology, 50, 97-108. doi:10.1037/0022-0167.50.1.97

112.

Leon

S. C.

Martinovich

Lutz

Lyons

J. S.

(2005). The effect of therapist experience on psychotherapy outcome. Clinical Psychology and Psychotherapy, 12, 417-426. doi:10.1002/cpp.473

113.

Kivlighan

D. M.

Jr. Gold

P. B.

(2015). Errors of commission and omission in novice group counseling trainees’ knowledge structures of group counseling situations. Journal of Counseling Psychology, 62, 159-172. doi:10.1037/cou0000070

114.

Luborsky

McLellen

Woody

G. E.

O’Brien

Auerbach

(1985). Therapist success and its determinants. Archives of General Psychiatry, 42, 602-611. doi:10.1001/archpsyc.1985.01790290084010

115.

Macnamara

B. N.

Hambrick

D. Z.

Oswald

F. L.

(2014). Deliberate practice and performance in music, games, sports, education, and professions: A meta-analysis. Psychological Science, 25, 1608-1618. doi:10.1177/0956797614535810

116.

Mallinckrodt

Nelson

(1991). Counselor training level and the formation of the psychotherapeutic working alliance. Journal of Counseling Psychology, 38, 133-138. doi:10.1037/0022-0167.38.2.133

117.

Marcus

D. K.

Kashy

D. A.

Wintersteen

M. B.

Diamond

G. S.

(2011). The therapeutic alliance in adolescent substance abuse treatment: A one-with-many analysis. Journal of Counseling Psychology, 58, 449-455. doi:10.1037/a0023196

118.

Martin

Slemon

A. G.

Hiebert

Hallberg

E. T.

Cummings

A. L.

(1989). Conceptualizations of novice and experienced counselors. Journal of Counseling Psychology, 36, 395-400. doi:10.1037/0022-0167.36.4.395

119.

Matarazzo

R. G.

Patterson

(1986). Research on the teaching and learning of therapeutic skills. In Garfield

S. L.

Bergin

A. E.

(Eds.), Handbook of psychotherapy and behavior change (3rd ed., pp. 821-843). New York, NY: John Wiley.

120.

Mathieson

F. M.

Barnfield

Beaumont

(2010). Are we as good as we think we are? Self-assessment versus other forms of assessment of competence in psychotherapy. Cognitive Behaviour Therapist, 2, 43-50. doi:10.1017/S1754470X0800008

121.

Mayfield

W. A.

Kardash

C. M.

Kivlighan

D. M.

Jr. (1999). Differences in experienced and novice counselors’ knowledge structures about clients: Implications for case conceptualization. Journal of Counseling Psychology, 46, 504-514. doi:10.1037/0022-0167.46.4.504

122.

McLeod

(2011). Qualitative research in counseling and psychotherapy (2nd ed.). London, UK: Sage.

123.

Meyer

H. H.

(1980). Self-appraisal of job performance. Personnel Psychology, 33, 291-295. doi:10.1111/j.1744-6570.1980.tb02351.x

124.

Miller

T. M.

Geraci

(2014). Improving metacognitive accuracy: How failing to retrieve practice items reduces overconfidence. Consciousness and Cognition: An International Journal, 29, 131-140. doi:10.1016/j.concog.2014.08.008

125.

Neimeyer

C. J.

Taylor

J. M.

Phillip

(2010). Continuing education in psychology: Patterns of participation and perceived outcomes among mandated and nonmandated psychologists. Professional Psychology: Research and Practice, 41, 435-441. doi:10.1037/a0021120

126.

Norcross

J. C.

(Ed.). (2002). Psychotherapy relationships that work. New York, NY: Oxford University Press.

127.

Norcross

J. C.

(Ed.). (2011). Psychotherapy relationships that work (2nd ed.). New York, NY: Oxford University Press.

128.

Norcross

J. C.

Bike

D. H.

Evans

K. L.

(2009). The therapist’s therapist: A replication and extension 20 years later. Psychotherapy: Theory, Research, Practice, Training, 46, 32-41. doi:10.1037/a0015140

129.

Norcross

J. C.

Strausser

D. J.

Faltus

F. J.

(1988). The therapist’s therapist. American Journal of Psychotherapy, 42, 53-66.

130.

Oddli

H. W.

Halvorsen

M. S.

Ronnestad

M. M.

(2014, November). Expertise demonstrated. What does it mean to be an expert psychotherapist? Retrieved from http://192.185.24.252/~societyp/expertise-demonstrated/

131.

Okiishi

J. C.

Lambert

M. J.

Eggett

Nielson

Dayton

D. D.

Vermeersch

D. A.

(2006). An analysis of therapist treatment effects: Toward providing feedback to individual therapists on their clients’ psychotherapy outcome. Journal of Clinical Psychology, 62, 1157-1172. doi:10.1002/jclp.20272

132.

Okiishi

J. C.

Lambert

M. J.

Nielson

S. L.

Ogles

B. M.

(2003). Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology and Psychotherapy, 10, 361-373. doi:10.1002/cpp.383

133.

Orlinsky

D. E.

Ronnestad

M. H.

(2005). Theoretical integration: Cycles of work and development. In How psychotherapists develop: A study of therapeutic work and professional growth (pp. 161-180). Washington, DC: American Psychological Association. doi:10.1037/11157-011

134.

Owen

Hilsenroth

M. J.

(2014). Treatment adherence: The importance of therapist flexibility in relation to therapy outcomes. Journal of Counseling Psychology, 61, 280-288. doi:10.1037/a0035753

135.

Owen

Wampold

B. E.

Kopta

Rousmaniere

Miller

S. D.

(2016). As good as it gets? Therapy outcomes of trainees over time. Journal of Counseling Psychology, 63, 12-19. doi:10.1037/cou0000112

136.

Owen

Wong

Y. L.

Rodolfa

(2009). Empirical search of psychotherapists’ gender competence in psychotherapy. Psychotherapy: Theory, Research, Practice, Training, 44, 448-458. doi:10.1037/a0017958

137.

Padykula

Horwitz

(2012). Using psychodynamic concepts to measure interpersonal competencies during social work training. International Journal of Applied Psychoanalytic Studies, 9, 48-61. doi:10.1002/aps.312

138.

Parker

Z. J.

Waller

(2015). Factors related to psychotherapists’ self-assessment when treating anxiety and other disorders. Behaviour Research and Therapy, 66, 1-7. doi:10.1016/j.brat.2014.12.010

139.

Ricks

D. F.

(1974). Supershrink: Methods of a therapist judged successful on the basis of adult outcomes of adolescent patients. In Ricks

D. F.

Roff

Thomas

(Eds.), Life history research in psychopathology (pp. 288-308). Minneapolis: University of Minnesota Press.

140.

Ridley

C. R.

(1985). Imperatives for ethnic and cultural relevance in psychology training programs. Professional Psychology: Research and Practice, 16, 611-622. doi:10.1037/0735-7028.16.5.611

141.

Rogers

C. R.

(1957). The necessary and sufficient conditions of therapeutic personality change. Journal of Consulting Psychology, 21, 95-103. doi:10.1037/h0045357

142.

Rounsaville

Chevron

Prusoff

Elkin

Imber

Sotsky

Watkins

(1987). The relation between specific and general dimensions of the psychotherapy process in interpersonal treatment of depression. Journal of Consulting and Clinical Psychology, 55, 379-384. doi:10.1037/0022-006X.55.3.379

143.

Ryan

Safran

J. D.

Doran

J. M.

Moran

J. C.

(2012). Therapist mindfulness, alliance and treatment outcome. Psychotherapy Research, 22, 289-297. doi:10.1080/10503307.2011.650653

144.

Safran

J. D.

Muran

J. C.

Eubanks-Carter

(2011). Repairing alliance ruptures. In Norcross

J. C.

(Ed.), Psychotherapy relationships that work: Evidenced-based responsiveness (2nd ed., pp. 224-238). New York, NY: Oxford University Press.

145.

Schlösser

Dunning

Johnson

K. L.

Kruger

(2013). How unaware are the unskilled? Empirical tests of the “signal extraction” counterexplanation for the Dunning-Kruger effect in self-evaluation of performance. Journal of Economic Psychology, 3, 985-100. doi:10.1016/j.joep.2013.07.004

146.

Schöttke

Flückiger

Goldberg

S. B.

Eversmann

Lange

(2016). Predicting psychotherapy outcome based on therapist interpersonal skills: A five-year longitudinal study of a therapist assessment protocol. Psychotherapy Research. Advance online publication. doi:1.18/10503307.2015.1125546

147.

Shanteau

(1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252-266. doi:10.1016/0749-5978(92)90064-E

148.

Shanteau

Weiss

D. J.

(2014). Individual expertise versus domain expertise. American Psychologist, 69, 711-712. doi:10.1037/a0037874

149.

Shedler

Mayman

Manis

(1993). The illusion of mental health. American Psychologist, 48, 1117-1131. doi:10.1037/0003-066X.48.11.1117

150.

Skovholt

T. M.

Ronnestad

M. H.

Jennings

(1997). Searching for expertise in counseling, psychotherapy, and professional psychology. Educational Psychology Review, 9, 361-369. http://www.jstor.org/stable/23359408

151.

Spangler

P. T.

Hill

C. E.

Dunn

M. G.

Hummel

Walden

Liu

. . . Salahuddin

(2014). Training undergraduate students to use immediacy. The Counseling Psychologist, 42, 729-757. doi:10.1177/001100001454825

152.

Stahl

Hill

C. E.

(2008). A comparison of four methods for assessing natural helping ability. Journal of Community Psychology, 64, 289-298. doi:10.1002/jcop.20195

153.

Stahl

J. V.

Hill

C. E.

Jacobs

Kleinman

Isenberg

Stern

(2009). When the shoe is on the other foot: A qualitative study of intern-level trainees’ perceived learning from clients. Psychotherapy: Theory, Research, Practice, Training, 46, 376-389. doi:10.1037/a0017000

154.

Stein

D. M.

Lambert

M. J.

(1995). Graduate training in psychotherapy: Are therapy outcomes enhanced? Journal of Consulting and Clinical Psychology, 63, 182-196. doi:10.1037/0022-006X.63.2.182

155.

Stiles

W. B.

(2013). The variables problem and progress in psychotherapy research. Psychotherapy, 50, 33-41. doi:10.1037/a0030569

156.

Stiles

W. B.

Hill

C. E.

Elliott

(2015). Looking both ways. Psychotherapy Research, 25, 282-293. doi:10.1080/10503307.2014.981681

157.

Strupp

H. H.

Hadley

S. W.

(1977). A tripartite model of mental health and therapeutic Outcomes: With special reference to negative effects in psychotherapy. American Psychologist, 32, 187-196. doi:10.1037/0003-066X.32.3.187

158.

Swift

J. K.

Greenberg

R. P.

(2012). Premature discontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 80, 547-559. doi:10.1037/a0028226

159.

Tao

K. W.

Owen

Pace

B. T.

Imel

Z. E.

(2015). A meta-analysis of multicultural competencies and psychotherapy process and outcome. Journal of Counseling Psychology, 62, 337-350. doi:10.1037/cou00086

160.

Taylor

(2003). Portrait of the self-enhancer: Well adjusted and well liked or maladjusted and friendless? Journal of Personality and Social Psychology, 84, 165-176. doi:10.1037/0022-3514.84.1.165

161.

Thompson

B. J.

Hill

C. E.

(1993). Client perceptions of therapist competence. Psychotherapy Research, 3, 124-130. doi:10.1037/t)1210-000

162.

Tracey

T. J.

Hays

K. A.

Malone

Herman

(1988). Changes in counselor response as a function of experience. Journal of Counseling Psychology, 35, 119-126. doi:10.1037/0022-0167.35.2.119

163.

Tracey

T. J. G.

Wampold

B. E.

Goodyear

R. K.

Lichtenberg

J. W.

(2015). Improving expertise in psychotherapy. Psychotherapy Bulletin, 50, 7-13.

164.

Tracey

T. J. G.

Wampold

B. E.

Lichtenberg

J. W.

Goodyear

R. K.

(2014). Expertise in psychotherapy: An elusive goal. American Psychologist, 69, 218-229. doi:10.1037/a0035099

165.

Tryon

G. S.

(1985). The engagement quotient: One index of a basic counseling task. Journal of College Student Personnel, 26, 351-354.

166.

Tryon

G. S.

(1989a). A study of engagement of premature termination in a university counseling center. Counselling Psychology Quarterly, 2, 419-429. doi:10.1080/09515078908256693

167.

Tryon

G. S.

(1989b). Study of variables related to client engagement using practicum trainees and experienced clinicians. Psychotherapy, 26, 54-61. doi:10.1037/h0085405

168.

Tryon

G. S.

(2002). Engagement in counseling. In Tryon

G. S.

(Ed.), Counseling based on process research: Applying what we know (pp. 1-26). Boston, MA: Allyn & Bacon.

169.

Tryon

G. S.

Winograd

(2011). Goal consensus and collaboration. In Norcross

(Ed.), Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed., pp. 168-186). New York, NY: Oxford University Press.

170.

Walfish

McAlister

O’Donnell

Lambert

M. J.

(2012). An investigation of self- assessment bias in mental health providers. Psychological Reports, 110, 639-644. doi:10.2466/02.07.17.PR0.110.2.639-644

171.

Wampold

B. E.

Brown

G. S.

(2005). Estimating variability in outcomes attributable to therapists: A naturalistic student of outcomes in managed care. Journal of Consulting and Clinical Psychology, 73, 914-923. doi:10.1037/0022-006X.73.5.914

172.

Webb

C. A.

DeRubeis

R. J.

Barber

J. P.

(2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200-211. doi:10.1037/a0018912

173.

Weck

Rudari

Hilling

Hautzinger

Heidenreich

Schermelleh-Engel

Stangier

(2013). Relapses in recurrent depression 1 year after maintenance cognitive-behavioral therapy: The role of therapist adherence, competence, and the therapeutic alliance. Psychiatry Research, 210, 140-145. doi:10.1016/j.psychres.2013.05.036

174.

Weiss

D. J.

Shanteau

(2014). Who’s the best? A relativistic view of expertise. Applied Cognitive Psychology, 28, 447-457. doi:10.1002/acp.3015

175.

Wicas

E. A.

Mahan

T. W.

(1966). Characteristics of counselors rated effective by supervisors and peers. Counselor Education and Supervision, 6, 50-56. doi:10.1002/j.1556-6978.1966.tb00448.x

176.

Wiggins

J. D.

Weslander

D. L.

(1979). Personality characteristics of counselors rated as effective or ineffective. Journal of Vocational Behavior, 15, 175-185. doi:10.1016/0001-8791(79)90036-8

177.

Williams

Judge

Hill

C. E.

Hoffman

M. A.

(1997). Experiences of novice therapists in prepracticum: Trainees’, clients’, and supervisors’ perceptions of personal reactions and management strategies. Journal of Counseling Psychology, 44, 390-399. doi:10.1037/0022-0167.44.4.390

178.

Yalom

I. D.

(2002). The gift of therapy. New York, NY: HarperCollins.

179.

Yalom

I. D.

Leszcz

(2005). The theory and practice of group psychotherapy (5th ed.). New York, NY: Basic Books.

180.

Zarafonitis-Müller

Kuhr

Bechdolf

(2014). Der Zusammenhang der Therapeutenkompetenz und Adhärenz zum Therapieerfolg in der Kognitiven Verhaltenstherapie—Metaanalytische Ergebnisse [The relationship between therapist’s competence and adherence to outcome in cognitive-behavioral therapy—Results of a meta-analysis]. Fortschritte Der Neurologie, Psychiatrie, 82, 502-510. doi:10.1055/s-0034-1366798