Problem solving is fundamental to psychoeducational assessment practices and generally grounded in activities related to identifying problems, developing and refining hypotheses, generating solutions, developing and implementing actions, and evaluating outcomes. While the process is central to response-to-intervention practices as well, little research has addressed the form, content, or outcomes of decision-making teams as they operate in schools. One barrier to building a program of research on team problem solving has been the absence of a credible and feasible measure of team performance. We developed the Decision Observation, Recording, and Analysis (DORA) tool to document problem-solving behaviors during team meetings. We were interested in evaluating problem solving during team meetings that focus on academic and behavior concerns in school. We describe the development and preliminary psychometric data for DORA in this article. Our discussion focuses on the implications of DORA for expanding the study of team processes and for improvement of problem-solving practices in schools.
AlgozzineB.HornerR. H.SugaiG.BarrettS.DickeyS. R.EberL.. . . TobinT. (2010). Evaluation blueprint for school-wide positive behavior support. Eugene, OR: National Technical Assistance Center on Positive Behavior Interventions and Support. Retrieved from www.pbis.org
2.
AndersonK. (1994). Making meetings work: How to plan and conduct effective meetings. West Des Moines, IA: American Media.
BaerD. M.WolfM.RisleyT. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91-97.
6.
BarlowD. H.HayesS. C.NelsonR. O. (1984). The scientist practitioner: Research and accountability in clinical and educational settings. New York, NY: Pergamon Press.
7.
BartelsS. M.MortensonB. P. (2005). Enhancing adherence to a problem-solving model for middle-school pre-referral teams: A performance feedback and checklist approach. Journal of Applied School Psychology, 22, 109-123. doi:10.1300/J008v22n01_0610.1300/J008v22n01_06
8.
BerganJ. R.KratochwillT. R. (1990). Behavioral consultation and therapy. New York, NY: Plenum Press.
9.
BoudettK. P.CityE. A.MurnaneR. J. (Eds.) (2006). Data wise: A step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard University Press.
10.
BradfordL. P. (1976). Making meetings work. La Jolla, CA: University Associates.
11.
BransfordJ. D.SteinB. S. (1993). The IDEAL problem solver: A guide for improving thinking, learning, and creativity (2nd ed.) New York, NY: W. H. Freeman.
12.
Brown-ChidseyR.SteegeM. W. (2005). Response to intervention: Principles and strategies for effective practice. New York, NY: Guildford Press.
13.
BurnsM. K.ScholinS. E.KosciolekS.LivingstonJ. (2010). Reliability of decision-making frameworks for response to intervention in reading. Journal of Psychoeducational Assessment, 28, 102-114.
14.
CohenR.KincaidD.ChildsK. (2007). Measuring school-wide positive behavior support implementation: Development and validation of the Benchmarks of Quality. Journal of Positive Behavior Interventions, 9, 203-213.
15.
Council for Exceptional Children. (2003). What every special educator must know: Ethics, standards, and guidelines for special educators (5th ed.). Arlington, VA: Author.
16.
DenoS. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219-232.
17.
DenoS. L. (2005). Problem-solving assessment. In Brown-ChidseyR. (Ed.), Assessment for intervention: A problem-solving approach (pp. 10-40). New York, NY: Guilford.
18.
D’ZurillaT. J.NezuA. M. (1990). Development and preliminary evaluation of the Social Problem-Solving Inventory. Psychological Assessment, 2, 156-163.
19.
D’ZurillaT. J.NezuA. M.Maydeu-OlivaresA. (2002). Social Problem-Solving Inventory—Revised (SPSI—R): Technical manual. North Tonawanda, NY: Multi-Health Systems.
20.
D’ZurillaT. J.NezuA. M.Maydeu-OlivaresA. (2004). Social problem solving: Theory and assessment. In ChangE. C.D’ZurillaT. J.SannaL. J. (Eds.), Social problem solving: Theory, research, and training (pp. 11-27). Washington, DC: American Psychological Association.
21.
FuchsD.DeshlerD. D. (2007). What we need to know about responsiveness to intervention (and shouldn’t be afraid to ask). Learning Disabilities Research & Practice, 22, 129-136.
22.
GallM. D.BorgW. R.GallJ. P. (1996). Educational research: An introduction (6th ed.). White Plains, NY: Longman.
23.
GroszC. A.DensonD. B. (1988). Conducting effective team meetings. In BrossD. C.KrugmanR. D.LenherrM. R.RosenbergD.SchmittB. D.BrossD. C.. . . SchmittB. D. (Eds.), The new child protection team handbook (pp. 287-298). New York: Garland Publishing.
24.
HamiltonL.HalversonR.JacksonS.MandinachE.SupovitzJ.WaymanJ. (2009). Using student achievement data to support instructional decision making (NCEE 2009-4067). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides/
25.
HawkinsR. P.DotsonV. A. (1975). Reliability scores that delude: An Alice in Wonderland trip through the misleading characteristics of interobserver agreement scores in interval recording. In RampE.SembG. (Eds.), Behavior analysis: Areas of research and application (pp. 359-376). Englewood Cliffs, NJ: Prentice-Hall.
26.
HornerR. H.ToddA.Lewis-PalmerT.IrvinL.SugaiG.BolandJ. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Intervention, 6, 3-12.
27.
HospJ. L.ArdoinS. P. (2008). Assessment for instructional planning. Assessment for Effective Intervention, 33, 69-77.
28.
Individuals with Disabilities Education Improvement Act of 2004. (2004). 20 U.S.C. § 1400 et seq. (Reauthorization of the Individuals with Disabilities Education Act of 1990).
29.
JohnstonJ. M.PennypackerH. S. (1993). Strategies and tactics of behavioral research (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
30.
Joint Committee on Standards for Educational and Psychological Testing. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
31.
LawsheC. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563-575.
32.
LencioniP. (2005). Overcoming the five dysfunctions of a team: A field guide for leaders, managers, and facilitators. San Francisco: Jossey-Bass.
33.
LindseyP. (2005). Review of the Social Problem-Solving Inventory-Revised. Mental Measurements Yearbook, 16, 227, 948-950.
34.
MackinD. (2007). The team-building tool kit: Tips and tactics for effective workplace teams (2nd ed.). New York: AMACOM.
35.
National Association of School Psychologists. (2003). Position statement on using large scale assessment for high stakes decisions. Bethesda, MD: Author.
NewtonJ. S.AlgozzineR.AlgozzineK.HornerR.ToddA. W. (2011). Building local capacity for training and coaching data-based problem solving with positive behavior intervention and support teams. Journal of Applied School Psychology, 27, 228-245.
39.
NewtonJ. S.ToddA.HornerR.AlgozzineB.AlgozzineK. M. (2009). Direct Observation, Recording and Analysis. Eugene, OR: University of Oregon, Educational and Community Supports.
40.
NewtonJ. S.HornerR.AlgozzineB.ToddA.AlgozzineK. M. (2009). Using a problem-solving model for data-based decision making in schools. In SailorW.DunlapG.SugaiG.HornerR. (Eds.). Handbook of positive behavior support (pp. 551-580). New York, NY: Springer.
41.
No Child Left Behind Act of 2001, 20 U.S.C. 70 § 6301 et seq. (2002).
42.
OwensG. (2005). Review of the Social Problem-Solving Inventory-Revised. Mental Measurements Yearbook, 16, 227, 950-952.
43.
PerkinsR. D. (2009). How executive coaching can change leader behavior and improve meeting effectiveness: An exploratory study. Consulting Psychology Journal: Practice and Research, 61, 298-318. doi:10.1037/a001784210.1037/a0017842
44.
PfeifferS. (1982). The superiority of team decision making. Exceptional Children, 49, 68-69.
45.
PfeifferS.NaglieriJ. (1983). An investigation of multidisciplinary team decision-making. Journal of Learning Disabilities, 16, 588-590.
46.
ReppA. C.HornerR. H. (Eds.). (1999). Functional analysis of problem behavior: From effective assessment to effective support. Belmont, CA: Wadsworth.
47.
ReschlyD. J.YsseldykeJ. E. (1995). School psychology paradigm shift. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology III (pp. 17–31). Washington, DC: National Association of School Psychologists.
48.
SalviaJ.YsseldykeJ. E.BoltS. (2007). Assessment in special and inclusive education (10th ed.). Boston, MA: Houghton Mifflin.
49.
ShinnM. R. (1989). Curriculum-based measurement: Assessing special children. New York, NY: Guilford.
50.
SugaiG.HornerR. H. (2009). Responsiveness-to-intervention and school-wide positive behavior supports: Integration of multi-tiered system approaches. Exceptionality, 17, 223-237.
51.
SugaiG.HornerR. H.ToddA. W. (2003). The PBIS Self-Assessment Survey (SAS): Version 2.0. Eugene, OR: University of Oregon, Educational and Community Supports. Retrieved from https://www.pbisassessment.org/Evaluation/Surveys
52.
SugaiG.HornerR.AlgozzineB.BarrettS.LewisT.AndersonC.. . . SimonsenB. (2010). Implementation blueprint and self-assessment: Positive behavior interventions and supports. Eugene, OR: OSEP Center on Positive Behavioral Interventions and Supports. Retrieved from http://www.pbis.org/
53.
SugaiG.HornerR.H.Lewis-PalmerT.Rossetto DickeyC. (2011). Team Implementation Checklist (Version 3.1). Eugene, OR: University of Oregon, Educational and Community Supports. Retrieved from http://www.pbis.org/evaluation/evaluation_tools.aspx
54.
TillyW. D. (2008). The evolution of school psychology to a science-based practice: Problem solving and the three-tiered model. In GrimesJ.ThomasA. (Eds.), Best practices in school psychology V (pp. 17-36.). Bethesda, MD: National Association of School Psychologists.
55.
TobiaP. M.BeckerM. C. (1990). Making the most of meeting time. Training and Development Journal, 44, 34-38.
56.
ToddA.HornerR.NewtonJ. S.AlgozzineR.AlgozzineK. M.FrankJ. L (2011). Effects of team-initiated problem solving on decision making by school-wide behavior support teams. Journal of Applied School Psychology, 27, 42-59.
57.
ToddA.NewtonJ. S.HornerR.AlgozzineB.AlgozzineK. M. (2009). Direct observation, recording and analysis training manual. University of Oregon.
58.
VanDerHeydenA. M.WittJ. C.BarnettD. W. (2005). The emergence and possible futures of response to intervention. Journal of Psychoeducational Assessment, 23, 339-361.
59.
YsseldykeJ. E.AlgozzineB.MitchellJ. (1982). Special education team decision making: An analysis of current practice. Personnel and Guidance Journal, 60, 308-313.
60.
YsseldykeJ.BurnsM.DawsonP.KelleyB.MorrisonD.OrtizS.. . . TelzrowC. (2006). School psychology: A blueprint for training and practice III. Bethesda, MD: National Association of School Psychologists.
