With the push for evidence-informed policy and practice, schools and policy makers are now increasingly encouraged and supported to use and enagage with research evidence. This means that consumers of research will now need to be discerning in judging the quality of research evidence that will inform their decisions. This paper evaluates the quality of evidence behind some well-known education programmes using examples from previous reviews of over 5,000 studies on a range of topics. It shows that much of the evidence is weak, and fundamental flaws in research are not uncommon. This is a serious problem if teaching practices and important policy decisions are made based on such flawed evidence. Lives may be damaged and opportunities missed. The aim of this paper is to show how widespread this problem is and to suggest ways by which the quality of education research may be improved. For example, funders of research and research bodies need to insist on quality research and fund only those that meet the minimum quality criteria. Journal editors and reviewers need to be cognizant of fundamental flaws in research and reject such submissions. One way to do this is to encourage submission of the research design and research protocol prior to acceptance, so acceptance or rejection is based on the design and not on the outcomes. This helps prevent publication bias and biased reporting. Individual researchers can improve quality by making it their moral responsibility to be truthful and transparent.
Albright RE (2012) The impact of music on student achievement in the third and fifth grade math curriculum. 3492175 Ed.D. Thesis, AZ: Northcentral University, USA.
2.
Ayers WE (1993) A study of the effectiveness of expressive writing as a learning enhancement in middle school science. Unpublished dissertation, Temple University, USA.
3.
Baker CN (2011) Relationships between contextual characteristics, parent implementation, and child outcome within an academic preventive intervention for preschoolers. Unpublished PhD Thesis, University of Massachusetts, USA.
4.
BELL (2001) BELL Accelerated Learning Summer Program 2001 evaluation report, Dorchester, MA.
5.
BELL (2002) BELL Accelerated Learning Summer Program 2002 national evaluation report, Dorchester, MA.
6.
BELL (2003) BELL Accelerated Learning Summer Program: 2003 program outcomes, Dorchester, MA.
7.
Bettencourt CL (2009) Promoting social change through writing: A quantitative study of research-based best practices in eighth-grade mathematics. EdD Thesis, Walden University, USA.
8.
BlackPHarrisonCLeeCet al.(2003) Assessment for Learning – Putting it into Practice, Maidenhead: Open University Press.
9.
BlackPWiliamD (1998) Inside the Black Box: Raising Standards through Classroom Assessment, London: GL Assessment.
10.
BormanGDowlingN (2006) Longitudinal Achievement Effects of Multiyear Summer School: Evidence From the Teach Baltimore Randomized Field Trial. Educational Evaluation and Policy Analysis28(1): 25–48.
11.
Bressler RA (2003) Music and Cognitive Abilities: A look at the Mozart Effect. Unpublished PhD Thesis, The Chicago School of Professional Psychology, USA.
12.
Brugar KA (2012) What difference does curricular integration make? An inquiry of fifth graders' learning of history through the use of literacy and visual arts skills. Unpublished PhD Thesis, Michigan State University, USA.
13.
Bullock H, Mountford J and Stanley R (2001) Better Policy-making. London: Cabinet Office, Centre for Management and Policy Studies, 83pp. Available at: http://www.cmps.gov.uk/better_policy_making.pdf. (accessed 3 March 2016).
14.
Cabinet Office (1999) Professional Policy Making for the Twenty First Century. Report by Strategic Policy CentreMaking Team. London: Cabinet Office.
15.
CampbellDTStanleyJC (1963) Experimental and quasi-experimental designs for research on teaching. In: GageNL (ed.) Handbook of Research on Teaching, Chicago, IL: Rand McNally, pp. 171–246.
16.
Chaplin C and Capizzano J (2006) Impacts of a Summer Learning Program: A Random Assignment Study of Building Educated Leaders for Life (BELL). Washington DC: The Urban Institute. http://www.urban.org/UploadedPDF/411350_bell_impacts.pdf. (accessed 29 September 2012).
17.
Clark SJ (2007) The relationship between fine arts participation and the emotional intelligence of fifth-grade elementary students. PhD Thesis, University of Georgia, USA.
18.
CoeRAloisiCHigginsSet al.(2014) What Makes Great Teaching? Review of the Underpinning Research, London: Sutton Trust.
CooperHCharltonKValentineJMuhlenbruckL (2000) Monographs of the Society for Research into Child Development65(1): 1–118.
22.
Costa-GiomiE (1999) The effects of three years of piano instruction on children's cognitive development. Journal of Research in Music Education47(3): 198–212.
23.
CreemersB (1994) The Effective Classroom, London: Cassell.
24.
CrncecRWilsonSJPriorM (2006) No evidence for the Mozart effect in children. Music Perception23(4): 305–317.
25.
Danner SA (2003) Prevention through performance: A creative drama enrichment program for arts school students. Unpublished PhD Thesis, Bowling Green State, USA.
26.
Davies HTO, Nutley SM and Smith PC (eds) (2000) What Works? Evidence-based Policy and Practice in Public Services. Bristol: Policy Press, 380pp.
27.
Department for Education and Employment (DfEE) (1997) White Paper: Excellence in Schools, London: Her Majesty’s Stationery Office.
28.
Department for Education and Employment (DfEE) (2000) Research into teacher effectiveness: A model of teacher effectiveness. Research Report No. 216 by Hay McBer. London: DfEE.
Dong N and Lipsey M (2011) Biases in estimating treatment effects due to attrition in randomised controlled trials: A Simulation study. Society for Research on Educational Effectiveness conference abstract template. http://files.eric.ed.gov/fulltext/ED517992.pdf (accessed 5 October 2016).
FordRMMcDougallSJP (2009) Parent-delivered compensatory education for children at risk of educational failure: Improving the academic and self-regulatory skills of a Sure Start preschool sample. British Journal of Psychology100(4): 773–797.
33.
FosterMEFangGY (2004) Alternatives to handling attrition: An illustration using data from the fast track evaluation. Evaluation Review28: 434–464.
34.
Fountain HLR (2007) Using art to differentiate instruction: An analysis of its effect on creativity and the learning environment. Unpublished PhD Thesis. Purdue University, USA.
35.
FuchsLSFuchsD (1986) Effects of systematic formative evaluation: A meta-analysis. Exceptional Children53: 199–208.
36.
GoldacreB (2013) Building Evidence into Education, London: DfE.
37.
GoldfeldSNapizaNQuachJReillySUkoumuneOCWakeM (2011) Outcomes of a universal shared reading intervention by 2 years of age: The Let's Read trial. Pediatrics127(3): 445–453.
38.
GorardS (2014) A proposal for judging the trustworthiness of research findings. Radical Statistics110: 47–60.
39.
Gorard S (2016) Damaging real lives through obstinacy: Reemphasising why significant testing is wrong. Sociological Research Online, 29 February 2016. http://journals.sagepub.com/doi/abs/10.5153/sro.3857 (accessed 4 August 2017).
40.
Gorard S, Siddiqui N and See BH (2015) How effective is a summer school for catch-up attainment in English and maths? International Journal of Educational Research 73: 1–11.
41.
Gorard S, See BH and Morris B (2016) The most effective approaches to teaching in primary schools: rigorous evidence on effective teaching. Saarbrucken: Lambert Academic Publishing. ISBN 978-3-659-95669-0.
42.
Gorard S, See BH and Siddiqui N (2017) The trials of evidence-based education: The promises, opportunities and problems of trials in education. London: Routledge.
43.
HarlandJKinderKLordPet al.(2000) Arts Education in Secondary Schools: Effects and Effectiveness, Slough: NFER.
44.
HarrisRRatcliffeM (2005) Socio-scientific issues and the quality of exploratory talk – What can be learned from schools involved in a ‘collapsed day’ project?The Curriculum Journal16(4): 439–453.
HattieJTimperleyH (2007) The power of feedback. Review of Educational Research77(1): 81–112.
47.
HetlandLWinnerE (2001) The arts and academic achievement: What the evidence shows. Reviewing Education and the Arts Project; Executive Summary102(5): 3–6.
48.
Hetland L (2000) The relationship between music and spatial processes: A meta-analysis. Unpublished EdD Thesis, Harvard University, USA.
49.
HigginsSKokotsakiDCoeR (2012) The Teaching and Learning Toolkit, London: Education Endowment Foundation.
50.
HopfenbeckTStobartG (2015) Large scale implementation of assessment for learning. Assessment in Education22(1): 1–2.
51.
House of Lords (2014) The case for arts education in schools. House of Lords library note, LLN-2014-037-2. London: House of Lords.
52.
Her Majesty's Stationery Office (HMSO) (2003) Every Child Matters. Report presented by the Chief Secretary to the Treasury by Command of Her Majesty. Cm 5860. Norwich: The Stationery Office.
53.
Hsieh HY (2011) The effect of music on spatial ability. Conference paper. Internationalisation, Design and Global Development 6775: 185–190.
54.
HudziakJJAlbaughMDDucharmeSet al.(2014) Cortical thickness maturation and duration of music training: Health-promoting activities shape brain development. Journal of the American Academy of Child & Adolescent Psychiatry53(11): 1153–1161.
55.
Jacob BA and Lefgren L (2002) Remedial Education and Student Achievement: A Regression-Discontinuity Analysis. National Bureau of Economic Research Working Paper 8918, May.
56.
James CY (2011) Does arts infused instruction make a difference? An exploratory study of the effects of an arts infused instructional approach on engagement and achievement of third, fourth, and fifth grade students in mathematics. PhD Thesis, American University, USA.
57.
JausovecNJausovecKGerlicI (2006) The influence of Mozart’s music on brain activity in the process of learning. Clinical Neurophysiology117(12): 2703–2714.
58.
Joseph A (2014) The Effects of Creative Dramatics on Vocabulary Achievement of Fourth Grade Students in a Language Arts Classroom: An Empirical Study. EdD Thesis, Seattle Pacific University, USA.
59.
Kauffman Best Practices Project (2004) Kauffman Best Practices Project Final Report: Closing the Quality Chasm in Child Abuse Treatment; Identifying and Disseminating Best Practices, Kasas City, Missouri: The Ewing Marion KauffmanFoundation.
60.
Khan M and Gorard S (2012) A randomised controlled trial of the use of a piece of commercial software for the acquisition of reading skills. Educational Review 64(1): 21–36.
61.
KlugerANDeNisiA (1996) The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin119(2): 254–284.
62.
LintsAGadboisS (2003) Is listening to Mozart the only way to enhance spatial reasoning?Perceptual and Motor Skills97(3): 1163–1174.
63.
LittleRJARubinDB (1987) Statistical Analysis with Missing Data, New York, NY: Wiley.
64.
LoniganCWhitehurstG (1998) Relative efficacy of parent and teacher involvement in a sharedreading intervention for preschool children from low-income backgrounds. Early Childhood Research Quarterly13(2): 263–290.
65.
LysakowskiRWalbergH (1982) Instructional effects of cues, participation, and corrective feedback: A quantitative synthesis. American Educational Research Journal19: 559–578.
66.
Martin K, Sharp C and Mehta P (2013) The impact of the summer schools programme on pupils. Research Report. Slough: NFER.
67.
Melhuish E, Belsky J and Leyland A (2010) The impact of Sure Start local programmes on five year olds and their families, DfE Research Report No. RR067. London: DfE.
68.
MorganA (2005) Shared reading interactions between mothers and pre-school children: Case studies of three dyads from a disadvantaged community. Journal of Early Childhood Literacy5(3): 279–304.
69.
NelsonJO’BierneC (2014) Using Evidence in the Classroom: What Works and Why?, Slough: NFER.
70.
Nering ME (2002) The effect of piano and music instruction on intelligence of monozygotic twins. Unpublished PhD Thesis, University of Hawaii, USA.
Ofsted (2008) Assessment for Learning: The impact of National Strategy Support. Ofsted Report reference no. 070244. London: Ofsted.
74.
OlsonCA (2010) Music training causes changes in the brain. Teaching Music17(6): 22–22.
75.
PepplerKAPowellCWThompsonNet al.(2014) Positive impact of arts integration on student academic achievement in English language arts. Educational Forum78(4): 364–377.
76.
Puma MJ, Olsen, RB, Bell SH, and Price C (2009) What to Do When Data Are Missing in Group Randomized Controlled Trials (NCEE 2009-0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
77.
RobinsonAH (2013) Arts integration and the success of disadvantaged students: A research evaluation. Arts Education Policy Review114(4): 191–204.
78.
RoseDSParksMAndroesK (2000) Imagery-based learning: improving elementary students' reading comprehension with drama techniques. Journal of Educational Research94(1): 55–63.
79.
Rossini MD (2005) An investigation of the psychosocial and behavior changes experienced by children participating in the Gallery 37 Connections art program. PhD thesis. Chicago: Loyola University.
80.
Runfola M, Etopio E, Hamlen, K and Rozendal M (2012) Effect of Music Instruction on Preschoolers' Music Achievement and Emergent Literacy Achievement. [Article]. Bulletin of the Council for Research in Music Education, 192: 7–27.
81.
RychetnikLFrommerMHawePet al.(2002) Criteria for evaluating evidence on public health interventions. Journal of Epidemiology & Community Health56(2): 119–127.
82.
See BH and Kokotsaki D (2016) The impact of arts education on children's learning and wider outcomes. Review of Education 4(3): 234–262.
83.
See BH, Gorard S and Siddiqui N (2016) Teachers' use of research evidence in practice? A pilot study of the use of feedback to enhance learning. Educational Research 58(1): 56–72.
84.
See BH and Gorard S (2015) Does intervening to enhance parental involvement in education lead to better academic results for children? An extended review. Journal of Children's Services 10(3): 252–264.
85.
SchacterJJoB (2005) Learning when school is not in session: a reading summer day-camp intervention to improve the achievement of exiting First-Grade students who are economically disadvantaged. Journal of Research in Reading28(2): 158169.
86.
ScheerensJ (1992) Effective Schooling: Research, Theory and Practice, London: Cassell.
87.
Scheerens, J (1999) School effectiveness in developed and developing countries: A review of the research evidence.
88.
Schellenberg G and Hallam S (2005) Music listening and cognitive abilities in 10 and 11 year olds: The blur effect. Conference paper presented in Neurosciences and Music II, Lipzig, Germany.
89.
SchlaugGNortonAOveryKWinnerE (2005) Effects of Music Training on the Child's Brain and Cognitive Development. In: AvanziniGLopezLKoelschSManjnoM (eds) The neurosciences and music II: From perception to performance, New York, US: New York Academy of Sciences, pp. 219–230.
90.
SenéchalMPaganSLevelRet al.(2008) Relations among the frequency of shared reading and 4-year old children’s vocabulary, morphological and syntax comprehension and narrative skills. Early Education and Development19(1): 27–44.
91.
ShadishWRCookTDCampbellDT (2001) Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Boston, MA: Houghton Mifflin.
92.
SimpsonA (2017) The misdirection of public policy: Comparing and combining standardised effect sizes. Journal of Education Policy32(4): 450–466.
93.
SkibaRCaseyACenterB (1985–1986) Nonaversive procedures in the treatment of classroom behavior problems. Journal of Special Education19: 459–481.
Smith H (2011) The effects of a drama-based language intervention on the development of theory of mind and executive function in urban kindergarten children. PhD Thesis, Georgia State University, USA.
96.
SmithEGorardS (2012) ‘Teachers are kind to those who have good marks’: A study of Japanese young people’s views of fairness and equity in school. Compare42(1): 27–46.
97.
SmithrimKUpitisR (2005) Learning through the arts: Lessons of engagement. Canadian Journal of Education28(1/2): 109–127.
98.
Siraj-BlatchfordITaggartB (2014) Exploring Effective Pedagogy in Primary Schools: Evidence from Research, London: Pearson.
99.
Stevens BJ (1996) Parental influences in getting children “ready to learn”. PhD dissertation, Texas Woman’s University. Dissertation Abstracts International Section A. Humanities and Social Sciences 1996; 57(5-A): 121pp.
100.
StraitDLParbery-ClarkAHittnerEet al.(2012) Musical training during early childhood enhances the neural encoding of speech in noise. Brain and Language123(3): 191–201.
101.
SunmonuKLarsonJVan HornYCooper-MartinENielsenJ (2002) Evaluation of the Extended Learning Opportunities Summer Program, Rockville, MD: Office of Shared Accountability, Montgomery County Public Schools.
102.
Terry M (2011) Exploring the additive benefit of parental nurturance training on parent and child shared reading outcome. Unpublished PhD Thesis, Texas A&M University, USA.
103.
ThomasRArnoldA (2011) The A+ schools: A new look at curriculum integration. Visual Arts Research37(72): 96–104.
104.
Thompson DJ (2005) The impact of classical music on the developmental skills of preschool children. Unpublished PhD Thesis, Mississippi State University, USA.
105.
VaughnKWinnerE (2000) SAT scores of students who study the arts: What we can and cannot conclude about the association. The Journal of Aesthetic Education34(3–4): 77–89.
106.
von Rossberg-Gempton IE (1998) Creative dance: Potentiality for enhancing psychomotor, cognitive, and social-affective functioning in seniors and young children. PhD Thesis, Simon Fraser University, Canada.
107.
VoracekM (2008) Wishful thinking: Meta-analysis reveals publication bias, lab differences and nonspecificity in the Mozart effect. International Journal of Psychology43(3–4): 690–690.
108.
Walker PA (2008) The impact of a series of poetry workshops on the cognitive development of middle school students. Unpublished dissertation, University of California, USA.
109.
Yorke-Viney SA (2007) An examination of the effectiveness of arts integration in education on student achievement, creativity, and self perception. PhD Thesis, Marywood University, USA.
