In order to evaluate rating behavior among modern music critics, a previously compiled database that randomly sampled 352 different critics' ratings of more than 15,000 albums was utilized. Potential quantitative markers of rating refinement were explored by analyzing frequency distributions of album ratings. As a group, critics' ratings were found to be roughly normally distributed, but individual critics were found to vary widely in rating behavior, as evidenced by both visual inspection of histograms and related statistics, such as skewness, kurtosis, z-score ranges, and Kolmogorov-Smirnov calculations. Precision remains an area in need of improvement among all critics, with the rating scales varying between only 10-20% precision, and ceiling effects with somewhat negatively skewed distributions occurring among some critics.
Get full access to this article
View all access options for this article.
References
1.
AikenL. R., & Groth-MarnatG. (2006). Psychological testing and assessment (12th ed.). Needham Heights, MA: Allyn & Bacon.
2.
AmabileT. M. (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43, 997–1013.
3.
BaerJ.KaufmanJ. C., & GentileC. A. (2004). Extension of the consensual assessment technique to nonparallel creative products. Creativity Research Journal, 16, 113–117.
4.
BirdM. H. (1932). A study in aesthetics. Cambridge, MA: Harvard University Press.
5.
BogdanovV.WoodstraC., & ErlewineS. (Eds.). (2002). All music guide to rock (3rd ed.). San Francisco, CA: Backbeat Books.
6.
BoorM. (1990). Reliability of ratings of movies by professional movie critics. Psychological Reports, 67, 243–257.
7.
BoorM. (1992). Relationships among ratings of motion pictures by viewers and six professional movie critics. Psychological Reports, 70, 1011–1021.
8.
ChatterjeeS., & LehmanR. (1997). Evolution of team sports: A case study for National Basketball Association. Journal of Sport Behavior, 20, 412–423.
9.
ChildI. L. (1964). Development of sensitivity to esthetic values.New Haven, CT: Yale University.
10.
ChristgauR. (1990). Christgau's record guide: The '80s. New York, NY: Perseus Books.
11.
CostaP. T., & McCraeR. R. (1992). NEO PI-R professional manual.Lutz, FL: Psychological Assessment Resources, Inc.
12.
DeCurtisA.HenkeJ., & George-WarrenH. (Eds.). (1992). Rolling Stone album guide (3rd ed.). New York, NY: Random House.
13.
EmbretsonS. E. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
14.
EysenckH. J. (1940/1941). The general factor in aesthetic judgments. British Journal of Psychology, 31, 94–102.
15.
EysenckH. J. (1957). Sense and nonsense in psychology.Baltimore, MD: Penguin.
16.
FarnsworthP. R. (1949). Agreement with the judgments of musicologists as a measure of musical taste. Journal of Psychology, 28, 421–425.
17.
FarnsworthP. R. (1950). Musical taste: Its measurement and cultural nature. Stanford, CA: Stanford University Press.
18.
FarnsworthP. R. (1969). The social psychology of music (2nd ed.). Ames, IA: Iowa State University Press.
19.
GearyR. C. (1947). Testing for normality. Biometrika, 34, 209–242.
20.
GeorgeD., & MalleryP. (2011). SPSS for windows, step by step (11th ed.). Boston, MA: Allyn & Bacon.
21.
GraffG., & DurchholzD. (Eds.). (1999). MusicHound rock: The essential album guide (2nd ed.). Farmington Hills, MI: Visible Ink Press.
22.
HargreavesD. J. (1988). Verbal and behavioral responses to familiar and unfamiliar music. Current Psychological Research and Reviews, 6, 323–330.
23.
HargreavesD. J.MesserschmidtP., & RubertC. (1980). Musical preference and evaluation. Psychology of Music, 8, 13–18.
24.
HickeyM. (2001). An application of Amabile's Consensual Assessment Technique for rating the creativity of children's musical compositions. Journal of Research on Music Education, 49, 234–244.
25.
HolbrookM. B. (1999). Popular appeal versus expert judgments of motion pictures. Journal of Consumer Research, 26, 144–155.
26.
HowellD. C. (2008). Fundamental statistics for the behavioral sciences (6th ed.). Belmont, CA: Thomson Wadsworth.
27.
HumeD. (1757/1965). Of the standard of taste. In On the standard of taste and other essays. Indianapolis: The Bobbs-Merrill Company.
28.
KaufmanJ. C.BaerJ., & ColeJ. C. (2009). Expertise, domains, and the Consensual Assessment Technique. Journal of Creative Behavior, 43, 223–233.
29.
KaufmanJ. C.BaerJ.ColeJ. C., & SextonJ. D. (2008). A comparison of expert and nonexpert raters using the Consensual Assessment Technique. Creativity Research Journal, 20, 171–178.
30.
LundyD. E. (2010). A test of consensus in aesthetic evaluation among professional critics of modern music. Empirical Studies of the Arts, 28, 243–258.
31.
LundyD. E. (2012). Degrees of quality: A method for quantifying aesthetic impact. Psychology Research, 2, 205–221.
32.
McClaveJ. T., & DietrichF. H. (1988). Statistics (4th ed.). San Francisco, CA: Dellen.
33.
MicceriT. (1989). The unicorn, the normal distribution, and other improbable creatures. Psychological Bulletin, 105, 156–166.
34.
NorthA. C., & HargreavesD. J. (1995). Eminence in pop music. Popular Music and Society, 19, 41–66.
35.
NorthA. C., & HargreavesD. J. (1996). Affective and evaluative responses to the arts. Empirical Studies of the Arts, 14, 207–222.
36.
NorthA. C., & HargreavesD. J. (1998). Affective and evaluative responses to pop music. Current Psychology, 17, 102–108.
37.
PluckerJ. A.KaufmanJ. C.TempleJ. S., & QianM. (2009). Do experts and novices evaluate movies the same way?Psychology & Marketing, 26, 470–478.
38.
RentfrowP. J., & GoslingS. D. (2003). The do re mi's of everyday life: The structure and personality correlates of music preferences. Journal of Personality and Social Psychology, 84, 1236–1256.
39.
SimontonD. K. (2004). Film awards as indicators of cinematic creativity and achievement: A quantitative comparison of the Oscars and six alternatives. Creativity Research Journal, 16, 163–172.
40.
SmithL., & SmithJ. (2006). The nature and growth of aesthetic fluency. In LocherP.MartindaleC., & DorfmanL. (Eds.), New directions in aesthetics, creativity and the arts (pp. 47–58). Amityville, NY: Baywood.
41.
SpatzC., & KardasE. P. (2008). Research methods in psychology. New York, NY: McGraw-Hill.
42.
SulsJ.MartinR., & WheelerL. (2000). Three kinds of opinion comparison: The triadic model. Personality and Social Psychology Review, 4, 219–237.
43.
TabachnikB. G., & FidellL. S. (1989). Using multivariate statistics (2nd ed.). Northridge, CA: HarperCollins.
44.
ThompsonD. (2000). Alternative rock. San Francisco, CA: Miller Freeman Books.
45.
WalbergH. J.StrykowskiB. F.RovaiE., & HungS. S. (1984). Exceptional performance. Review of Educational Research, 54, 87–112.
46.
WapnickJ., & EkholmE. (1997). Expert consensus in solo voice performance evaluation. Journal of Voice, 11, 429–436.
47.
WechslerD. (1958). The measurement and appraisal of adult intelligence (4th ed.). Baltimore, MD: Williams & Wilkins.
48.
WeisbardE., & MarksC. (Eds.). (1995). Spin alternative record guide. New York. NY: Vintage.
49.
ZickarM. J., & SlaughterJ. E. (1999). Examining creative performance over time using hierarchical linear modelling: An illustration using film directors. Human Performance, 12, 211–230.
50.
ZipfG. K. (1949). Human behavior and the principle of least effort.Reading, MA: Addison-Wesley.
