Effects of Category Labeling With Emojis on Likert-Type Scales on the Psychometric Properties of Measurements

Abstract

Today, emoji have become a popular option for anchoring the categories of Likert-type scales applied to not only adults but also children. The aim of this study was to investigate the effects of category labeling with emoji by comparing the psychometric properties of the emoji- and verbal-anchored versions of the mathematics motivation scale applied to students aged 8–11 years (grades 3, 4, and 5). The participants of the study comprised 658 students. According to the research results, students used the upper categories of the scale more intensively in the emoji-anchored version than in the verbal-anchored one. Parallel to this, the means calculated for the emoji-anchored version were found to be significantly higher. Moreover, the results of the research revealed that the verbal-anchored scale form yielded more reliable and valid measures than the emoji-anchored version.

Keywords

Likert-type scales emoji children psychometric properties reliability validity

Get full access to this article

View all access options for this article.

References

Adelson

J. L.

McCoach

D. B.

(2010). Measuring the mathematical attitudes of elementary students: The effects of a 4-point or 5-point Likert-type scale. Educational and Psychological Measurement, 70(5), 796-807. http://dx.doi.org/10.1177/0013164410366694.

Alan

Ü.

Atalay Kabasakal

(2020). Effect of number of response options on the psychometric properties of Likert-type scales used with children. Studies in Educational Evaluation, 66, 100895. https://doi.org/10.1016/j.stueduc.2020.100895.

Alismail

Zhang

(2020). Exploring and understanding participants’ perceptions of facial emoji likert scales in online surveys. ACM Transactions on Social Computing, 3(2), 1-12. https://doi.org/10.1145/3382505.

Anastasi

(1976). Psychological testing (4th ed.). Macmillan.

Annamalai

Abdulsalam

S. N.

(2017). Undergraduates’ interpretation on Whatsapp smiley emoji. Jurnal Komunikasi, Malaysian Journal of Communication, 33(4), 89-103. https://doi.org/10.17576/JKMJC-2017-3304-06.

Baghdarnia

Soreh

R. F.

Gorji

(2014). The comparison of two methods of maximum likelihood (ML) and diagonally weighted least squares (DWLS) in testing construct validity of achievement goals. Journal of Educational and Management Studies, 4(1), 22-38.

Bai

Dan

Yang

(2019). A systematic review of emoji: Current research and future perspectives. Frontiers in Psychology, 10(2), 2221. https://doi.org/10.3389/fpsyg.2019.02221.

Bacon

Barlas

F. M.

Dowling

Thomas

R. K.

(2017). How Effective Are Emojis In Surveys Taken on Mobile Devices? Journal of Advertising Research, 57(4), 462-470. https://doi.org/10.2501/jar-2017-053.

Bozgün

Akın Kösterelioğlu

(2020). Reliability and validity study of the elementary school motivation scale’s Turkish form. Anadolu Journal of Educational Sciences International, 10(1), 209-236. https://doi.org/10.18039/ajesi.682030.

10.

Brandwatch (2018). The emoji report. www.brandwatch.com/reports/the-emoji-report/.

11.

Buchanan

Niven

(2002). Validation of a facial image scale to assess child dental anxiety. International Journal of Paediatric Dentistry, 12(1), 47-52.

12.

Cohen

R. J.

Swerdlik

M. E.

(2018). Psychological testing and assessment: An introduction to tests and measurement (9th ed.). McGraw Hill Education.

13.

Corder

G. W.

Foreman

D. I.

(2009). Nonparametric statistics for non-statisticians A step-by-step approach. Wiley.

14.

Crocker

Algina

(1986). Introduction to classical and modern test theory. Harcourt Brace Jovanovich.

15.

De Angeli

Kelly

R. M.

O’Neill

(2020). Beyond happy‐or‐not: Using emoji to capture visitors’ emotional experience. Curator: The Museum Journal, 63(2), 167-191. https://doi.org/10.1111/cura.12352.

16.

Deitz

J. C.

(1989). Reliability. In Milller

L. J.

(Ed.), Developing norm-referenced standardized tests (pp. 125-144). The Haworth.

17.

Derks

Bos

A. E. R.

von Grumbkow

(2008). Emoticons and online message interpretation. Social Science Computer Review, 26(3), 379-388. https://doi.org/10.1177/0894439307311611.

18.

Deubler

Swaney‐Stueve

Jepsen

Su‐Fern

B. P.

(2019). The K‐State emoji scale. Journal of Sensory Studies, 35(1), e12545. https://doi.org/10.1111/joss.12545.

19.

Diedenhofen

Musch

(2016). cocron: A web interface and R package for the statistical comparison of Cronbach’s alpha coefficients. International Journal of Internet Science, 11(1), 51-60.

20.

Domino

M. L.

(2006). Psychological testing: An introduction. Cambridge University.

21.

Ebel

R. L.

Frisbie

D. A.

(1991). Essentials of educational measurement (5th ed.). Prentice-Hall.

22.

Emoji Research Team . (2016). 2016 emoji report. http://cdn.emogi.com/docs/reports/2016_emoji_report.pdf.

23.

Feldman

(2016). Three books every geek should read this summer: The story of emoji. Retrieved from https://www.wsj.com/articles/three-books-every-geek-should-read-this-summer-1466715350.

24.

Feldt

L. S.

Woodruff

D. J.

Salih

F. A.

(1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93-103. https://doi.org/10.1177/014662168701100107.

25.

Field

(2018). Discovering statistics using IBM SPSS statistics (5th ed.). Sage.

26.

Finchilescu

(2004). Measurements. In Tredoux

Durrheim

(Eds.), Numbers, hypotheses & conclusions (pp. 201-229). UCT.

27.

Flora

D. B.

Curran

P. J.

(2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466.

28.

Forero

C. G.

Maydeu-Olivares

Gallardo-Pujol

(2009). Factor analysis with ordinal indicators: A monte carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625-641. https://doi.org/10.1080/10705510903203573.

29.

Gao

VanderLaan

D. P.

(2020). Cultural influences on perceptions of emotions depicted in emojis. Cyberpsychology, Behavior, and Social Networking, 23(8), 567–570. https://doi.org/10.1089/cyber.2020.0024.

30.

Gazeloğlu

Aşan Greenacre

(2020). Comparison of weighted least squares and robust estimation in structural equation modeling of ordinal categorical data with larger sample sizes. Cumhuriyet Science Journal, 41(1), 193-211. https://doi.org/10.17776/csj.648054.

31.

Guay

Marsh

H. W.

Dowson

Larose

(2005, November–1 December). Assessing academic motivation among elementary school children: The elementary school motivation scale (ESMS). In AARE annual conference, Parramatta, Australia. University of Western Sydney.

32.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2010). Multivariate data analysis (7th ed.). Prentice Hall.

33.

Hovermale

(2020). Individual differences in the perception of emoji: Effects of depression and self-esteem. [Unpublished honors thesis, Ball State University Muncie]. https://cardinalscholar.bsu.edu/bitstream/handle/123456789/202534/2020HovermaleOlivia-combined.pdf?sequence=1&isAllowed=y.

34.

İlhan

Güler

(2017). The number of response categories and the reverse scored item problem in likert-type scales: A study with the rasch model. Journal of Measurement and Evaluation in Education and Psychology, 8(3), 321-343. https://doi.org/10.21031/epod.321057.

35.

Jaeger

S. R.

Roigard

C. M.

Jin

Vidal

Ares

(2019). Valence, arousal and sentiment meanings of 33 facial emoji: Insights for the use of emoji in consumer research. Food Research International, 119, 895-907. https://doi.org/10.1016/j.foodres.2018.10.074.

36.

Kaplan

R. M.

Saccuzzo

D. P.

(2018). Psychological testing: Principles, applications, & issues (9th ed.). Cengage Learning.

37.

Koğar

Yılmaz Koğar

(2015). Comparison of different estimation methods for categorical and ordinal data in confirmatory factor analysis. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 351-364. https://doi.org/10.21031/epod.94857.

38.

Kralj Novak

Smailović

Sluban

Mozetič

(2015). Sentiment of emojis. PLoS One, 10(12), e0144296. https://doi.org/10.1371/journal.pone.0144296.

39.

Kılıç

A. F.

Uysal

İ.

Kalkan

(2021). An alternative to likert scale: Emoji. Journal of Measurement and Evaluation in Education and Psychology, 12(2), 182-191. https://doi.org/10.21031/epod.864336.

40.

Leech

N. L.

Barrett

K. C.

Morgan

G. A.

(2005). SPSS for intermediate statistics: Use and interpretation (2nd ed.). Lawrence Erlbaum Associates.

41.

C.-H.

(2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369-387. https://doi.org/10.1037/met0000093.

42.

Lohmann

Pyka

S. S

Zanger

(2017). The effects of smileys on receivers' emotions. Journal of Consumer Marketing, 34(6), 489-495. https://doi.org/10.1108/jcm-02-2017-2120.

43.

Lozano

L. M.

García-Cueto

Muñiz

(2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 4(2), 73-79. http://dx.doi.org/10.1027/1614-2241.4.2.73.

44.

Marengo

Giannotta

Settanni

(2017). Assessing personality using emoji: An exploratory study. Personality and Individual Differences, 112(1), 74-78. https://doi.org/10.1016/j.paid.2017.02.037.

45.

Maydeu-Olivares

(2001). Limited information estimation and testing of thurstonian models for paired comparison data under multiple judgment sampling. Psychometrika, 66(2), 209-227. https://doi.org/10.1007/BF02294836.

46.

McDougald

B. R.

Carpenter

E. D.

Mayhorn

C. B.

(2011). Emoticons: What does this one mean? Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55(1), 1948-1951. https://doi.org/10.1177/1071181311551406.

47.

Miller

M. D.

Linn

R. L.

Gronlund

N. E.

(2009). Measurement and assessment in teaching (10th ed.). Pearson.

48.

Miller

Thebault-Spieker

Chang

Johnson

Terveen

Hecht

(2016). “Blissfully happy” or “ready to fight”: Varying interpretations of emoji. In Proceedings of the 10th international conference on web and social media, (ICWSM 2016) (pp. 259-268). AAAI press.

49.

Mîndrilă

(2010). Maximum likelihood (ML) and diagonally weighted least squares (DWLS) estimation procedures: A comparison of estimation bias with ordinal and multivariate non-normal data. International Journal of Digital Society (IJDS), 1(1), 60-66. https://doi.org/10.20533/ijds.2040.2570.2010.0010.

50.

Mitchell

J. E.

Crosby

R. D.

Wonderlich

Adson

D. E.

(2000). Elements of clinical research in psychiatry. American Psychiatric.

51.

Moussa

(2019). An emoji-based metric for monitoring consumers’ emotions toward brands on social media. Marketing Intelligence & Planning, 37(2), 211-225. https://doi.org/10.1108/mip-07-2018-0257.

52.

Muthén

L. K.

Muthén

(2016). Mplus (Version 7.4). [Computer software].

53.

Oxford Dictionaries . (2015). Word of the year 2015. Retrieved May 19, 2021, from https://languages.oup.com/word-of-the-year/2015/.

54.

Oxford English Dictionary . (2021). emoji, n. OED online. Oxford University Press. Retrieved May 19, 2021, from https://oed.com/view/Entry/389343.

55.

Pallant

(2005). SPSS survival manual (2nd ed.). Allen & Unwin.

56.

Phan

W. M. J.

Amrhein

Rounds

Lewis

(2019). Contextualizing interest scales with emojis: Implications for measurement and validity. Journal of Career Assessment, 27(1), 114-133. https://doi.org/10.1177/1069072717748647.

57.

Post

Ryan

Shepherd

(2017, 21 April) Interpreting emojis: Millennials versus non-millennials. In 16th annual celebration of undergraduate research and creative performance, Holland, Michigan. Retrieved from https://digitalcommons.hope.edu/cgi/viewcontent.cgi?article=1001&context=curcp_16.

58.

Rounds

Phan

W. M. J.

Amrhein

(2016). Examining the efficacy of emoji anchors for the O*NET interest profiler short form. U.S. Department of Labor Employment and Training Administration Office of Workforce Investment Division of National Programs. Tools, & Technical Assistance Washington, DC. Technical Report. https://doi.org/10.13140/RG.2.1.4981.9763.

59.

Royeen

C. B.

(1985). Adaptation of likert scaling for use with children. The Occupational Therapy Journal of Research, 5(1), 59-69. https://doi.org/10.1177/153944928500500104.

60.

Rupar

(2018). The different functions of emojis: The pragmatic use of five emojis in London and New York. Retrieved from http://kau.diva-portal.org/smash/get/diva2:1303122/FULLTEXT01.pdf.

61.

Schermelleh-Engel

Moosbrugger

Müller

(2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.

62.

Schouteten

J. J.

Verwaeren

Lagast

Gellynck

De Steur

(2018). Emoji as a tool for measuring children's emotions when tasting food. Food Quality and Preference, 68, 322-331. https://doi.org/10.1016/j.foodqual.2018.03.005.

63.

Scott

(1997). Children as respondents: Methods for improving data quality. In Lyberg

L.E.

Biemer

Collins

De Leeuw

Dippo

Schwarz

Treman

(Eds.), Survey measurement and process quality. John Wiley & Sons.

64.

Setty

J. V.

Srinivasan

Radhakrishna

Melwani

A. M.

M. K.

(2019). Use of an animated emoji scale as a novel tool for anxiety assessment in children. Journal of Dental Anesthesia and Pain Medicine, 19(4), 227. https://doi.org/10.17245/jdapm.2019.19.4.227.

65.

Shultz

K. S.

Whitney

D. J.

(2005). Measurement theory in action: Case studies and exercises. Sage.

66.

Streiner

D. L.

Norman

G. R

Cairney

(2015). Health measurement scales. A practical guide to their development and use (5th ed.). Oxford University Press.

67.

Sümer

(2000). Structural equation modeling: Basic concepts and applications. Turkish Psychological Articles, 3(6), 49-74.

68.

Swaney-Stueve

Jepsen

Deubler

(2018). The emoji scale: A facial scale for the 21st century. Food Quality and Preference, 68, 183-190. https://doi.org/10.1016/j.foodqual.2018.03.002.

69.

Tabachnick

B. G.

Fidell

L. S.

(2007). Using multivariate statistics (5th ed.). Pearson.

70.

Tigwell

G. W.

Flatla

D. R.

(2016, 6–9 September). “Oh that’s what you meant!”: Reducing emoji misunderstanding. In Proceedings of the 18th international conference on human-computer interaction with mobile devices and services adjunct – MobileHCI, Florence, Italy, 2016. https://doi.org/10.1145/2957265.2961844.

71.

Urbina

(2014). Essentials of psychological testing (2nd ed.). Wiley.

72.

Weijters

Geuens

Baumgartner

(2013). The effect of familiarity with the response category labels on item response to likert scales. Journal of Consumer Research, 40(2), 368-381. https://doi.org/10.1086/670394.

73.

Weiß

Bille

Rodrigues

Hewig

(2020). Age-related differences in emoji evaluation. Experimental Aging Research 46(5), 416-432. https://doi.org/10.1080/0361073X.2020.1790087.

74.

Wright

K. D.

Asmundson

G. J. G.

(2003). Health anxiety in children: Development and psychometric properties of the childhood illness attitude scales. Cognitive Behaviour Therapy, 32(4), 194-202. https://doi.org/10.1080/16506070310014691.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB