We used this experiment to determine the degree to which cues to difficulty are used to make judgments of difficulty (JODs).
Background:
Traditional approaches involve seeking to standardize the information people used to evaluate subjective workload; however, it is likely that conscious and unconscious cues underlie peoples’ JODs.
Method:
We designed a video game task that tested the degree to which time-on-task, performance-based feedback, and central cues to difficulty informed JODs. These relationships were modeled along five continuous dimensions of difficulty.
Results:
Central cues most strongly contributed to JODs; judgments were supplemented by peripheral cues (performance-based feedback and time-on-task) even though these cues were not always valid. In addition, participants became more likely to rate the task as “easier” over time.
Conclusion:
Although central cues are strong predictors of task difficulty, people confuse task difficulty (central cues), effort allocation and skill (performance-based feedback), and proxy cues to difficulty (time) when making JODs.
Application:
Identifying the functional relationships between cues to difficulty and JODs will provide valuable insight regarding the information that people use to evaluate tasks and to make decisions.
AckermanR. (2014). The diminishing criterion model for metacognitive regulation of time investment. Journal of Experimental Psychology: General, 143, 1349–1368.
2.
AckermanR.ParushA.NassarF.ShtubA. (2016). Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 54, 101–113.
3.
AkaikeH. (1973). Maximum likelihood identification of Gaussian autoregressive moving average models. Biometrika, 60, 255–265.
4.
AltmannE. M.TraftonJ. G. (2007). Timecourse of recovery from task interruption: Data and a model. Psychonomic Bulletin & Review, 14, 1079–1084.
5.
BaumeisterR. F.ShowersC. J. (1986). A review of paradoxical performance effects: Choking under pressure in sports and mental tests. European Journal of Social Psychology, 16, 361–383.
6.
BöckenholtU. (2004). Comparative judgments as an alternative to ratings: Identifying the scale origin. Psychological Methods, 9, 453–465.
7.
BorgG. (1998). Borg’s Perceived Exertion and Pain Scales. Champaign, IL: Human Kinetics.
8.
BrockmyerJ. H.FoxC. M.CurtissK. A.McBroomE.BurkhartK. M.PidruznyJ. N. (2009). The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology, 45, 624–634.
CohenG. L.Purdie-VaughnsV.GarciaJ. (2012). An identity threat perspective on intervention. In InzlichtM.SchmaderT. (Eds.), Stereotype threat: Theory, process, and application (pp. 280–296). New York, NY: Oxford University Press.
11.
CordesR. E. (1993). The relationship between post-task and continuous-vicarious ratings of difficulty. International Journal of Human-Computer Interaction, 5, 115–127.
12.
CovingtonM. V. (2000). Goal theory, motivation, and school achievement: An integrative review. Annual Review of Psychology, 51, 171–200.
13.
DesenderK.Van OpstalF.Van den BusscheE. (2017). Subjective experience of difficulty depends on multiple cues. Scientific Reports, 7, 1–14. doi: 10.1038/srep44222
14.
DunningD.JohnsonK.EhrlingerJ.KrugerJ. (2003). Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 12, 83–87.
15.
FeltovichP. J.HoffmanR. R.WoodsD.RoeslerA. (2004). Keeping it too simple: How the reductive tendency affects cognitive engineering. IEEE Intelligent Systems, 19, 90–94.
16.
FisherO.OysermanD. (2017). Assessing interpretations of experienced ease and difficulty as motivational constructs. Motivation Science, 3, 133–163.
17.
GaethG. J.ShanteauJ. (1984). Reducing the influence of irrelevant information on experienced decision makers. Organizational Behavior and Human Performance, 33, 263–282.
18.
GigerenzerG.ToddP. M.ABC Research Group. (1999). Simple heuristics that make us smart. New York, NY: Oxford University Press.
19.
GonzalezC. (2005). Decision support for real-time, dynamic decision-making tasks. Organizational Behavior and Human Decision Processes, 96(2), 142–154.
20.
GopherD.DonchinE. (1986). Workload: An examination of the concept. In BoffK. R.KaufmanL.ThomasJ. P. (Eds.), Handbook of perception and human performance (pp. 1–49). Oxford, UK: John Wiley.
21.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX: Results of empirical and theoretical research. British Journal of Social Psychology, 1, 139–183.
22.
HighamP. A.ZawadzkaK.HanczakowskiM. (2016). Internal mapping and its impact on measures of absolute and relative metacognitive accuracy. In DunloskyJ.TauberS. K. (Eds.), The Oxford handbook of metamemory (pp. 39–63). New York, NY: Oxford University Press.
23.
HillS. G.IavecchiaH. P.ByersJ. C.BittnerA. C.ZakladeA. L.ChristR. E. (1992). Comparison of four subjective workload rating scales. Human Factors, 34, 429–439.
24.
HullC. L. (1932). The goal-gradient hypothesis and maze learning. Psychological Review, 39, 25–43.
25.
JemstedtA.KubikV.JönssonF. U. (2017). What moderates the accuracy of ease of learning judgments?Metacognition and Learning, 12, 337–355. doi: 10.1007/s11409-017-9172-3
26.
JönssonF. U.LindströmB. R. (2010). Using a multidimensional scaling approach to investigate the underlying basis of ease of learning judgments. Scandinavian Journal of Psychology, 51, 103–108. doi: 10.1111/j.1467-9450.2009.00762.x
27.
KoehlerD. J.BrennerL.GriffinD. (2002). The calibration of expert judgment: Heuristics and biases beyond the laboratory. In GilovichT.GriffinD.KahnemanD. (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 686–715). Cambridge, UK: Cambridge University Press.
28.
KörberM.PraschL.BenglerK. (2018). Why do I have to drive now? Post hoc explanations of takeover requests. Human Factors, 60, 305–323.
29.
KoriatA. (1997). Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126, 349–370.
30.
KoriatA.AckermanR.AdivS.LocklK.SchneiderW. (2014). The effects of goal-driven and data-driven regulation on metacognitive monitoring during learning: A developmental perspective. Journal of Experimental Psychology: General, 143, 386–403.
31.
KoriatA.AdivS. (2016). The self-consistency theory of subjective confidence. In DunloskyJ.TauberS. K. (Eds.), The Oxford handbook of metamemory (pp. 39–63). New York, NY: Oxford University Press.
32.
KoriatA.Ma’ayanH.NussinsonR. (2006). The intricate relationships between monitoring and control in metacognition: Lessons for the cause-and-effect relation between subjective experience and behavior. Journal of Experimental Psychology: General, 135, 36–69.
33.
KrugerJ.DunningD. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121–1134.
34.
KurzbanR.DuckworthA.KableJ. W.MyersJ. (2013). An opportunity cost model of subjective effort and task performance. Behavioral and Brain Sciences, 36, 661–679.
35.
LenthR. (2016). Least-squares means: The R package lsmeans. Journal of Statistical Software, 69, 1–33.
36.
MacnamaraB. N.FrankD. J. (in press). How do task characteristics affect learning and performance? The roles of variably mapped and dynamic tasks. Journal of Experimental Psychology: Learning, Memory, and Cognition. doi: 10.1037/xlm0000487
37.
MansfieldE. R.HelmsB. P. (1982). Detecting multicollinearity. American Statistician, 36, 158–160.
38.
MassarS. A.LimJ.SasmitaK.CheeM. W. (2016). Rewards boost sustained attention through higher effort: A value-based decision making approach. Biological Psychology, 120, 21–27.
39.
McClellandA. G.BolgerF. (1994). The calibration of subjective probabilities: Theories and models 1980–1994. In WrightG.AytonP. (Eds.), Subjective probability (pp. 453–482). Chichester, UK: Wiley.
40.
MetcalfeJ. (1998). Cognitive optimism: Self-deception or memory-based processing heuristics?Personality and Social Psychology Review, 2, 100–110.
41.
MitchellS. (2017). Devaluation of outcomes due to their cost: Extending discounting models beyond delay. In StevensJ. R. (Ed.), Nebraska symposium on motivation: Impulsivity (Vol. 64, pp. 145–161). Berlin, Germany: Springer.
NelsonT. O.LeonesioR. J. (1988). Allocation of self-paced study time and the “labor-in vain effect.”Journal of Experimental Psychology, 14, 676–686.
44.
NisbettR. E.WilsonT. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259.
45.
OrtnerT. M.WeißkopfE.GerstenbergF. X. R. (2013). Skilled but unaware of it: CAT undermines a test taker’s metacognitive competence. European Journal of Psychology of Education, 28, 37–51.
46.
PottsC. A.PastelS.RosenbaumD. A. (2017). How are cognitive and physical difficulty compared?Attention, Perception, & Psychophysics, 80, 500–511.
47.
ReidG. B.NygrenT. E. (1988). The subjective workload assessment technique: A scaling procedure for measuring mental workload. Advances in Psychology, 52, 185–218.
48.
ShanteauJ. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252–266.
49.
SigurdssonS. O.TaylorM. A.WirthO. (2013). Discounting the value of safety: Effects of perceived risk and effort. Journal of Safety Research, 46, 127–134.
50.
SimonH. A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69, 99–118.
51.
StevensS. S. (1957). On the psychophysical law. Psychological Review, 64, 153–181.
52.
SwellerJ. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instructional Design, 4, 295–312.
53.
SwellerJ. (2011). Cognitive load theory. In RossB. H. (Ed.), Psychology of learning and motivation (pp. 37–76). Amsterdam, the Netherlands: Elsevier.
54.
TsangP. S.VelazquezV. L. (1996). Diagnosticity and multidimensional subjective workload ratings. Ergonomics, 39, 358–381.
55.
VallièresB. R.HodgettsH. M.VachonF.TremblayS. (2016). Supporting dynamic change detection: Using the right tool for the task. Cognitive Research: Principles and Implications, 1, 32–52.
56.
VangsnessL.YoungM. E. (2017). The role of difficulty in dynamic risk mitigation decisions. Journal of Dynamic Decision Making, 3, 1–12.
57.
VidulichM. A.TsangP. S. (1987). Absolute magnitude estimation and relative judgement approaches to subjective workload assessment. Proceedings of the Human Factors Society Annual Meeting, 31, 1057–1061.
58.
WickensC. (1992). Engineering psychology and human performance (2nd ed.). New York, NY: Harper Collins.
59.
WickensC. (2014). Effort in human factors performance and decision making. Human Factors, 56, 1329–1336.
60.
YangY.WickensC.ParkT.SiahK.FongL.YinS. (2013). The effect of information access cost and overconfidence bias on junior doctors’ pre-handover performance. In Proceedings of the Human Factors and Ergonomics Society 57th annual meeting (pp. 1755–1759). Santa Monica, CA: Human Factors and Ergonomics Society.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
