Faculty mentorship is thought to be a linchpin of graduate education in STEM disciplines. This mixed-method study investigates agreement between student mentees’ and their faculty mentors’ perceptions of the students’ developing research knowledge and skills in STEM. We also compare both assessments against independent ratings of the students’ written research proposals. In most cases, students and their mentors identified divergent strengths and weaknesses. However, when mentor-mentee pairs did identify the same characteristics, mentors and mentees disagreed about the mentee’s abilities in 44% of cases in the Fall semester and 75% of cases in the Spring semester. When compared against performance-based assessments of mentees’ work, neither faculty mentors’ nor their mentees’ perceptions aligned with rubric scores at rates greater than chance in most categories.
ÅkerlindG. S. (2005). Variation and commonality in phenomenographic research methods. Higher Education Research and Development, 24, 321–334.
3.
AndersonM. S.SwazeyJ. P. (1998). Reflections on the graduate student experience: An overview. New Directions for Higher Education, 101, 3–13.
4.
AustinA. E. (2002). Preparing the next generation of faculty: Graduate school as socialization to the academic career. The Journal of Higher Education, 73, 94–122.
5.
AustinA. E.McDanielsM. (2006). Preparing the professoriate of the future: Graduate student socialization for faculty roles. Higher Education: Handbook of Theory and Research, 21, 397–456.
6.
BarnesB. J.AustinA. E. (2009). The role of doctoral advisors: A look at advising from the advisor’s perspective. Innovative Higher Education, 33, 297–315.
7.
BaruchY. (1996). Self-performance appraisal vs direct-manager appraisal: A case of congruence. Journal of Managerial Psychology, 11, 50–65.
8.
BishopG. F.TuchfarberA.OldendickR. W. (1986). Opinions on fictitious issues: The pressure to answer survey questions. Public Opinion Quarterly, 50, 240–250.
9.
BlochD. A.KraemerH. C. (1989). 2 × 2 kappa coefficients: Measures of agreement or association. Biometrics, 45, 269–287.
10.
BoyleP.BoiceB. (1998). Best practices for enculturation: Collegiality, mentoring, and structure. New Directions for Higher Education, 101, 87–94.
11.
BronfenbrennerU. (1976). The experimental ecology of education. Educational Researcher, 5, 5–15.
12.
BrownJ. S.CollinsA.DuguidP. (1989). Situated cognition and the culture of learning. Educational Researcher, 18, 35–42.
13.
CaputoD.DunningD. (2005). What you don’t know: The role played by errors of omission in imperfect self-assessments. Journal of Experimental Social Psychology, 41, 488–505.
14.
Carnegie Initiative on the Doctorate. (2001). Overview of doctoral educational studies and reports: 1990-present. Stanford, CA: The Carnegie Foundation for the Advancement of Teaching.
15.
ChanT. F. (2006). A time for a change? The mathematics doctorate. In GoldeC. M.WalkerG. E., & Associate (Eds.), Envisioning the future of doctoral education (pp. 120–134). San Francisco, CA: Jossey-Bass.
16.
CharlesworthM.FarrallL.StokesT.TurnbullD. (1989). Life among the scientists. Melbourne: Oxford University Press.
17.
CicchettiD. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119–135.
18.
ClarkR.HardenS.JohnsonW. (2000). Mentor relationships in clinical psychology doctoral training: Results of a national survey. Teaching of Psychology, 27, 262–268.
19.
CollinsA.BrownJ. S.HolumA. (1991). Cognitive apprenticeship: Making thinking visible. American Educator, 6, 38–46.
20.
ConwayJ. M.HuffcuttA. I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331–360.
21.
CooperW. H. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218–244.
22.
CoxM.AndriotA. (2009). Mentor and undergraduate student comparisons of students’ research skills. Journal of STEM Education, 10, 31–39.
23.
CreswellJ. W.Plano ClarkV. L.GutmannM. L.HansonW. E. (2003). Advanced mixed methods research designs. In TashakkoriA.TeddlieC. (Eds.), Handbook of mixed methods in social & behavioral research (pp. 209–240). Thousand Oaks, CA: Sage Publications.
24.
CummingJ. (2009). The doctoral experience in science: Challenging the current orthodoxy. British Educational Research Journal, 35, 877–890.
25.
DelamontS.AtkinsonP. (2001). Doctoring uncertainty: Mastering craft knowledge. Social Studies of Science, 31, 87–107.
26.
DelamontS.AtkinsonP.ParryO. (2000). The doctoral experience: Success and failure in graduate school. London: Falmer Press.
27.
DenicoloP. (2003). Assessing the PhD: A constructive view of criteria. Quality Assurance in Education, 11, 84–91.
28.
DunbarK. (1997). How scientists think: On-line creativity and conceptual change in science. In WardT. B.SmithS. M. (Eds.), Creative thought: An investigation of conceptual structures and processes (pp. 461–493). Washington, DC: American Psychological Association.
29.
DunbarK. (2000). How scientists think in the real world: Implications for science education. Journal of Applied Developmental Psychology, 21, 49–58.
30.
DunningD.HeathC.SulsJ. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, 69–106.
31.
DunningD.JohnsonK.EhrlingerJ.KrigerJ. (2003). Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 12, 83–87.
32.
EhrlingerJ.DunningD. (2003). How chronic self-views influence (and potentially mislead) estimates of performance. Journal of Personality and Social Psychology, 84, 5–17.
33.
FalchikovN.BoudD. (1989). Student self-assessment in higher education: A meta-analysis. Review of Educational Research, 59, 395–430.
34.
FalchikovN.GoldfinchJ. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70, 287–322.
35.
FeldonD. F. (2007). Cognitive load and classroom teaching: The double-edged sword of automaticity. Educational Psychologist, 42, 123–137.
36.
FeldonD. F. (2010). Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy. Instructional Science, 38, 395–415.
37.
FeldonD. F.MaherM.TimmermanB. (2010). A call for performance-based data in the study of STEM graduate education. Science, 329, 282–283.
38.
FeldonD. F.PeughJ.TimmermanB. E.MaherM. A.HurstM.StricklandD.GilmoreJ. A.StiegelmeyerC. (2011). Graduate students’ teaching experiences improve their methodological research skills. Science, 333, 1037–1039.
39.
FeldonD. F.TimmermanB. E.StoweK.ShowmanR. (2010). Translating expertise into effective instruction: The impacts of cognitive task analysis (CTA) on lab report quality and student retention in the biological sciences. Journal of Research in Science Teaching, 47, 1165–1185.
40.
FleissJ. L.CohenJ. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619.
41.
FrankeA.ArvidssonB. (2011). Research supervisors’ different ways of experiencing supervision of doctoral students. Studies in Higher Education, 36, 7–19.
42.
GardnerS. K. (2007). “I heard it through the grapevine”: Doctoral student socialization in chemistry and history. Higher Education, 54, 723–740.
43.
GardnerS. K. (2008). “What’s too much and what’s too little?”: The process of becoming an independent researcher in doctoral education. Journal of Higher Education, 79, 326–350.
44.
GardnerS. K. (2009). Conceptualizing success in doctoral education: Perspectives of faculty in seven disciplines. Review of Higher Education, 32, 383–406.
45.
GilmoreJ.StricklandD.TimmermanB.MaherM.FeldonD. F. (2010). Weeds in the flower garden: An exploration of plagiarism in graduate students’ research proposals and its connection to enculturation, ESL, and contextual factors. International Journal of Educational Integrity, 6, 13–28.
46.
GlaserB. G. (1965). The constant comparative method of qualitative analysis. Social Problems, 12, 436–445.
47.
GoldeC. M. (1998). Beginning graduate school: Explaining first-year doctoral attrition. New Directions for Higher Education, 101, 55–64.
48.
GoldeC. M.Conklin BueschelA.JonesL.WalkerG. E. (2009). Advocating apprenticeship and intellectual community: Lessons from the Carnegie Initiative on the Doctorate. In EhrenbergR. G.KuhC. V. (Eds.), Doctoral education and faculty of the future (pp. 53–64). Ithaca, NY: Cornell University Press.
49.
GoldeC. M.DoreT. (2001). At cross purposes: What the experiences of doctoral students reveal about doctoral education. Philadelphia, PA: The Pew Charitable Trusts.
50.
GoldeC. M.WalkerG. E. (Eds.). (2006). Envisioning the future of doctoral education: Preparing stewards of the discipline. San Francisco, CA: Jossey-Bass.
51.
GoldsteinS. D.LindemanB.Colbert-GetzJ.ArbellaT.DudasR.LidorA.SacksB. (2014). Faculty and resident evaluations of medical students on a surgery clerkship correlate poorly with standardized exam scores. The American Journal of Surgery, 207, 231–235.
52.
GreenS. G.BauerT. N. (1995). Supervisory mentoring by advisers: Relationships with doctoral student potential, productivity, and commitment. Personnel Psychology, 48, 537–562.
53.
GreeneJ. C.CaracelliV. J.GrahamW. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274.
54.
GreenwaldA. G.BanajiM. R. (1995). Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102, 4–27.
55.
HaagaD. A. F. (1993). Peer review of term papers in graduate psychology courses. Teaching of Psychology, 20, 28–32.
56.
HalonenJ. S.BosackT.ClayS.McCarthyM.DunnD. S.HillG. W.IV. . . . WhitlockK.IV. (2003). A rubric for learning, teaching, and assessing scientific inquiry in psychology. Teaching of Psychology, 30, 196–208.
57.
HarrisM. M.SchaubroeckJ. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41, 43–62.
58.
HawleyP. (1993). Being bright is not enough: The unwritten rules of doctoral studies. Springfield, IL: Charles C. Thomas.
59.
HindsP. J. (1999). The curse of expertise: The effects of expertise and debiasing methods on predictions of novice performance. Journal of Experimental Psychology: Applied, 5, 205–221.
60.
HindsP.PattersonM.PfefferJ. (2001). Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of Applied Psychology, 86, 1232–1243.
61.
JohnsonL.LeeA.GreenB. (2000). The Ph.D. and the autonomous self: Gender, rationality, and postgraduate pedagogy. Studies in Higher Education, 25, 135–147.
62.
JohnsonR. L.PennyJ.GordonB. (2009). Assessing performance: Designing, scoring, and validating performance tasks. New York, NY: Guilford Press.
63.
JohnsonR. L.PennyJ.GordonB.ShumateS. R.FisherS. P. (2005). Resolving score differences in the rating of writing samples: Does discussion improve the accuracy of scores?Language Assessment Quarterly: An International Journal, 2, 117–146.
64.
JohnsonW. (2002). The intentional mentor: strategies and guidelines for the practice of mentoring. Professional Psychology: Research and Practice, 33, 88–96.
65.
KamB. H. (1997). Style and quality in research supervision: The supervisor dependency factor. Higher Education, 34, 81–103.
66.
KardashC. (2000). Evaluation of an undergraduate research experiences: perceptions of undergraduate interns and their faculty mentors. Journal of Educational Psychology, 92, 191–201.
67.
KellyG. J.TakaoA. (2002). Epistemic levels in argument: An analysis of university oceanography students’ use of evidence in writing. Science Education, 86, 314–342.
68.
Knorr-CetinaK. (1999). Epistemic cultures: How the sciences make knowledge. Cambridge, MA: Harvard University Press.
69.
KrosnickJ. A.HolbrookA. L.BerentM. K.CarsonR. T.HannemanW. M.KoppR. J.. . . ConawayM. (2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice?Public Opinion Quarterly, 66, 371–403.
70.
LagowskiJ.VickJ. (1995). Faculty as mentors. New Direction for Teaching and Learning, 62, 79–85.
71.
LarsonJ. R.Jr. (1989). The dynamic interplay between employees’ feedback-seeking strategies and supervisors’ delivery of performance feedback. Academy of Management Review, 14, 408–422.
72.
LovittsB. (2001). Leaving the ivory tower: The causes and consequences of departure from doctoral study. Lanham, MD: Rowman & Littlefield.
73.
LovittsB. E. (2007). Making the implicit explicit: Creating performance expectations for the dissertation. Sterling, VA: Stylus Publishing.
74.
LovittsB. E. (2008). The transition to independent research: Who makes it, who doesn’t, and why. Journal of Higher Education, 79, 296–325.
75.
MacraeC. N.BodenhausenG. V.MilneA. B.WheelerV. (1996). On resisting the temptation for simplification: Counterintentional effects of stereotype suppression on social memory. Social Cognition, 14, 1–20.
76.
MaherM.GilmoreJ. A.FeldonD. F.DavisT. (2014, April). Doctoral student mentoring and the role of cognitive apprenticeship. Paper presented at the 2014 meeting of the American Educational Research Association, Philadelphia, PA.
77.
MarshH. W.BallS. (1989). The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education, 57, 151–169.
78.
MarshH. W.BazeleyP. (1999). Multiple evaluations of grant proposals by independent assessors: Confirmatory factor analysis evaluations of reliability, validity, and structure. Multivariate Behavioral Research, 34, 1–30.
79.
MartonF.PongW. Y. (2005). On the unit of description in phenomenography. Higher Education Research & Development, 24, 335–348.
80.
MertonR. K. (1973). The sociology of science. Chicago, IL: University of Chicago Press.
81.
MilemJ. F.BergerJ. B.DeyE. L. (2000). Faculty time allocation: A study of change over twenty years. Journal of Higher Education, 71, 545–475.
82.
NettlesM. T.MillettC. M. (2006). Three magic letters: Getting to Ph.D. Baltimore, MD: John Hopkins University Press.
83.
NickersonR. S. (1999). How we know—and sometimes misjudge—what others know: Imputing one’s own knowledge to others. Psychological Bulletin, 125, 737-159.
84.
NorciniJ. (2003). Peer assessment of competence. Medical Education, 37, 539–543.
85.
PaglisL.GreenS.BauerT. (2006). Does mentor mentoring add value? A longitudinal study of mentoring and doctoral student outcomes. Research in Higher Education, 47, 451–476.
86.
ParryS. (2007). Disciplines and the doctorate. Dordrecht, the Netherlands: Springer.
87.
PearsonM.BrewA. (2002). Research training and supervision development. Studies in Higher Education, 27, 135–150.
88.
PetersM. A.OlssenM. (2005). Useful knowledge: Redefining research and teaching in the learning economy. In BarnettR. (Ed.), Reshaping the university: New relationships between research, scholarship and teaching (pp. 37–47). Buckingham: Society for Research into Higher Education/Open University Press.
89.
PettyR. E.FlemingM. A.FabrigarL. R. (1999). The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25, 188–203.
90.
PoleC. (2000). Technicians and scholars in pursuit of the PhD: Some reflections on doctoral study. Research Papers in Education, 15, 95–111.
91.
PoleC. J.SprokkereefA.BurgessR. G.LakinE. (1997). Supervision of doctoral students in the natural sciences: Expectations and experiences. Assessment and Evaluation in Higher Education, 22, 49–63.
92.
RemlerD. K.PemaE. (2009). Why do institutions of higher education reward research while selling education? (NBER Working Paper No. 14974). Cambridge, MA: National Bureau of Economic Research.
93.
SchwarzN. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 95–105.
94.
SlomanS. A. (2002). Two systems of reasoning. In GilovichT.GriffinD.KahnemanD. (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 379–396). New York, NY: Cambridge University Press.
95.
StajkovicA. D.LuthansF. (1998). Self-efficacy and work-related performance: A meta-analysis. Psychological Bulletin, 124, 240–261.
96.
SvenssonL. (1997). Theoretical foundations of phenomenography. Higher Education Research & Development, 16, 159–171.
97.
TariqV. N.StefanL. A. J.ButcherA. C.HeylingsD. J. A. (1998). Developing a new approach to the assessment of project work. Assessment & Evaluation in Higher Education, 23, 221–240.
98.
ThorntonG. C. (1968). The relationship between supervisory- and self-appraisals of executive performance. Personnel Psychology, 21, 441–455.
99.
TimmermanB. E. (2008). Peer review in an undergraduate biology curriculum: Effects on students’ scientific reasoning, writing, and attitudes (Unpublished doctoral dissertation). Curtin University of Technology, Perth, Australia.
100.
TimmermanB.FeldonD. F.MaherM.StricklandD.GilmoreJ. A. (2013). Performance-based assessment of graduate student research skills: Timing, trajectory, and potential thresholds. Studies in Higher Education, 38, 693–710.
101.
TimmermanB. E.JohnsonR. L.PayneJ. (2007, April). Development of a universal rubric for assessing students’ science inquiry skills. National Association of Research in Science Teaching 2007 Annual Meeting, New Orleans LA.
102.
TimmermanB. E. C.StricklandD. C.JohnsonR. L.PayneJ. R. (2011). Development of a “universal’ rubric for assessing undergraduates’ scientific reasoning skills using scientific writing. Assessment & Evaluation in Higher Education, 36, 509–547.
103.
ToppingK. J.SmithE. F.SwansonI.ElliotA. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25, 149–169.
104.
WaldeckJ. H.OrregoV. O.PlaxT. G.KearneyP. (1997). Graduate student/faculty mentoring relationships: Who gets mentored, how it happens, and to what end. Communication Quarterly, 45, 93–109.
105.
WalkerG. E.GoldeC.JonesL.Conklin BueschelA.HutchingsP. (2008). The formation of scholars: Rethinking doctoral education for the twenty-first century. San Francisco, CA: Jossey-Bass.
106.
WeidmanJ. C.TwaleD. J.SteinE. L. (2001). Socialization of graduate and professional students: A perilous passage? (ASHE-ERIC Higher Education Report 28, no. 3). Washington, DC: Association of the Study of Higher Education.
107.
WexleyK. N.AlexanderR. A.GreenawaltJ. P.CouchM. A. (1980). Attitudinal congruence and similarity as related to interpersonal evaluations in manager-subordinate dyads. Academy of Management Journal, 23, 320–330.
108.
WillisonJ.O’ReganK. (2007). Commonly known, commonly not known, totally unframed: A framework for students becoming researchers. Higher Education Research & Development, 26, 393–409.
109.
WittwerJ.NücklesM.RenklA. (2008). Is underestimation less detrimental than overestimation? The impacts of experts’ beliefs about a layperson’s knowledge on learning and question asking. Instructional Science, 36, 27–52.
110.
ZhaoC.GoldeC. M.McCormickA. C. (2007). More than a signature: How advisor choice and advisor behavior affect doctoral student satisfaction. Journal of Further and Higher Education, 31, 263–281.
111.
ZimmermanC. (2000). The development of scientific reasoning. Developmental Review, 20, 99–149.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
