Improving Construct Measurement In Management Research: Comments and a Quantitative Approach for Assessing the Theoretical Content Adequacy of Paper-and-Pencil Survey-Type Instruments
Restricted accessResearch articleFirst published online April, 1993
Improving Construct Measurement In Management Research: Comments and a Quantitative Approach for Assessing the Theoretical Content Adequacy of Paper-and-Pencil Survey-Type Instruments
A cursory review of the measurement practices found in four diverse management journals is briefly reported, suggesting that the current authors’ concerns about serious measurement deficiencies in at least some subdomains of the field may not be groundless. Then, it is suggested that the demonstration of instrument content adequacy be demanded as an initial step toward construct validation by all studies which use new, modified, or previously unexamined measures. Although adequate content is a necessary pre-condition for instrument validity, there are no well-established quantitative methods for examining content adequacy. Consequently, the main focus of this article is to propose a new approach for the quantitative assessment of content adequacy and to illustrate this approach by evaluating a commonly-used job satisfaction scale. Problems in and suggestions for the use of this method a represented, along with implications and potential new applications of the method for management research.
Get full access to this article
View all access options for this article.
References
1.
Aldag, R. J. & Stearns, T. M. (1988). Issues in research methodology. Journal of Management, 14, 253-276.
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
3.
Anastasi, A. (1982). Psychological resting, 5th ed.New York: MacMillan.
4.
Bass, B. M., Cascio, W. J., & O’Connor, E. J. (1974). Magnitude estimation of expressions of frequency and amount. Journal of Applied Psychology, 34, 313-320.
5.
Bish, J. & Schriesheim, C. A. (1974). An exploratory dimensional analysis of form XII of the Ohio State Leadership Scales. Academy of Management Proceedings. Thirty-fourth annual meeting.
6.
Carmines, E. G. & Zeller, R. A. (1979). Reliability and validly assessment. Beverly Hills: Sage.
7.
Carroll, R. M. & Field, J. (1974). A comparison of the classification accuracy of profile similarity measures. Multivariate Behavioral Research, 9, 373-380.
8.
Cattell, R. B. (1949). rp and other coefficients of pattern similarity. Psychometrika, 14, 279-298.
9.
Cattell, R. B..(1952). Fuctor analysis. New York: Harper & Bros.
10.
Cattell, R. B. (1966a). The data box: Its ordering of total resources in terms of possible relational systems. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology. Chicago: Rand McNally.
11.
Cattell, R. B. (1966b). The scree test for the number of factors. Multiivariate Behavioral Research, 1, 245-276.
12.
Cattell, R. B. (1978). The scientific use of factor analysis in behavioral and life sciences. New York: Plenum Press.
13.
Cattell, R. B., Blaine, D. D., & Brennan, J. M. (1984). N-way factor analysis for obtaining personality-environment-test contribution systems for any response: A plasmode illustration, In H. G. Law, C. W. Snyder, Jr., J. A. Hattie, & R. P. McDonald (Eds.), Research methods for multimode data analysis. New York: Praeger.
14.
Clark, K. E. & Clark, M. B. (Eds.) (1990). Measures of leadership. Greensboro, NC: Center for Creative Leadership.
15.
Comrey, A. L. (1973). Afirst course in factor analysis. New York: Academic Press.
16.
Cook, J. D., Hepworth, S. J., Wall, T. D., & Warr, P. B. (1981). The experience of work; A compendium and review of 249 measures and their use. New York: Academic Press.
17.
Cook, T. D. & Campbell, D. T. (1976). The design and conduct of quasi-experiments and true experiments in field settings. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology. Chicago: Rand McNally.
18.
Cracker, L. M., Miller, D., & Franks, E. A. (1989). Quantitative methods for assessing the fit between test and curriculum. Applied Measurement in Education, 2, 179-194.
19.
Davidson, M. L. (1985). Multidimensional scaling versus components analysis of test intercorrelations. Psychological Bulletin, 97, 94-105.
20.
Dunham, R. B., Aldag, R. J., & Brief, A. P. (1977). Dimensionality of task design as measured by the Job Diagnostic Survey. Academy of Management Journal, 20, 209-223.
21.
Ebel, R. L. (1977). Comments on some problems of employment testing. Personnel Psychology, 30, 55-63.
22.
Extejt, M. M. & Smith, J. E. (1990). The behavioral sciences and management: An evaluation of relevant journals. Journal of Management, 16, 539-551.
23.
Fitzpatrick, A. R. (1983). The meaning of content validity. Applied Psychological Measurement, 7, 3-13.
24.
Fleishman, E. A. (1973). Twenty years of consideration and structure. In E. A. Fleishman & J. G. Hunt (Eds.), Current developments in the study of leadership. Carbondale: Southern Illinois University Press.
25.
Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291-314.
26.
Fried, Y. & Ferris, G. R. (1986). The dimensionality of job characteristics: Some neglected issues. Journal of Applied Psychology, 71, 419-426.
27.
Gorsuch, R. L. (1983). Factor anulysis, 2nd ed.Hillsdale, NJ: Erlbaum.
28.
Guilford, J. P. (1954). Psychometric methods, 2nd ed.New York: McGraw-Hill.
29.
Guion, R. M. (1977). Content validity: The source of my discontent. Applied Psychological Measurement, 1, 11-10.
30.
Hackman, J. R. & Oldham, G. R. (1980). Work redesign. Reading, MA: Addison-Wesley.
31.
Halpin, A. W. & Winer, B. J. (1957). A factorial study of the leader behavior descriptions. In R. M. Stogdill & A. E. Coons (Eds.), Leader behavior: Its description and meusurement. Columbus: Bureau of Business Research, Ohio State University.
32.
Hambleton, R. K. (1984). Validating the test score. In R.A. Berk (Ed.), A guide to criterion-referenced test construction. Baltimore: Johns Hopkins University Press.
33.
Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167-184.
34.
Hinkin, T. R. & Schriesheim, C. A. (1989). Development and application of new scales to measure the French and Raven (1959) bases of social power. Journal of Applied Psychology, 74, 561-567.
35.
Hunt, J. G., Osborn, R. N., & Schriesheim, C. A. (1978). Some neglected aspects of leadership research. Proceedings of the Midwest Academy of Management Conference. Bloomington, IN: Academy of Management, Midwest Division,
36.
Jackson, D. N. (1971). The dynamics of structured personality tests. Psychological Review, 78, 229-248.
37.
Jöreskog, K. G. & Sörbom, D.1989. LISREL 7user's reference guide. Mooresville, IN: Scientific Software, Inc.
38.
Kerlinger, F. N. (1986). Foundations of behavioral research, 3rd ed.New York: Holt, Rhinehart, & Winston.
39.
Kerr, S. & Jermier, J. M. (1978). Substitutes for leadership: Their meaning and measurement. Organizational Behavior and Human Performance, 22, 375-403.
40.
Kerr, S.Schriesheim, C. A., Murphy, C. J., & Stogdill, R. M. (1974). Toward a contingency theory of leadership based upon the consideration and initiating structure literature. Organizational Behavior and Human Performance, 12, 62-82.
41.
Kruskal, J. B. & Wish, M. (1978). Multidimensional scaling. Newbury Park, CA: Sage.
42.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563-575.
43.
Locke, E. A. (1976). The nature and causes of job satisfaction. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology. Chicago: Rand McNally.
44.
MacMillan, I. C. (1991). The emerging forum for business policy scholars. Strategic Management Journal, 12, 161-165.
45.
Messick, S. (1989). Validity. In R. Linn (Ed.), Educarional measurement, 3rd ed.Washington, DC: American Council on Education.
46.
Morris, L. L. & Fitz-Gibbon, C. T. (1978). How to measure achievement. Beverly Hills: Sage.
47.
Nunnally. J. C. (1978). Psychometric theory, 2nd ed.New York: McGraw-Hill.
48.
Osgood, C. E., Ware, E. E., & Morris, C. (1961). Analysis of the connotative meanings of a variety of human values as expressed by American college students. Journal of Abnormal and Social Psychology, 62, 62-73.
49.
Podsakoff, P. M. & Dalton, D. R. (1987). Research methodology in organizational studies. Journal of Management, 13, 419-441
50.
Podsakoff. P. M. & Schriesheim, C. A. (1985). Field studies of French and Raven’s bases of social power: Reanalysis, critique, and suggestions for future research. Psychological Bulletin, 97, 387-411.
51.
Price, J. L. & Mueller, C. W. (1986). Handbook of organizational measurement. Marshfield, MA: Pitman.
52.
Roberts, K. H. & Click. W. (1981). The job characteristics approach to task design: A critical review. Journal of Applied Psycho1og, 66, 193-217.
53.
Rummel, R. J. (1970). Applied factor analysis. Evanston, IL: Northwestern University Press.
54.
Schiffman, S. S., Reynolds, M. L., & Young, F. W. (1981). Multidimensional scaling: Theory, methods, and applications. New York: Academic Press.
55.
Schoenfeldt, L. F. (1984). Psychometric properties of organizational research instruments. In T. S. Bateman & G. R. Ferris (Eds.), Method and analysis in organizational research. Reston, VA: Reston Publishing.
56.
Schriesheim, C. A. (1975). Use of multidimensional scaling. In J. G. Hunt & L. L. Larson (Eds.), Leadership, frontiers. Kent, OH: Kent State University Press.
57.
Schriesheim, C. A. (1978). Development, validation, and application of new leadership behavior and expectancy research instruments. Unpublished doctoral dissertation, Ohio State University.
58.
Schriesheim, C. A., Gardiner, C. C., Lankau. M. J., Powers, K. J., & Scandura. T. A. (1993). An empirical comparison of approaches for quantitatively assessing the content adequacy of paper-and-pencil measurement instruments. Unpublished paper, School of Business Administration, University of Miami.
59.
Schriesheim, C. A., Hinkin, T. R., & Podsakoff, P. M. (1991). Can ipsative measurement produce erroneous results in field studies of the five French and Raven bases of power? An empirical investigation. Journal of Applied Psychology, 76, 106-114.
60.
Schriesheim, C. A., House, R. J., & Kerr, S. (1976). Leader initiating structure: A reconciliation of discrepant research results and some empirical tests. Organizational Behavior and Human Performance, 16. 297-321.
61.
Schriesheim, C. A. & Kerr, S. (1974). Psychometric properties of the Ohio State leadership scales. Psychological Bulletin, 81, 756-765.
62.
Schriesheim, C. A. & Kerr, S. (1977). Theories and measures of leadership: A critical appraisal of current and future directions. In J. G. Hunt & L. L. Larson (Eds.), Leadership: The cutting edge. Carbondale: Southern Illinois University Press.
63.
Schriesheim, C. A., Kinicki, A. J., & Schriesheim, J. F. (1979). The effect of leniency on leader behavior descriptions. Organizational Behavior and Human Performance, 23, 1-29.
64.
Schriesheim, C. A. & Powers, K. J. (1983). The quantitative assessment of content validity. Southwest Academy of Management Proceedings. Houston: Academy of Management, Southwest Division.
65.
Schriesheim, C. A. & Von Glinow, M. A. (1977). The path-goal theory of leadership: A theoretical and empirical analysis. Academy of Management Journal, 20, 398-405.
66.
Schwab, D. P. (1980). Construct validity in organizational behavior. Pp. 3-43 in B. Staw & L.L. Cummings (Eds.), Research in Organizational Behavior, Vol. 2. Greenwich, CT: JAI.
67.
Siegel, S. (1957). Nonparametric statistics. New York: McGraw-Hill.
68.
Sireci, S. G. & Geisinger, K. F. (1992). Analyzing test content using cluster analysis and multidimensional scaling. Applied Psychological Measurement, 16, 17-31.
69.
Snyder, C. W., Lyw, H. G:, & Hattie, J. A. (1984). Overview of multimode analytic methods. In H. G. Law, C. W. Snyder. Jr..J. A. Hattie. & R. P. McDonald (Eds.), Research methods for multimode data, analysis. New York: Praeger.
70.
Stephenson, W. (1953). The Study of behavior: Q-technique and its methodology. Chicago: University of Chicago Press.
71.
Tenopyr, M. L. (1977). Content-construct confusion. Personnel Psychology, 30, 47-54.
72.
Tucker, L. T. (1966). Factor analysis of relevance judgments: An approach to content validity. In A. Anastasi (Ed.), Testing problems in perspective. Washington, D.C.: American Council on Education.
73.
Weiss, D. J., Dawis, R. V., England, G. W., & Lofquist, L. H. (1967). Manual for the Minnesota Satisfaction Questionnaire. Minneapolis: Industrial Relations Center, University of Minnesota.
74.
Weiss. D. J.Dawis, R. V., Lofquist, L. H., & England, G. W. (1966). Instrumentation for the theory of work adjustment. Minneapolis: Industrial Relations Center, University of Minnesota.
75.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
