Psychology 557
Psychometric Methods
Fall 2007
| Instructor | Patrick E. McKnight, Ph.D. |
| Office | David King 2064/2065 |
| Office Hours | Tues 3:30pm-4:30pm and by appointment |
| Phone | (703) 993–8292 |
| Class Location | DK 2072 |
| Class Date/Time | Tuesday 4:30pm-7:10pm |
| Class website | http://mres.gmu.edu/PSYC557/ |
| Important Dates | Please see GMU academic calendar |
| Syllabus | PDF format |
Overview
The following course is a survey of important measurement topics within social and behavioral science. Typical measurement classes tend to be heavily mathematical and overburden students on details they so rarely need. In sharp constrast, this course covers similar material to all other graduate courses in social science measurement but at the conceptual level. The course is technical in nature only in the sense that measurement is a technical aspect of all critical inquiry.
Prerequisites
Due to the nature of the material and the relevance to research, I assume all students will have successfully completed a graduate course in statistics \textbf{and} a graduate course in research methods. I do not intend to cover in great detail the statistical models underlying measurement tools but it is essential to understand statistical procedures in general and to appreciate how measurement fits into the process of research.
Course Requirements and Grading
The course covers many topics in only a semester so the reading requirements might be more than what most graduate students experience in other courses. Let me emphasize this point - this course values thinking about measurement in general and therefore requires an above average amount of reading. I expect all students to attend every class, complete all the readings prior to the class meeting, and come prepared to discuss the topic as outlined. In exchange for these requirements, I do not require written papers nor exams. One brief assignment will be discussed the first day of the course and will require some work outside class. Grades, therefore, are determined based upon class discussion and this brief assignment.
Readings and Required Texts
The readings will be made available in electronic format. Each article is scanned into an Adobe Acrobat file (i.e., pdf file). The quality of some readings is not great but all articles are readable in the format they are distributed. Some students may prefer to get the original articles from the source journals but that is left to each student to decide. The electronic versions are distributed at no charge to students enrolled in the course.
The following three books are required for the course. They may be purchased either at the bookstore or online. Readings from these books are not required until later in the semester, however, you might want to order the books at the beginning of the semester. I would urge you not to rely on the library for your course materials. Other students and faculty members may recall these books at any time and it might interfere with you completing the readings for the course.
Topic Outline and Readings
1: Introduction and orientation
The first class meeting covers the brief, semester-long assignment. Please come prepared to discuss your interests in the course and your expectations for learning measurement.
2: Logic of measurement
The following articles address the general topic and logic of measurement. In particular, the concept of measurement and what measures ought to deliver is the focus of these readings and our in-class discussion.
Webb, E., Campbell, D.T., Schwartz, R.D., Sechrest, L., and Grove, J. (1981). Nonreactive measures in the social sciences. Boston: Houghton-Mifflin, 1-40, 41-77, 78-87, 197-240, 275-330
3: Concepts of measurement
Measurement, like all technical areas within science, contains many specific terms. Frequently we perjoratively call these terms jargon but in the case of measurement, the terms convey important details worth knowing. The following readings cover the use and misuse of many measurement terms.
Wallace, J. (1966). An abilities conception of personality. The American Psychologist, 21, 132-138.
In addition to the preceding, you should read the following paper, which we will use as background and discuss on a continuing basis during the remainder of the course. This paper is of unusual importance, although it has, sadly in my estimation, fallen into neglect.
4: Statistics for measurement
This session will be devoted to the discussion of a variety of statistical procedures and tests that are especially useful in measurement research and development and that are not always covered adequately in more general statistics courses.
Sapolsky, R.M. (1987). The case of the falling nightwatchmen. Discover, July, 42-45.
Test Service Bulletin, (1953). Better than chance. The Psychological Corporation, No. 45, 8-12.
Test Service Bulletin. (1956). How accurate is a test score? The Psychological Corporation, No. 50.
5: Scaling and scoring responses
Frequently overlooked or downright neglected, scaling is an essential part of measurement. We will discuss the following articles in the context of scaling and scoring instruments. Pay attention to the broader perspective when reading these articles. The important matters are not necessarily in the details but in the points with the greatest implications for our approach to measurement.
Diamond, J. (1987). Soft sciences are often harder than hard sciences. Discover, Aug., 34-39.
6: Classical test theory
I begin covering specific measurement theory today. Classical test theory (CTT) is the oldest and longest-standing theory in social science. While the theory and methods has grown many detractors, it remains the most widely used theory.
7: Classical test theory
We continue our discussion of CTT but focus now on the implications of the methods and how they might lead us to appreciate measures.
8: Generalizability theory
In this class session, I will introduce the topic of generalizability. It is important that you read the following book chapters carefully. G-theory, as it is called, is not an easy topic but these authors do the best job at conveying this complex material.
9: Generalizability theory
We will continue our discussion of G-theory during this class. An in-class exercise will offer a bit more insight into the workings of the procedure.
Shavelson, R.J., and Webb, N.M. (1991). Generalizability theory: a primer. Newbury Park, CA: Sage Publications. pp. 83–98.
10: Factor Analysis and the Multitrait-Multimethod Matrix
Factor Analysis: Perhaps the oldest method in any measurement area, factor analysis remains a vital force in evaluating social science instruments. The following articles provide you with an introduction and application of factor analysis.
MTMM: This topic is one of the more vexing for most graduate students. I would encourage all students to carefully read through the first article before reading the second. The devils are in the details regarding MTMM but you must know the purpose of the procedure before you can understand - at any depth that is - the merits of different approachs to analyzing MTMM data.
11: Latent Response Models: Item response theory and Rasch models
We have now come to the point in the course where we cover what is termed ``modern measurement theory.’‘ As I noted previously, CTT is the longest-standing theory but item response theory (IRT) is the successor to CTT in many measurement domains. I will introduce IRT in some detail, however, for you to understand my lecture you will need to read the following article very carefully.
12: Applications of Latent Response Models
We continue our discussion of latent response models with these examples.
13: Measurement for decisions
The readings for this week is an entire book. While this might appear daunting, it is not. The book is short - fewer than 250 pages - and easy-to-read. Kraemer does a masterful job at conveying the topics and she provides excellent examples. Do not feel compelled to learn all the nuances of these methods - read the book for description and clarity.
Kraemer, H.C. (1992). Evaluating medical tests:objective and quantitative guidelines. Newbury Park, CA: Sage Publications.
We finish the course with a classic article by Paul Meehl and his colleague Al Rosen. This paper extends Kraemer’s descriptions of signal detection theory but provides a broader context and specific examples within the clinical psychology domain.
TBA: Meta-analysis and validity generalizability
If time permits, we will cover meta-analysis and validity generalizability. Measurement is not always about measurement theory. At times, we might be interested in collecting data that serves as an indicator of other unobservable entities such as research findings. Meta-analysis is one method for gathering and making sense out of multiple research findings.