On November 18th, Dr. Jennifer Case McKee successfully defended her dissertation on “Identifying Construct-Irrelevant Variance and Construct Under-Representations as Support of External Evidence of Validity.”
Test validity has commonly been defined in terms of a test measuring what it is supposed to measure, however there has been a century of discussion and debate on test validity and what qualifies as evidence of test validity. Some argue in favor of restricting test validity evidence to that which is internal to the test, such as the test items, while others argue that external information, such as the relationships with other variables, also offers evidence toward test validity. Cronbach and Meehl’s groundbreaking paper in 1955 introduced the concept of the nomological net, which is the construct’s lawful relationships with other variables, and they suggested that the nomological net provides validity evidence. Using a simulation study, I tested if relationships to other variables can offer information about test validity by looking at the ability to identify two sources of invalidity. Two simulation studies were conducted to test the two types of construct invalidity proposed by Messick (1989): construct under-representation and construct-irrelevant variance. Each used a 4 × 4 × 4 × 4 × 2 design, where four levels of factor loadings, structural coefficients, and sample size were analyzed along with four different mediation SEM models in two different validity conditions. Data was simulated using R to analyze whether we can detect construct invalidity using model fit. The results demonstrated that we can detect both construct under-representation and construct-irrelevant variance, particularly when structural coefficients and factor loadings were large. These findings were then demonstrated using a real-data example, where again both construct under-representation and construct-irrelevant variance were detected. These findings support the broad view of test validity, which includes external evidence in the definition of test validity.