Text

theses

The primary purpose of cognitively diagnostic assessment (CDA) is to provide useful information about students’ learning needs. The attributes (i.e., latent skills) possessed by examinees can be uncovered based on examinee responses to test items primarily in conjunction with cognitive diagnosis models (CDMs). Most, if not all, CDMs require a Q-matrix to specify the attributes measured by each item. When attributes are correctly specified, CDMs have been shown useful in identifying examinees’ mastery or nonmastery of attributes in a domain of interest. However, conventional Q-matrix development process involves some degree of subjectivity, which can result in validity concerns due to inaccurate attribute specifications. Although some statistical procedures exist in the literature, additional work is still needed to address some concerns about validating attribute specifications in the Q-matrix. Each of the three studies of this dissertation introduces new Q-matrix validation procedures. The first study presents an EM-based δ-method, namely, the iterative modified sequential search algorithm (IMSSA), to empirically validate the correctness of attribute specifications for the deterministic inputs, noisy “and” gate (DINA) model. In this study, the performance of the IMSSA is compared to that of some existing parametric and nonparametric methods through simulated and real data analyses. The second study proposes new indices under the generalized DINA (G-DINA) model, namely, the iterative Jensen-Shannon Divergence (iJSD) index and iterative G-DINA model discrimination index (iGDI), to determine the correctness of attribute specifications in the Q-matrix. The iJSD is more general than the iGDI that can be applied under both dichotomous and nondichotomous models, whereas, the iGDI can only be used under dichotomous models. As with the iJSD, the main advantage of the iGDI is the inclusion of an iterative algorithm in the original GDI so that better results can be obtained. The feasibility of the iJSD and iGDI is investigated using simulated and real data. In the final study, the Wald-Q, an adaptation of the Wald statistical test to the Q- matrix validation context, is presented. The Wald-Q is applied under situations where the true underlying process is known or unknown. Using simulated and real data, the Wald-Q was compared to the IMSSA proposed in the first study and to iGDI proposed in the second study in conjunction with the DINA and G-DINA models, respectively. Across the three simulation studies, different factors (i.e., sample sizes, test lengths, complexity of q-vectors, degrees of q-vector misspecifications, attribute structures, and item qualities) are varied to examine the performance of the new procedures. The new procedures are further applied to fraction-subtraction data. Practical applications of the proposed procedures can lead to the advancement of the use of CDAs in educational settings. Results leading to improvements in Q-matrix validation can also help other components of cognitive diagnosis modeling, such as the estimation of model parameters, model-data fit analyses, the accuracy of attribute classifications, and ultimately, validity of CDA inferences.

ETD_8289

Ph.D.

Includes bibliographical references

by Ragip Terzi

doi:10.7282/T3571G5G

ETD doctoral

The author owns the copyright to this work.