Strategies for addressing high-dimensional cognitively diagnostic assessment problems
Description
TitleStrategies for addressing high-dimensional cognitively diagnostic assessment problems
Date Created2019
Other Date2019-10 (degree)
Extent1 online resource (xiii, 117 pages) : illustrations
DescriptionIn recent years, cognitive diagnosis models (CDMs) have sparked the interest of educational measurement researchers and practitioners because of its capability to provide formative information on student mastery or nonmastery of a set of fine-grained skills. One of the advantages of CDMs is that, by treating latent variables as discrete, usually binary, CDMs can accommodate higher dimensional latent space than multidimensional latent trait (e.g., multidimensional item response theory) models. Theoretically, the number of attributes that can be estimated by a CDM is unlimited; however, in practice, this number may not exceed 20 due to a number of computational issues. This constraint limits the use of CDMs in scenarios where a comprehensive diagnosis of a complete knowledge space, such as large-scale diagnostic assessments or retrofitting summative assessments using CDMs, is of interest.
In this dissertation, a series of strategies are proposed to address issues in classifying examinees' proficiency profiles for high-dimensional testing data. In particular, these strategies can be used in situations where attributes can be partitioned into non-overlapping knowledge subsets. An approach, called the accordion procedure (AP), is proposed to address the high dimensionality estimation problem by focusing only on the attributes of one particular subset at a time, while the attributes of each of the remaining subsets are collapsed to create composite nuisance attributes. Simulation studies are conducted to examine the performance of AP compared to the complete profile estimation procedure in terms of classification accuracy and computation time. A real data illustration is also provided by retrofitting extant large-scale assessment data using AP.
To provide appropriate actionable feedback, one important prerequisite is ensuring the CDMs fitted to test data yield accurate classifications of examinees' proficiency profiles. However, due to various reasons (e.g., short test, poor item quality), tests sometimes do not provide sufficient information to classify examinees accurately.
When a test is not sufficiently informative, other sources of information might be needed to improve the classification accuracy. Thus, in the second study, covariates are incorporated in the context of AP using a four-step latent regression approach to supplement the information obtained from CDMs. The four-step approach is shown to be computationally more manageable when data are high-dimensional, as well as more flexible when specifications of each step need to be adjusted. Simulation and real-data studies are conducted to examine the performance of the proposed approach.
Cognitive diagnosis computerized adaptive testing (CD-CAT) has been proposed to administer a test more efficiently by selecting the optimal set of items for each examinee. However, when the number of skills of interest is large, practical issues, such as calibration of item pools and item selection method, emerge.
The third study aims to propose a series of strategies to make high-dimensional CD-CAT feasible, namely, an item pool calibration method, item selection method, and examinees' prior distribution estimation method. Simulation studies are conducted to evaluate the performance of the proposed strategies.
In summary, the issues associated with using CDMs in high-dimensional situations are addressed in this dissertation. Several strategies are proposed primarily with the aim of obtaining accurate classification results to ensure that the feedback and remedial procedures are informative and effective.
NotePh.D.
NoteIncludes bibliographical references
Genretheses, ETD doctoral
LanguageEnglish
CollectionSchool of Graduate Studies Electronic Theses and Dissertations
Organization NameRutgers, The State University of New Jersey
RightsThe author owns the copyright to this work.
Statistical Profile