Limits...
Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection.

Mulder J, van der Linden WJ - Psychometrika (2008)

Bottom Line: For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results.The preferences of each of these criteria for items with specific patterns of parameter values was also assessed.It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.

View Article: PubMed Central - PubMed

Affiliation: Department of Research Methodology, Measurement, and Data Analysis, Twente University, P.O. Box 217, 7500 AE Enschede, The Netherlands.

ABSTRACT
Several criteria from the optimal design literature are examined for use with item selection in multidimensional adaptive testing. In particular, it is examined what criteria are appropriate for adaptive testing in which all abilities are intentional, some should be considered as a nuisance, or the interest is in the testing of a composite of the abilities. Both the theoretical analyses and the studies of simulated data in this paper suggest that the criteria of A-optimality and D-optimality lead to the most accurate estimates when all abilities are intentional, with the former slightly outperforming the latter. The criterion of E-optimality showed occasional erratic behavior for this case of adaptive testing, and its use is not recommended. If some of the abilities are nuisances, application of the criterion of A(s)-optimality (or D(s)-optimality), which focuses on the subset of intentional abilities is recommended. For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results. The preferences of each of these criteria for items with specific patterns of parameter values was also assessed. It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.

No MeSH data available.


Surfaces of the criterion of c-optimality for Items 1 and 2 (left-hand panels) and contours of the criterion of c-optimality for the same items for θ = 0 (right-hand panels). (Note: b = 0 and c = 0.)
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC2813188&req=5

Fig4: Surfaces of the criterion of c-optimality for Items 1 and 2 (left-hand panels) and contours of the criterion of c-optimality for the same items for θ = 0 (right-hand panels). (Note: b = 0 and c = 0.)

Mentions: Indeed, this criterion prefers items with discrimination parameters that reflect the weights of importance in the composite ability, i.e., ai ∝ λ. The preference is demonstrated for a two-dimensional ability vector with equal weights λ1 = λ2 = 1 in Figure 4. (Note that we plotted the argument in the right-hand side of (24), so that a larger outcome can be interpreted as a more informative item.) Item 1 is generally more informative because λ · a1 is larger than λ · a2. Furthermore, unlike the criteria of D-, A-, and E-optimality, which yielded concave contours (see Figure 2), the contours in Figure 4 are convex. Thus, for this criterion, an item that tests several abilities simultaneously with ai ∝ λ is generally more informative than an item with a preference for a single ability.Figure 4


Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection.

Mulder J, van der Linden WJ - Psychometrika (2008)

Surfaces of the criterion of c-optimality for Items 1 and 2 (left-hand panels) and contours of the criterion of c-optimality for the same items for θ = 0 (right-hand panels). (Note: b = 0 and c = 0.)
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC2813188&req=5

Fig4: Surfaces of the criterion of c-optimality for Items 1 and 2 (left-hand panels) and contours of the criterion of c-optimality for the same items for θ = 0 (right-hand panels). (Note: b = 0 and c = 0.)
Mentions: Indeed, this criterion prefers items with discrimination parameters that reflect the weights of importance in the composite ability, i.e., ai ∝ λ. The preference is demonstrated for a two-dimensional ability vector with equal weights λ1 = λ2 = 1 in Figure 4. (Note that we plotted the argument in the right-hand side of (24), so that a larger outcome can be interpreted as a more informative item.) Item 1 is generally more informative because λ · a1 is larger than λ · a2. Furthermore, unlike the criteria of D-, A-, and E-optimality, which yielded concave contours (see Figure 2), the contours in Figure 4 are convex. Thus, for this criterion, an item that tests several abilities simultaneously with ai ∝ λ is generally more informative than an item with a preference for a single ability.Figure 4

Bottom Line: For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results.The preferences of each of these criteria for items with specific patterns of parameter values was also assessed.It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.

View Article: PubMed Central - PubMed

Affiliation: Department of Research Methodology, Measurement, and Data Analysis, Twente University, P.O. Box 217, 7500 AE Enschede, The Netherlands.

ABSTRACT
Several criteria from the optimal design literature are examined for use with item selection in multidimensional adaptive testing. In particular, it is examined what criteria are appropriate for adaptive testing in which all abilities are intentional, some should be considered as a nuisance, or the interest is in the testing of a composite of the abilities. Both the theoretical analyses and the studies of simulated data in this paper suggest that the criteria of A-optimality and D-optimality lead to the most accurate estimates when all abilities are intentional, with the former slightly outperforming the latter. The criterion of E-optimality showed occasional erratic behavior for this case of adaptive testing, and its use is not recommended. If some of the abilities are nuisances, application of the criterion of A(s)-optimality (or D(s)-optimality), which focuses on the subset of intentional abilities is recommended. For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results. The preferences of each of these criteria for items with specific patterns of parameter values was also assessed. It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.

No MeSH data available.