Limits...
The context dependency of the self-report version of the Strength and Difficulties Questionnaire (SDQ): a cross-sectional study between two administration settings.

Hoofs H, Jansen NW, Mohren DC, Jansen MW, Kant IJ - PLoS ONE (2015)

Bottom Line: Due to administration differences between the settings it remains unclear whether results and conclusions actually can be used interchangeably.Applying cut-off scores from the collective setting in the individual setting could, therefore, result in invalid conclusions and potential misuse of the instrument.To correctly apply cut-off scores these should be retrieved from the applied setting.

View Article: PubMed Central - PubMed

Affiliation: Department of Epidemiology, CAPHRI School for Public Health and Primary Care, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands; Academic Collaborative Centre for Public Health Limburg, Public Health Service Southern Limburg, Geleen, The Netherlands.

ABSTRACT

Background: The Strength and Difficulties Questionnaire (SDQ) is a screening instrument for psychosocial problems in children and adolescents, which is applied in "individual" and "collective" settings. Assessment in the individual setting is confidential for clinical applications, such as preventive child healthcare, while assessment in the collective setting is anonymous and applied in (epidemiological) research. Due to administration differences between the settings it remains unclear whether results and conclusions actually can be used interchangeably. This study therefore aims to investigate whether the SDQ is invariant across settings.

Methods: Two independent samples were retrieved (mean age = 14.07 years), one from an individual setting (N = 6,594) and one from a collective setting (N = 4,613). The SDQ was administered in the second year of secondary school in both settings. Samples come from the same socio-geographic population in the Netherlands.

Results: Confirmatory factor analysis showed that the SDQ was measurement invariant/equivalent across settings and gender. On average, children in the individual setting scored lower on total difficulties (mean difference = 2.05) and the psychosocial problems subscales compared to those in the collective setting. This was also reflected in the cut-off points for caseness, defined by the 90th percentiles, which were lower in the individual setting. Using cut-off points from the collective in the individual setting therefore resulted in a small number of cases, 2 to 3%, while ∼10% is expected.

Conclusion: The SDQ has the same connotation across the individual and collective setting. The observed structural differences regarding the mean scores, however, undermine the validity of the cross-use of absolute SDQ-scores between these settings. Applying cut-off scores from the collective setting in the individual setting could, therefore, result in invalid conclusions and potential misuse of the instrument. To correctly apply cut-off scores these should be retrieved from the applied setting.

Show MeSH

Related in: MedlinePlus

Theoretical density plots for the distributions of the SDQ total difficulties scores for the collective and individual setting, illustrating the shift between the two settings.90th percentile cut-off points are given for both settings. Segments illustrate children falling within the abnormal range (∼10%) of the individual (A+C) and collective (B+C) setting using the respective cut-off points. Children in the individual setting falling within segment C have extreme scores, defined by the cut-off point of the collective setting.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4401740&req=5

pone.0120930.g001: Theoretical density plots for the distributions of the SDQ total difficulties scores for the collective and individual setting, illustrating the shift between the two settings.90th percentile cut-off points are given for both settings. Segments illustrate children falling within the abnormal range (∼10%) of the individual (A+C) and collective (B+C) setting using the respective cut-off points. Children in the individual setting falling within segment C have extreme scores, defined by the cut-off point of the collective setting.

Mentions: Structural differences complicate the cross-use of cut-off points between settings. In the individual setting these cut-off points facilitate the screening purpose of the SDQ to differentiate between children with and without an elevated risk for developing psychopathology. Cut-off points in individual settings are often retrieved from validation studies, which are typically conducted in collective settings. There are, however, structural differences between the mean scores of the subscales and total difficulties score of the SDQ while ME/I and equal factor variances hold. These three findings combined imply that the sample distribution of the SDQ has the same form and represents the same construct for both settings, but shifts, as a whole, between the two settings (Fig. 1). The cross-use of cut-off scores between settings is therefore problematic. If, for example, the cut-off score from the collective setting, based on its 90th percentile (segment B + C; Fig. 1), is applied in the individual setting only 3% of the scores are abnormal (segment C). Only the most extreme scores are, therefore, seen as abnormal instead of the expected ∼10% (segment A + C). Predictive validity studies are based on these ∼10% highest scores (B + C). Children falling within the most extreme scores (segment C) are, however, more at risk for psychopathology compared to children who do not score as high but do score above the individual 90th percentile (segment A), as previous research indicates that an increase on the SDQ across the full range is related with increased odds for psychopathology [66]. Using the cut-off score from the collective setting in the individual setting would therefore increase the ratio of children who are correctly characterized as a case, true positives, compared to those incorrectly characterized as a case, false positives, (i.e. resulting in an increased positive predictive value) while the ratio of children who are correctly characterized as a non-case, true negatives, compared to those incorrectly characterized as a non-case, false negatives, would decrease (i.e. resulting in a decreased negative predictive value). This would bias the interpretation of caseness, as the chance that a child, scoring above the cut-off point, develops psychopathology is higher than expected. For an informed and valid usage of the SDQ in the individual setting the “actual” 90th percentile cut-off has to be determined. Or, the accuracy in this individual setting has to be reassessed using the collective setting cut-off scores.


The context dependency of the self-report version of the Strength and Difficulties Questionnaire (SDQ): a cross-sectional study between two administration settings.

Hoofs H, Jansen NW, Mohren DC, Jansen MW, Kant IJ - PLoS ONE (2015)

Theoretical density plots for the distributions of the SDQ total difficulties scores for the collective and individual setting, illustrating the shift between the two settings.90th percentile cut-off points are given for both settings. Segments illustrate children falling within the abnormal range (∼10%) of the individual (A+C) and collective (B+C) setting using the respective cut-off points. Children in the individual setting falling within segment C have extreme scores, defined by the cut-off point of the collective setting.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4401740&req=5

pone.0120930.g001: Theoretical density plots for the distributions of the SDQ total difficulties scores for the collective and individual setting, illustrating the shift between the two settings.90th percentile cut-off points are given for both settings. Segments illustrate children falling within the abnormal range (∼10%) of the individual (A+C) and collective (B+C) setting using the respective cut-off points. Children in the individual setting falling within segment C have extreme scores, defined by the cut-off point of the collective setting.
Mentions: Structural differences complicate the cross-use of cut-off points between settings. In the individual setting these cut-off points facilitate the screening purpose of the SDQ to differentiate between children with and without an elevated risk for developing psychopathology. Cut-off points in individual settings are often retrieved from validation studies, which are typically conducted in collective settings. There are, however, structural differences between the mean scores of the subscales and total difficulties score of the SDQ while ME/I and equal factor variances hold. These three findings combined imply that the sample distribution of the SDQ has the same form and represents the same construct for both settings, but shifts, as a whole, between the two settings (Fig. 1). The cross-use of cut-off scores between settings is therefore problematic. If, for example, the cut-off score from the collective setting, based on its 90th percentile (segment B + C; Fig. 1), is applied in the individual setting only 3% of the scores are abnormal (segment C). Only the most extreme scores are, therefore, seen as abnormal instead of the expected ∼10% (segment A + C). Predictive validity studies are based on these ∼10% highest scores (B + C). Children falling within the most extreme scores (segment C) are, however, more at risk for psychopathology compared to children who do not score as high but do score above the individual 90th percentile (segment A), as previous research indicates that an increase on the SDQ across the full range is related with increased odds for psychopathology [66]. Using the cut-off score from the collective setting in the individual setting would therefore increase the ratio of children who are correctly characterized as a case, true positives, compared to those incorrectly characterized as a case, false positives, (i.e. resulting in an increased positive predictive value) while the ratio of children who are correctly characterized as a non-case, true negatives, compared to those incorrectly characterized as a non-case, false negatives, would decrease (i.e. resulting in a decreased negative predictive value). This would bias the interpretation of caseness, as the chance that a child, scoring above the cut-off point, develops psychopathology is higher than expected. For an informed and valid usage of the SDQ in the individual setting the “actual” 90th percentile cut-off has to be determined. Or, the accuracy in this individual setting has to be reassessed using the collective setting cut-off scores.

Bottom Line: Due to administration differences between the settings it remains unclear whether results and conclusions actually can be used interchangeably.Applying cut-off scores from the collective setting in the individual setting could, therefore, result in invalid conclusions and potential misuse of the instrument.To correctly apply cut-off scores these should be retrieved from the applied setting.

View Article: PubMed Central - PubMed

Affiliation: Department of Epidemiology, CAPHRI School for Public Health and Primary Care, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands; Academic Collaborative Centre for Public Health Limburg, Public Health Service Southern Limburg, Geleen, The Netherlands.

ABSTRACT

Background: The Strength and Difficulties Questionnaire (SDQ) is a screening instrument for psychosocial problems in children and adolescents, which is applied in "individual" and "collective" settings. Assessment in the individual setting is confidential for clinical applications, such as preventive child healthcare, while assessment in the collective setting is anonymous and applied in (epidemiological) research. Due to administration differences between the settings it remains unclear whether results and conclusions actually can be used interchangeably. This study therefore aims to investigate whether the SDQ is invariant across settings.

Methods: Two independent samples were retrieved (mean age = 14.07 years), one from an individual setting (N = 6,594) and one from a collective setting (N = 4,613). The SDQ was administered in the second year of secondary school in both settings. Samples come from the same socio-geographic population in the Netherlands.

Results: Confirmatory factor analysis showed that the SDQ was measurement invariant/equivalent across settings and gender. On average, children in the individual setting scored lower on total difficulties (mean difference = 2.05) and the psychosocial problems subscales compared to those in the collective setting. This was also reflected in the cut-off points for caseness, defined by the 90th percentiles, which were lower in the individual setting. Using cut-off points from the collective in the individual setting therefore resulted in a small number of cases, 2 to 3%, while ∼10% is expected.

Conclusion: The SDQ has the same connotation across the individual and collective setting. The observed structural differences regarding the mean scores, however, undermine the validity of the cross-use of absolute SDQ-scores between these settings. Applying cut-off scores from the collective setting in the individual setting could, therefore, result in invalid conclusions and potential misuse of the instrument. To correctly apply cut-off scores these should be retrieved from the applied setting.

Show MeSH
Related in: MedlinePlus