Limits...
Evaluation of two selection tests for recruitment into radiology specialty training.

Patterson F, Knight A, McKnight L, Booth TC - BMC Med Educ (2016)

Bottom Line: The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both).The SJT did not correlate with either component of the examination.However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes.

View Article: PubMed Central - PubMed

Affiliation: Department of Psychology, Downing Street, Cambridge, CB2 3 EB, UK.

ABSTRACT

Background: This study evaluated whether two selection tests previously validated for primary care General Practice (GP) trainee selection could provide a valid shortlisting selection method for entry into specialty training for the secondary care specialty of radiology.

Methods: We conducted a retrospective analysis of data from radiology applicants who also applied to UK GP specialty training or Core Medical Training. The psychometric properties of the two selection tests, a clinical problem solving (CPS) test and situational judgement test (SJT), were analysed to evaluate their reliability. Predictive validity of the tests was analysed by comparing them with the current radiology selection assessments, and the licensure examination results taken after the first stage of training (Fellowship of the Royal College of Radiologists (FRCR) Part 1).

Results: The internal reliability of the two selection tests in the radiology applicant sample was good (α ≥ 0.80). The average correlation with radiology shortlisting selection scores was r = 0.26 for the CPS (with p < 0.05 in 5 of 11 shortlisting centres), r = 0.15 for the SJT (with p < 0.05 in 2 of 11 shortlisting centres) and r = 0.25 (with p < 0.05 in 5 of 11 shortlisting centres) for the two tests combined. The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both). The SJT did not correlate with either component of the examination.

Conclusions: The current CPS test may be an appropriate selection method for shortlisting in radiology but would benefit from further refinement for use in radiology to ensure that the test specification is relevant. The evidence on whether the SJT may be appropriate for shortlisting in radiology is limited. However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes. Further validation work (e.g. with non-academic outcome variables) is required to evaluate whether an SJT will add value in recruitment for radiology specialty training and will further inform construct validity of SJTs as a selection methodology.

No MeSH data available.


Flow diagram of radiology applicants who sat the Clinical Problem Solving and Situational Judgement Tests
© Copyright Policy - OpenAccess
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4940701&req=5

Fig1: Flow diagram of radiology applicants who sat the Clinical Problem Solving and Situational Judgement Tests

Mentions: Data for the combined GP/CMT population of 2009 applicants (n = 6671) were used for comparison with the radiology applicant sample (Fig. 1). Of 3108 radiology applications to 12 Deaneries (a Deanery is a regional organisation responsible for postgraduate medical training, within the structure of the UK National Health Service), 895 (28.8 %) of the applications were from individuals who had applied for GP training or CMT or both, and radiology shortlisting scores were available for 799 of the 895 (89.3 %) applications to 11 Deaneries. Because individuals could apply to more than one Deanery, this represented a total of 297 individual applicants. If an individual was successfully shortlisted they proceeded to interview. Of the cohort that had applied for GP training or CMT or both, radiology interview scores were available for a total of 69 shortlisting applications from 11 Deaneries. In 2009, both radiology shortlisting and radiology interviews were conducted by individual Deaneries rather than at a national level, so statistical correlations were conducted separately for each Deanery. Sample sizes from separate Deaneries were too small to conduct statistically meaningful correlations between the GP/CMT selection tests and radiology interview scores, but were of sufficient size to explore correlations between the GP/CMT selection tests and radiology shortlisting scores. The mean age of the radiology shortlisting sample was 31 years (range 24 – 46). Further demographic characteristics of the radiology shortlisting sample are presented in Table 2, showing a high proportion of Asian participants and participants from outside of the UK. The only available comparative demographic data of the radiology shortlisting cohort are those successfully recruited into radiology, which is a different subgroup. However, in contrast to all the GP/CMT applicants, the different radiology subgroups are similar in that the largest ethnic group is Asian and that there are more males than females.Fig. 1


Evaluation of two selection tests for recruitment into radiology specialty training.

Patterson F, Knight A, McKnight L, Booth TC - BMC Med Educ (2016)

Flow diagram of radiology applicants who sat the Clinical Problem Solving and Situational Judgement Tests
© Copyright Policy - OpenAccess
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4940701&req=5

Fig1: Flow diagram of radiology applicants who sat the Clinical Problem Solving and Situational Judgement Tests
Mentions: Data for the combined GP/CMT population of 2009 applicants (n = 6671) were used for comparison with the radiology applicant sample (Fig. 1). Of 3108 radiology applications to 12 Deaneries (a Deanery is a regional organisation responsible for postgraduate medical training, within the structure of the UK National Health Service), 895 (28.8 %) of the applications were from individuals who had applied for GP training or CMT or both, and radiology shortlisting scores were available for 799 of the 895 (89.3 %) applications to 11 Deaneries. Because individuals could apply to more than one Deanery, this represented a total of 297 individual applicants. If an individual was successfully shortlisted they proceeded to interview. Of the cohort that had applied for GP training or CMT or both, radiology interview scores were available for a total of 69 shortlisting applications from 11 Deaneries. In 2009, both radiology shortlisting and radiology interviews were conducted by individual Deaneries rather than at a national level, so statistical correlations were conducted separately for each Deanery. Sample sizes from separate Deaneries were too small to conduct statistically meaningful correlations between the GP/CMT selection tests and radiology interview scores, but were of sufficient size to explore correlations between the GP/CMT selection tests and radiology shortlisting scores. The mean age of the radiology shortlisting sample was 31 years (range 24 – 46). Further demographic characteristics of the radiology shortlisting sample are presented in Table 2, showing a high proportion of Asian participants and participants from outside of the UK. The only available comparative demographic data of the radiology shortlisting cohort are those successfully recruited into radiology, which is a different subgroup. However, in contrast to all the GP/CMT applicants, the different radiology subgroups are similar in that the largest ethnic group is Asian and that there are more males than females.Fig. 1

Bottom Line: The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both).The SJT did not correlate with either component of the examination.However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes.

View Article: PubMed Central - PubMed

Affiliation: Department of Psychology, Downing Street, Cambridge, CB2 3 EB, UK.

ABSTRACT

Background: This study evaluated whether two selection tests previously validated for primary care General Practice (GP) trainee selection could provide a valid shortlisting selection method for entry into specialty training for the secondary care specialty of radiology.

Methods: We conducted a retrospective analysis of data from radiology applicants who also applied to UK GP specialty training or Core Medical Training. The psychometric properties of the two selection tests, a clinical problem solving (CPS) test and situational judgement test (SJT), were analysed to evaluate their reliability. Predictive validity of the tests was analysed by comparing them with the current radiology selection assessments, and the licensure examination results taken after the first stage of training (Fellowship of the Royal College of Radiologists (FRCR) Part 1).

Results: The internal reliability of the two selection tests in the radiology applicant sample was good (α ≥ 0.80). The average correlation with radiology shortlisting selection scores was r = 0.26 for the CPS (with p < 0.05 in 5 of 11 shortlisting centres), r = 0.15 for the SJT (with p < 0.05 in 2 of 11 shortlisting centres) and r = 0.25 (with p < 0.05 in 5 of 11 shortlisting centres) for the two tests combined. The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both). The SJT did not correlate with either component of the examination.

Conclusions: The current CPS test may be an appropriate selection method for shortlisting in radiology but would benefit from further refinement for use in radiology to ensure that the test specification is relevant. The evidence on whether the SJT may be appropriate for shortlisting in radiology is limited. However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes. Further validation work (e.g. with non-academic outcome variables) is required to evaluate whether an SJT will add value in recruitment for radiology specialty training and will further inform construct validity of SJTs as a selection methodology.

No MeSH data available.