Limits...
Hitting and missing targets by ambulance services for emergency calls: effects of different systems of performance measurement within the UK.

Bevan G, Hamblin R - J R Stat Soc Ser A Stat Soc (2009)

Bottom Line: In other countries, the target has been missed by such large margins that services would have been publicly reported as failing, if they had been covered by the English system of star ratings.The paper argues that this case-study adds to evidence from comparisons of different systems of hospital performance measurement that, to have an effect, these systems need to be designed to inflict reputational damage on those that have performed poorly; and it explores implications of this hypothesis.The paper also asks questions about the adequacy of systems of performance measurement of ambulance services in UK countries.

View Article: PubMed Central - PubMed

ABSTRACT
Following devolution, differences developed between UK countries in systems of measuring performance against a common target that ambulance services ought to respond to 75% of calls for what may be immediately life threatening emergencies (category A calls) within 8 minutes. Only in England was this target integral to a ranking system of 'star rating', which inflicted reputational damage on services that failed to hit targets, and only in England has this target been met. In other countries, the target has been missed by such large margins that services would have been publicly reported as failing, if they had been covered by the English system of star ratings. The paper argues that this case-study adds to evidence from comparisons of different systems of hospital performance measurement that, to have an effect, these systems need to be designed to inflict reputational damage on those that have performed poorly; and it explores implications of this hypothesis. The paper also asks questions about the adequacy of systems of performance measurement of ambulance services in UK countries.

No MeSH data available.


Related in: MedlinePlus

Spike in frequency of response times to category A calls at 8 minutes by one service (source: Commission for Health Improvement (2003c))
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC2667302&req=5

fig07: Spike in frequency of response times to category A calls at 8 minutes by one service (source: Commission for Health Improvement (2003c))

Mentions: The CHI found out that, in one service, the times of responses taking longer than 8 minutes had been ‘corrected’ to be recorded as taking less than 8 minutes. This was mainly because targets created a culture in which staff felt under pressure to record the ‘right’ answer. A subsequent reanalysis of all English ambulance service data showed that such manual ‘corrections’ had been undertaken in around a third of trusts. This investigation concentrated on the frequency distributions of response times and identified oddly shaped frequency distributions that exhibited sharp discontinuities around the 8-minute target (Commission for Health Improvement, 2003c, d). Fig. 3 gives the expected (‘uncorrected’) distribution from one service: of a ‘noisy’ decline in numbers of responses over time with no obvious jump around the 8-minute target. Figs 4, 5, 6, 7 and 8 are examples from different trusts of different types of ‘corrections’: Fig. 4 shows a marginal discontinuity, which becomes more marked in Figs 5 and 6; Fig. 7 has a dramatic spike at 8 minutes. In each of these there is a clear implication that calls just over 8 minutes were reassigned to give a response time of 8 minutes or less. Fig. 8 shows a spike at each minute; in this case the problem was not one of gaming, but such poor data recording systems that for a significant proportion of calls manual entries were recorded to the nearest minute. The CHI estimated, however, that the effect of the most dramatic corrections would have improved performance by at most 6%; hence star ratings did produce substantial improvements in performance against the category A 8-minute target (of up to 20% since 1999).


Hitting and missing targets by ambulance services for emergency calls: effects of different systems of performance measurement within the UK.

Bevan G, Hamblin R - J R Stat Soc Ser A Stat Soc (2009)

Spike in frequency of response times to category A calls at 8 minutes by one service (source: Commission for Health Improvement (2003c))
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC2667302&req=5

fig07: Spike in frequency of response times to category A calls at 8 minutes by one service (source: Commission for Health Improvement (2003c))
Mentions: The CHI found out that, in one service, the times of responses taking longer than 8 minutes had been ‘corrected’ to be recorded as taking less than 8 minutes. This was mainly because targets created a culture in which staff felt under pressure to record the ‘right’ answer. A subsequent reanalysis of all English ambulance service data showed that such manual ‘corrections’ had been undertaken in around a third of trusts. This investigation concentrated on the frequency distributions of response times and identified oddly shaped frequency distributions that exhibited sharp discontinuities around the 8-minute target (Commission for Health Improvement, 2003c, d). Fig. 3 gives the expected (‘uncorrected’) distribution from one service: of a ‘noisy’ decline in numbers of responses over time with no obvious jump around the 8-minute target. Figs 4, 5, 6, 7 and 8 are examples from different trusts of different types of ‘corrections’: Fig. 4 shows a marginal discontinuity, which becomes more marked in Figs 5 and 6; Fig. 7 has a dramatic spike at 8 minutes. In each of these there is a clear implication that calls just over 8 minutes were reassigned to give a response time of 8 minutes or less. Fig. 8 shows a spike at each minute; in this case the problem was not one of gaming, but such poor data recording systems that for a significant proportion of calls manual entries were recorded to the nearest minute. The CHI estimated, however, that the effect of the most dramatic corrections would have improved performance by at most 6%; hence star ratings did produce substantial improvements in performance against the category A 8-minute target (of up to 20% since 1999).

Bottom Line: In other countries, the target has been missed by such large margins that services would have been publicly reported as failing, if they had been covered by the English system of star ratings.The paper argues that this case-study adds to evidence from comparisons of different systems of hospital performance measurement that, to have an effect, these systems need to be designed to inflict reputational damage on those that have performed poorly; and it explores implications of this hypothesis.The paper also asks questions about the adequacy of systems of performance measurement of ambulance services in UK countries.

View Article: PubMed Central - PubMed

ABSTRACT
Following devolution, differences developed between UK countries in systems of measuring performance against a common target that ambulance services ought to respond to 75% of calls for what may be immediately life threatening emergencies (category A calls) within 8 minutes. Only in England was this target integral to a ranking system of 'star rating', which inflicted reputational damage on services that failed to hit targets, and only in England has this target been met. In other countries, the target has been missed by such large margins that services would have been publicly reported as failing, if they had been covered by the English system of star ratings. The paper argues that this case-study adds to evidence from comparisons of different systems of hospital performance measurement that, to have an effect, these systems need to be designed to inflict reputational damage on those that have performed poorly; and it explores implications of this hypothesis. The paper also asks questions about the adequacy of systems of performance measurement of ambulance services in UK countries.

No MeSH data available.


Related in: MedlinePlus