It is reported by the BBC that the Welsh Ambulance Service has improved its response times for Category A emergency calls. These calls are targeted to have an ambulance arrive within 8 minutes 65% of the time. The BBC article states that the percentage has been above 65% for the last three months and that the service “shows improvement”.
If you battle your way through the Stats Wales web site you can get the actual data series for this figure and it looks like this.
- Apr 09 – 65.5%
- May 09 – 66.5%
- Jun 09 – 66.7%
- Jul 09 – 63.9%
- Aug 09 – 54.6%
- Sep 09 – 67.1%
- Oct 09 – 66.4%
- Nov 09 – 65.8%
- Dec 09 – 59.4%
- Jan 10 – 58.5.%
- Feb 10 – 65.3%
- Mar 10 – 69.2%
- Apr 10 – 70.5%
Aug 09 doesn’t look so great. I bet they had some explaining to do that month, and the last month is really good. If you look at the last four months there has been a definite improvement. Or has there?
If we take these figures and plot them on a run chart we see this:
We can see that the percentages are up and down month by month. Again Aug 09 is bad and the latest is good. But quite jumpy nonetheless.
Now let’s show the mean on the chart to provide a bit of balance:
That just gives us a bit of perspective. We can see the points bouncing above and below the mean. There are some questions that we can ask:
- Why does it seem so variable?
- Is that variation excessive?
- Where is the variation coming from?
A technique which can help us answer this question is called Statistical Process Control. It gives a method to calculate limits from the data. If the data points lie within the limits then the variation is “common cause” i.e. normal variation due to the system and if points lie outside the limits then the variation is “special cause” i.e. due to some one off circumstance that can be pointed to as a particular cause of a very high or low figure.
Let’s apply these limits to our chart.
We can see that all the data points lie within the limits. It is worth emphasising that any data point that lies within the limits is entirely expected and is thus assumed to be due to normal variation within the system.
So we can conclude that the Mar 10 figure of 70.5% is within the limits. There is no special cause and there is no shift in the system and so that month’s figure is nothing special in the same way that the Aug 09 figure of 54.6% is also within the expectations of normal system caused variation.
Therefore the Welsh Ambulance emergency call response rates show no improvement.
In fact next month’s figure would be expected to fall anywhere between the limits of 54.3% and 74.9%. (Perhaps a keen reader could remind me to check up on that!)
So what do the Welsh Ambulance service need to do? They need to change the system to 1) move the average and 2) reduce the variation. It would be good to see a shift like this
Where we can clearly see a change in the system causing not only a shift in the mean but a reduction of variation.
This is the evidence of change that we need, not the clutching at straws that accompanies a month-by-month examination of figures.
But also something worse happens when you can’t judge whether the last month’s figure is due to common variation. If it is good, but within the limits, then the temptation is to congratulate people and to rest on our laurels, while all along nothing has really changed. The opposite case is as bad. If the figure is bad, we go looking to blame someone, to find fault, when again, if it is within the limits it is just due to normal variation. In both cases we are tampering, mistaking common variation for something special and acting on it (congratulating or blaming) when in fact we should just ignore it and look at the system as a whole and how we can change it for the better.
Understanding variation due to a system using the techniques described above is the first step toward taking action that is effective.