If you battle your way through the Stats Wales web site you can get the actual data series for this figure and it looks like this.

• Apr 09 – 65.5%
• May 09 – 66.5%
• Jun 09 – 66.7%
• Jul 09 – 63.9%
• Aug 09 – 54.6%
• Sep 09 – 67.1%
• Oct 09 – 66.4%
• Nov 09 – 65.8%
• Dec 09 – 59.4%
• Jan 10 – 58.5.%
• Feb 10 – 65.3%
• Mar 10 – 69.2%
• Apr 10 – 70.5%

Aug 09 doesn’t look so great. I bet they had some explaining to do that month, and the last month is really good. If you look at the last four months there has been a definite improvement. Or has there?

If we take these figures and plot them on a run chart we see this:

We can see that the percentages are up and down month by month. Again Aug 09 is bad and the latest is good. But quite jumpy nonetheless.

Now let’s show the mean on the chart to provide a bit of balance:

That just gives us a bit of perspective. We can see the points bouncing above and below the mean. There are some questions that we can ask:

• Why does it seem so variable?
• Is that variation excessive?
• Where is the variation coming from?

A technique which can help us answer this question is called Statistical Process Control. It gives a method to calculate limits from the data. If the data points lie within the limits then the variation is “common cause” i.e. normal variation due to the system and if points lie outside the limits then the variation is “special cause” i.e. due to some one off circumstance that can be pointed to as a particular cause of a very high or low figure.

Let’s apply these limits to our chart.

We can see that all the data points lie within the limits. It is worth emphasising that any data point that lies within the limits is entirely expected and is thus assumed to be due to normal variation within the system.

So we can conclude that the Mar 10 figure of 70.5% is within the limits. There is no special cause and there is no shift in the system and so that month’s figure is nothing special in the same way that the Aug 09 figure of 54.6% is also within the expectations of normal system caused variation.

Therefore the Welsh Ambulance emergency call response rates show no improvement.

In fact next month’s figure would be expected to fall anywhere between the limits of 54.3% and 74.9%. (Perhaps a keen reader could remind me to check up on that!)

So what do the Welsh Ambulance service need to do? They need to change the system to 1) move the average and 2) reduce the variation. It would be good to see a shift like this

Where we can clearly see a change in the system causing not only a shift in the mean but a reduction of variation.

This is the evidence of change that we need, not the clutching at straws that accompanies a month-by-month examination of figures.

But also something worse happens when you can’t judge whether the last month’s figure is due to common variation. If it is good, but within the limits, then the temptation is to congratulate people and to rest on our laurels, while all along nothing has really changed. The opposite case is as bad. If the figure is bad, we go looking to blame someone, to find fault, when again, if it is within the limits it is just due to normal variation. In both cases we are tampering, mistaking common variation for something special and acting on it (congratulating or blaming) when in fact we should just ignore it and look at the system as a whole and how we can change it for the better.

Understanding variation due to a system using the techniques described above is the first step toward taking action that is effective.

Best,

Rob

Simplistic
Meaningless
Arbitrary
Road blocks
Torture

Simplistic

Targets are simplistic. Targets are a sledgehammer to crack a nut and they come with no context or method. Targets are imposed by managers and governments and they walk away thinking by setting a target that their job is done.

Meaningless

Targets don’t have any relation to the system they are meant to apply to. They can’t have. Targets are not related to purpose. Hitting or missing a target does not give you any new information. Having a target set does not give you any new method to improve it simply gives you something to focus on. And not anything useful. Aiming for a target does not give staff or managers any new understanding of the system. They will be so absorbed trying to hit the target they may well be even more oblivious of what is happening around them.

Arbitrary

An understanding of variation gives the insight that any level at which you set your target is wrong. A study of common cause variation for any measure will give you upper and lower limits within which the system will exhibit predictable variation. Setting a target above or below the limits means the target will always/never be hit (depending on whether higher or lower is better or worse). Setting the target at a level between the limits means that it is virtually random whether the target will be hit in any given period. Therefore there is no reliable way to set a target and hence all targets are arbitrary.

Road Blocks

Targets distract from genuine improvement by sapping energy toward the collecting, analysis and reporting of useless data. Worse, targets also actively get in the way and drag down performance. Think of the 4 hour A&E target that induced one hospital trust to keep patients in ambulances outside A&E, only letting them in once they were sure they could hit the target. Schools are measured and ranked on exam results and so they cheat to get the brightest children in their intake. This is not improving education for all, this is a road block to improving education.

Torture

Targets mean pressure to achieve meaningless, arbitrary, numbers. This is tortuous for staff, managers and inspectors alike.

Staff are judged, rated and rewarded by whether they hit arbitrary targets. They know that they have to apply their minds to hitting the targets when that means they have to degrade overall performance to do so. When staff have to cheat to hit targets that has an effect on morale, self-esteem and respect for their managers who are putting them in the position of having to do these things.

Managers are ravaged by the constant hitting and missing of targets in a seemingly random way. They can never seem to get a good explanation about why a target was missed even though when it is hit they tell everyone they can.

Inspectors often know that what they are doing is ruinous to the performance of the organisation that they are trying to judge. They know that most of the time they are simply assessing compliance to a standard and not looking at an intrinsic ability to give good service. This knowledge must eat away at a person.

Targets are tortuous to users. Using a target driven service will be boring, annoying, frustrating and perhaps dangerous and yet they will be constantly being told that the organisation is hitting or making good progress toward its targets.

So let’s get rid of this culture of the target, SMART or otherwise, and move toward measures that are VVAPID.

Value
Variation
Aligned
Purpose
Informative
Deliver

Best,

Rob