If you battle your way through the Stats Wales web site you can get the actual data series for this figure and it looks like this.

• Apr 09 – 65.5%
• May 09 – 66.5%
• Jun 09 – 66.7%
• Jul 09 – 63.9%
• Aug 09 – 54.6%
• Sep 09 – 67.1%
• Oct 09 – 66.4%
• Nov 09 – 65.8%
• Dec 09 – 59.4%
• Jan 10 – 58.5.%
• Feb 10 – 65.3%
• Mar 10 – 69.2%
• Apr 10 – 70.5%

Aug 09 doesn’t look so great. I bet they had some explaining to do that month, and the last month is really good. If you look at the last four months there has been a definite improvement. Or has there?

If we take these figures and plot them on a run chart we see this:

We can see that the percentages are up and down month by month. Again Aug 09 is bad and the latest is good. But quite jumpy nonetheless.

Now let’s show the mean on the chart to provide a bit of balance:

That just gives us a bit of perspective. We can see the points bouncing above and below the mean. There are some questions that we can ask:

• Why does it seem so variable?
• Is that variation excessive?
• Where is the variation coming from?

A technique which can help us answer this question is called Statistical Process Control. It gives a method to calculate limits from the data. If the data points lie within the limits then the variation is “common cause” i.e. normal variation due to the system and if points lie outside the limits then the variation is “special cause” i.e. due to some one off circumstance that can be pointed to as a particular cause of a very high or low figure.

Let’s apply these limits to our chart.

We can see that all the data points lie within the limits. It is worth emphasising that any data point that lies within the limits is entirely expected and is thus assumed to be due to normal variation within the system.

So we can conclude that the Mar 10 figure of 70.5% is within the limits. There is no special cause and there is no shift in the system and so that month’s figure is nothing special in the same way that the Aug 09 figure of 54.6% is also within the expectations of normal system caused variation.

Therefore the Welsh Ambulance emergency call response rates show no improvement.

In fact next month’s figure would be expected to fall anywhere between the limits of 54.3% and 74.9%. (Perhaps a keen reader could remind me to check up on that!)

So what do the Welsh Ambulance service need to do? They need to change the system to 1) move the average and 2) reduce the variation. It would be good to see a shift like this

Where we can clearly see a change in the system causing not only a shift in the mean but a reduction of variation.

This is the evidence of change that we need, not the clutching at straws that accompanies a month-by-month examination of figures.

But also something worse happens when you can’t judge whether the last month’s figure is due to common variation. If it is good, but within the limits, then the temptation is to congratulate people and to rest on our laurels, while all along nothing has really changed. The opposite case is as bad. If the figure is bad, we go looking to blame someone, to find fault, when again, if it is within the limits it is just due to normal variation. In both cases we are tampering, mistaking common variation for something special and acting on it (congratulating or blaming) when in fact we should just ignore it and look at the system as a whole and how we can change it for the better.

Understanding variation due to a system using the techniques described above is the first step toward taking action that is effective.

Best,

Rob

Before the election in May, the three health spokesmen for the major parties debated on television. Obviously the question of targets came up and Andrew Lansley said he would scrap “politically motivated” NHS targets but keep those that were “clinically justified”.

The problem is that there is no such thing as a clinically justified target. The reason is that targets are arbitrary and make performance worse. That is never clinically justifiable.

A target to have every suspected cancer patient seen within two weeks has behind it a laudable goal – that patients with serious conditions should get the best treatment possible. There is nothing wrong with that goal but everything wrong with having a target.

Study Demand

To achieve the goal you would start by looking at demand. “How many suspected cancer patients do we get every day/week/month?” You would then look at the current system to see how this demand was dealt with and you would measure it from the patients’ point of view. They want to be seen quickly, they want to get the right treatment and have the best outcome possible. Also they want dignity, respect and to be treated like human beings.

If you are thinking of hitting a target your mind is filled with resources, measures and reporting. I went to Exeter to help run a workshop on how to meet the 18 week target from diagnosis to treatment. One of the questions from the floor was about what monitoring should be put in place to spot when patients are just about to breach the target and so fast-track them so they won’t breach. If you put that monitoring in you are moving resource from the core flow of diagnosis and treatment and over to monitoring. If then you add a fast-track expediting of patients just about to breach you add a further complication to the system where suddenly certain patients jump the queue to make the trust’s figures look better. But remember that patient has jumped over other patients who are now delayed and so more likely to get close to breach. If they get close to breaching they will jump the queue and so the cycle continues. You are in effect increasing the variation in the system while at the same time reducing the resources available to the core flow.

Measures of Purpose

The correct approach is not to have monitoring of near breaches but to measure the end-to-end times of patients from diagnosis to treatment, while also measuring the type of treatment given. You will see that the end-to-end time will vary. This variation is normal in any system. Your first job is to remove any abnormal variation until the system is stable (which is not to say it is yet effective – just stable) then work on the flow of the patient through the system to remove the batching, waiting, errors and rework so as to reduce the variation and improve the system to get the time from diagnosis to right treatment as short as possible.

This method will leave any target setting in the dust. Why 18 weeks? Why not 18 days? Why not 18 hours?

Why two weeks for cancer patients? Why not two days?

Method Not Targets

Targets don’t help you to improve, in fact they distract from improvement. People set targets when they have no method. When they have a effective methods, they get results that would be seen as ludicrous to set as a target.

Best,

Rob

Simplistic
Meaningless
Arbitrary
Road blocks
Torture

Simplistic

Targets are simplistic. Targets are a sledgehammer to crack a nut and they come with no context or method. Targets are imposed by managers and governments and they walk away thinking by setting a target that their job is done.

Meaningless

Targets don’t have any relation to the system they are meant to apply to. They can’t have. Targets are not related to purpose. Hitting or missing a target does not give you any new information. Having a target set does not give you any new method to improve it simply gives you something to focus on. And not anything useful. Aiming for a target does not give staff or managers any new understanding of the system. They will be so absorbed trying to hit the target they may well be even more oblivious of what is happening around them.

Arbitrary

An understanding of variation gives the insight that any level at which you set your target is wrong. A study of common cause variation for any measure will give you upper and lower limits within which the system will exhibit predictable variation. Setting a target above or below the limits means the target will always/never be hit (depending on whether higher or lower is better or worse). Setting the target at a level between the limits means that it is virtually random whether the target will be hit in any given period. Therefore there is no reliable way to set a target and hence all targets are arbitrary.

Road Blocks

Targets distract from genuine improvement by sapping energy toward the collecting, analysis and reporting of useless data. Worse, targets also actively get in the way and drag down performance. Think of the 4 hour A&E target that induced one hospital trust to keep patients in ambulances outside A&E, only letting them in once they were sure they could hit the target. Schools are measured and ranked on exam results and so they cheat to get the brightest children in their intake. This is not improving education for all, this is a road block to improving education.

Torture

Targets mean pressure to achieve meaningless, arbitrary, numbers. This is tortuous for staff, managers and inspectors alike.

Staff are judged, rated and rewarded by whether they hit arbitrary targets. They know that they have to apply their minds to hitting the targets when that means they have to degrade overall performance to do so. When staff have to cheat to hit targets that has an effect on morale, self-esteem and respect for their managers who are putting them in the position of having to do these things.

Managers are ravaged by the constant hitting and missing of targets in a seemingly random way. They can never seem to get a good explanation about why a target was missed even though when it is hit they tell everyone they can.

Inspectors often know that what they are doing is ruinous to the performance of the organisation that they are trying to judge. They know that most of the time they are simply assessing compliance to a standard and not looking at an intrinsic ability to give good service. This knowledge must eat away at a person.

Targets are tortuous to users. Using a target driven service will be boring, annoying, frustrating and perhaps dangerous and yet they will be constantly being told that the organisation is hitting or making good progress toward its targets.

So let’s get rid of this culture of the target, SMART or otherwise, and move toward measures that are VVAPID.

Value
Variation
Aligned
Purpose
Informative
Deliver

Best,

Rob