Zero defects does not mean the problem is solved

One of the common mistakes made by companies is the assumption that the lack of defects means the problem has gone away.

Especially in low volume companies, when a particular problem does not reoccur in a small sample, it is easy to claim victory and move on to a new problem. However, without adequate sample size, that can be a mistake.

Determining adequate sample size depends on two factors: how confident you want to be, and how big the problem was prior to the improvement.

Let's assume a defect was occurring in a process approximately 20% of the time. The team comes up with a solution and implements it. 10 more units are produced from the process with the "new solution" and there are zero defects. Success! Actually, not so fast...

How likely were we to get zero defects, if nothing was improved? It turns out, the odds are pretty good. We can calculate that exactly using Minitab.

Go to Calc --> Probability Distributions --> Binomial (Pass/Fail data) 

Let's say the failure rate is 20%, so enter 0.2 for Event Probability and 10 for # of trials, with a input constant = zero (number of failures) 

Binomial with n = 10 and p = 0.2 
x  P( X <= x )0     0.107374 

This output means that there is only a 10-11% chance of seeing zero failures in 10 samples, if we still have a failure rate of 20%. We ideally would like to see less than 5% chance.

If the failure rate is higher at 50%, then the chance of seeing zero failures would be much lower (since it's pretty likely to have a failure show up under normal conditions).

Binomial with n = 10 and p = 0.5 
x  P( X <= x ) 
0    0.0009766 

If the failure rate is only 10%, then the chance of seeing zero failures would be higher (since it's less likely to have a failure under normal conditions) 

Binomial with n = 10 and p = 0.1 
x  P( X <= x ) 
0     0.348678 

34% is too high of a risk to conclude that the problem went away. If you get a probability less than 5% (such as with a failure rate of 50%), then you can conclude that the problem likely has gone away. If greater than 5% (such as with failure rate of 10% or 20%), then we don't have enough samples to "claim victory". We would need to collect more samples. You can keep adding trials to Minitab until it gives you a sample size that shows a probability less than 5% (less than 0.05). 

In this example, with 10 samples and zero failures observed, the original failure rate should have been at least 25% before the problem was fixed, in order to statistically say that the problem has been resolved. If your previous failure rate was less than that (say 15%), then you will need more samples (trials) before you can feel confident the problem has been resolved.

Minitab has another method for figuring out the correct sample size and confidence. We will cover that in a later discussion (or email us if you need help).

Don't have Minitab? The calculations are easy manually for zero defects...

• Probability of zero defects with 10% failure rate = (0.9 * 0.9 * 0.9 * 0.9 * 0.9 * 0.9 * 0.9 * 0.9 * 0.9 * 0.9) = 0.3486

• Probability of zero defects with 20% failure rate = (0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8) = 0.1073

Next time you are reviewing your data, make sure you have the statistical confidence to say that the problem has gone away, so you don't get embarrassed later when it returns.