It really doesn’t matter when you plan to release, nor how much testing has occurred – someone will always be worried that it was not enough. Whether you are on an agile schedule of releasing every 4 – 8 weeks, or a more traditional waterfall process of 6, 12 or 18 month releases, it always seems that there was not enough testing time to really feel comfortable shipping a product. If you test too long, you will miss your window of opportunity in the market, and if you don’t test enough you risk shipping a bad product.
So, how much is enough, and when do you know it? Test automation helps, but still nothing compares to putting hands on the product and getting a good feeling about it. There is much more subjectivity to testing than most people realize.
It has been said that good managers make decisions when they have only 40% – 60% of the data needed for that decision. The reason being that if they wait for all of the data, they usually miss the opportunity, but if they make the decision with less than 40% of the data, they risk making the wrong decision.
So, does this concept apply to testing as well? It certainly seems to fit (if you wait too long, you miss the window) – but then, if you ship a product where only 40% – 60% of the features have been tested, that doesn’t sit well with anyone.
The bottom line here is that the decision to ship a product always comes with risk. You can never test all possible scenarios, because you never know exactly how someone will use your product. And you really never know exactly what features have been affected by the recent code changes, even though you might think you do. You can mitigate that risk with automated testing that has significant coverage, and you can also reduce the risk with acceptance-level tests that cover the most-often used features in the product.
However, there will always be risk when shipping new code – regardless of whether you are using agile methodologies or not. But, the agile world gives you two significant benefits: first, the amount of code change for any release is minimized, so the risk of something slipping through is smaller, and secondly with short iterations you can react faster to a “bad” release – which you hope you never have to do, but the reality is that there are just too many variables in todays software environments to avoid it completely.