Risk based testing is one of the many testing methodologies that has not delivered promised returns. Its not really hard to understand why. Ordering risks requires knowing two things about each risk: the likelihood of the risk occurring and impact if it does. For software, neither of these is knowable.
If bugs were distributed randomly, the likelihood of a risk occurring would simply reflect the usage model of the software. But we know that bugs are not distributed randomly. Instead they cluster. They cluster in the code of a particular developer. The cluster in a design that doesn't fully meet its need. They cluster in how an environmental issue is handled across different areas of the software. Bugs cluster for all sorts of reasons. And we can't identify where these clusters are until we have already tested and found the bugs.
Difficult as the problem of likelihood is, predicting impact is much harder. The severity of a bug may well be random and this randomness thwarts us. The severity of the bug is not related to the importance of the functionality the code implements. Even bugs in remote byways of a system can cause it to crash or corrupt data. The honest answer to the question of what can take the system down is any line of code in it.
Sure, we find bugs using risk based techniques. We find bugs using any testing technique. Does it really result, though, in the identification and removal of our riskiest bugs? The evidence strongly suggests not.
No comments:
Post a Comment