On Bugs

Another debate in the Test community is what the appropriate response should be to bugs found during testing.  One camp believes that not all bugs should necessarily be fixed.  As bugs are filed, some representative of the business (product owner, product manager, etc.) prioritizes them and determines which bugs should be fixed.  In many organizations, this is accompanied by test exit criteria expressed in terms of bugs that remain open: no high priority bugs, and some number of of lower priority bugs. This is also often accompanied by a queue of bugs to be fixed due to the delay introduced to prioritizing them.

A different school believes that all bugs should be fixed.  Immediately.  When a bug comes in, work on some new feature stops and a developer is assigned to fix the bug.  In this way, software is ready to release when the functionality is done.  This paragraph is shorter, because the rule is a simpler even though it usually produces the response that you can't possibly fix all your bugs when you develop software.

Finally, the agilists seems to straddle both camps.  Produce no bugs but let the business prioritize the ones you do.

Personally, I believe that no bugs is the most responsible position.  First, let me be clear, when I talk about bugs, I mean the kinds of issues where the software does not do what it is supposed to do or fails in ways that the user would consider an error - like going down or destroying data.  Software has lots of other kinds of issues.  Sometimes it does what is it supposed to do but what it is supposed to do is not actually useful.  Sometimes, it does what it is supposed to do, but in too complicated a way.  Bugs, however, are developer errors that should be fixed.

When you hand over the responsibility for determining whether a bug gets fixed to the business, you assume that the incorrect behavior is the only reason to fix a but.  Its not.  One thing we know about bugs is that they cluster.  Finding one increases the likelihood that there are others that we haven't found yet.  And these other bugs may be worse than the one we found.  Counting on further testing to find them?  That's just playing Russian roulette.  I've tried one chamber and its empty and then another and its empty, that must mean all the chambers are empty, right?  You can't prioritize bugs by their symptoms, you need to understand their causes.  Which means you need to do the hard bit anyway, you need to debug the bug.  Every bug.

Debugging bugs has another benefit, it teaches developers how to stop coding them.  It is a strange property of software development that while we try to avoid coding the same solution twice, we use the same programming patterns over and over again.  The consequence of this is that any mistake a programmer makes is likely to be repeated over and over again.   We found this bug, won't testing find all those others?  That's just another game of Russian roulette.  You can't find all the bugs,  you can't even find all the important ones.  So you had better learn to stop creating them.

Finally, a bug is a broken window.  When you don't fix it, when you have other bugs that you also don't fix, you are creating a social norm that bugs aren't all that important.  That working software is not all that important.  And when you tell developers that working software is not important, you damage their morale and effectiveness as a team.  I do not believe that this is a choice that the business gets to make.  Agile, at least Scrum and XP, make the distinction between decisions that the business gets to make and decisions that engineering gets to make.  The business does not get to make decisions about the engineering process and question of whether or not to fix bugs is an engineering process decision.  It makes no more sense to have the business choose which bugs to fix than it does to have the business choose whether to do test driven design or choose which code review issues to address,

That's fine in theory, you may say, but it can't work in practice.  Let me ask you this.  I often hear the complaint that there aren't enough test resources.  Most companies have one tester for every 3, 4, or more developers.  How is it that one testers is finding more bugs than those 3, 4, or more developers can fix?  Could it be because the software contains too many bugs that are too easily found?  Not being able to fix all the bugs that testers find reflects deeper problems on the team.  Prioritizing bugs won't fix those problems, although it may allow them to linger.  It is a crutch that teams use to avoid having to improve.

You may think that you can't commit to fixing all the bugs.  Others in the industry would disagree.  Jeff McKenna, one of the founders of Scrum discussed fixing all bugs in a recent video interview.  Joel on Software discussed Microsoft adopting a zero defects methodology in his 12 Steps to Better Code.  In "The Art of Agile Development," James Shore discusses several XP projects that adopted a no bugs philosophy.  If these teams can do it, so can yours.