Saturday, October 8, 2011

On STARWest 2011

"Agile" is everywhere. Agile, on the other hand, is continues to be adopted only fitfully. For many attendees, agile meant the addition of iterations and standups and not much else. Practitioners seem to understand that this is not actually working so sessions about making agile work were very popular.

The executives who presented during the leadership summit seemed to be really out of touch with agile. You really can't claim to be agile when you have weekly status reports delivered to a central PMO and you had better be making your weekly progress "or else". The contempt for agile by the next speaker was almost dripping. It should come as no surprise that executives are clueless... I mean out of step with the current state of practice.

Pockets of traditional development and test remain. There were several tutorials from SQE instructors that have probably not changed since the early 1990s. This was much less true in the sessions where there were only a couple of sessions from the Military/Industrial/SEI faction.

Acceptance Test Driven Development (ATDD) was a hot topic. I think its supporters fail to do it justice. ATDD changes how we do specification and provides a mechanism for demonstrating that the software has implemented the intended functionality. But fulfilling requirements is not the same as not having bugs. ATDD is a technique for preventing bugs not a technique for detecting them. Practitioners intuitively understand this which accounts for the resistance of many test professionals to ATDD. It is unfortunate because it is unnecessary.

Another source of resistance to ATDD is its often incorrect implementation. Done correctly, ATDD means specifying the story's tests, preferably in an executable framework, and then developing the code to make the tests run one by one. It is TDD in the large. You know what still needs to be completed on a story by the tests that are still failing. In fact, the percentage of passing acceptance tests may be a more accurate measure of progress during a sprint than the hours remaining in the sprint burndown. (I think it would be interesting to put both measures on a single, highly visible, graph.)

Exploratory testing was also popular. It seems fair to say that one theme of the conference is that test cases are out (or at least limited to the things that can be automated) and exploratory testing is in. It is not hard to understand why exploratory testing is so popular. It encourages testers to do more of the testing they most enjoy and best do. It also helps that the practice has charismatic champions and is amenable to easy demonstration. The various exploratory demonstrations in the test lab were surprisingly popular given the limited space and that attending meant missing prepared sessions. Practitioners resistant to ATDD should consider exploratory testing as a complementary practice.

Test automation is something that everyone seems to do but everyone also seems to be dissatisfied with the results and wonders whether they should do more. Tool vendors keep pushing the concept that we can make this easy enough for the business analysts to do so testers won't have to. (Why you would try to sell that message to a room full of testers is beyond me.) Perhaps they should get together with the ATDD folks since their pitches now seem more directed at demonstrating that the functionality was implemented rather than finding bugs. I suspect they spend too much time talking to CIOs. As the vendors work harder and harder to make test automation accessible to non-technical testers, technical testers like me will have more incentive to find other solutions.

I wonder if some of the uneasiness around automation reflects a recognition that we are simply not finding and fixing enough bugs with automated tests. Sure, we find regression issues but these are, or should be, only a small proportion of the bugs. For years, I have talked about the role serendipity plays in finding bugs. You are running a manual test and notice something wrong completely unrelated to the purpose of the test. Automated tests completely fail to find these kinds of problems. (To which the exploratory testing folks would reply, "Of course.")

Mobile devices and cloud computing are new software platforms that were also program topics. Cloud computing offers some real opportunities for testing. Building environments to support realistic performance testing has always been expensive and these environments usually end up underutilized. Cloud computing offers a solution. Environments that can be built to scale when needed and released when not. Cloud computing also presents unique testing challenges. Services built in the cloud use highly parallel architectures that are susceptible to all sorts of concurrency issues which are difficult to test. Testing software for mobile devices seems more familiar. Its really just a configuration problem but the rapid improvement of these platforms means that the number of configurations and the difference in their capabilities is larger than most of us are used to.

Controversy came in the form of James Whittaker's keynote "All That Testing Is Getting in the Way of Quality." In it, he claimed testing was dead. Or, in more subtle terms, that the practice of testing to find and fix bugs had not resulted in significant improvements in software and could, in any case, be done more effectively, thoroughly, and cheaply by crowd-sourcing it. I think we can safely say by the hisses that accompanied any further mention of google and the topics of table conversations over the rest of the week, that he struck some sort of nerve. Since I had entered the conference with that very point of view, it was fascinating to watch the reaction. On this, there will be much more in subsequent posts.

The other question that seemed to be on many minds is "what's new here?" The answer from my part of the world is not much. I believe that James Bach did his original presentation on good enough software and rapid testing in the mid 90's. Agile, too, has passed its 15th birthday. ATDD is new but (perhaps justifiably) is not fully accepted as a test technique by practitioners. The software platforms are new, but the testing challenges are not.

I last attended a STAR conference in 1993. The differences were striking. Sessions had an air of discovery about them. Concepts like risk based testing or model based testing were new and promising. The internet had not yet happened so new information traveled more slowly. Conferences were the conduits where concepts transitioned from IEEE and ACM papers to practitioners. There were more sessions given by researchers from either academic or industrial labs. (Do those kinds of labs even exist anymore - outside of google that is?) Everything was open to investigation. I remember attending one very interesting session on regression test selection. Now we just run 'em all. Another session suggested that we all needed to be doing mutation testing to prove our test sets. And another promised automated test generation from program analysis.

There is a deeper reality here too, one that Dr. Whittaker alluded to. The practice of testing hasn't really changed all that much since the mid-90s. Sure, we've repackaged some of the things that we all knew in, hopefully, more effective forms. Long before ATDD, testers understood that tests were a much better specification than the specs we usually got. I remember suggesting that we just write the specs as tests years ago. Exploratory testing? Have we ever actually not done exploratory testing whether we told our bosses or not? We use the same tools, models, strategies, The design strategies in Lee Copeland's tutorial "Key Test Design Techniques" are the same ones that I learned from Dr. Boris Beizer at STAR in 1992. And the simple fact is that the application of these techniques has not, on the whole, resulted in software that works all that much better. As Dr. Whittaker suggests, the drivers of whatever improvements we've had in software have not come from improvements in testing. They could not, since there haven't really been any improvements in testing. And I believe that the industry is (or should be) entering a period of evolution to diminish the dependence on testing to develop working software. And that is the jumping off point for the posts to come.

7 comments:

  1. The point I try to stress is that the goal of test automation is to create time for exploratory testing.

    In that perspective, a good measure of the effectiveness automation is that it:

    1) doesn't take more time than manual testing would -- this allowing time for manual exploratory testing,

    2) catches and prevents defects. And it should do it very quickly. Ideally with each build.

    If you're not accomplishing both of these, I'd question the value you're getting from automation, but even more, the value you're missing from neglecting manual tests.

    Automation should primarily be used for regression testing & other scenarios that require repetition and accuracy. But there's also a balance that it takes time to get to the point where automation pays off, both in experience and in infrastructure. So while effective automation is essential to successful agile, the process of implementing automation is difficult to accomplish in an Agile way.

    ReplyDelete
  2. I agree with your points. I have recently started to think differently about test automation. Instead of thinking of it as a technique for finding bugs, I consider it more of a technique for being notified of changes. When tests that were passing are now failing, you know that something has changed. You don't, yet, know whether it is a bug.

    ReplyDelete
  3. It sounds like conferences are, in the internet age, becoming better venues for technical conversation (rather than just presentation) and brainstorming, presenting problems, getting experience at skills that are hard to transmit without conversation face-to-face, and a venue for discussions on how to work on known problems.

    I'm uncomfortable with the idea testing hasn't improved software, thus it's dead. Testing is about providing information quickly to help make business decisions (as Kaner and Bach might point out--If I understand them). To that end, testing is very much alive. I suspect that there's a mental disconnect here, blaming software testing for not making software perfect. I think it's enabled a different kind of value: faster delivery and more confident shipping, better risk management, but not perfect software--which was probably never the unconscious goal anyway. Any thoughts on testing as helping to accelerate delivery instead of improve customer-facing quality, in the face of harsh competition?

    ReplyDelete
  4. From my point of view (and I will write at more length about this), I don't think testing is dying so much as expectations are coming in line with capabilities. No amount of testing will make a bad product good. Fixing bugs without fixing process won't help a team improve. Testing has a place as part of a development effort but the road to real improvement lies elsewhere.

    ReplyDelete
  5. Which processes do you suggest, for which contexts? How qualified are testers to teach managers/developers (dictate?) how to manage a company's processes?

    I suspect expectations are different at your company than mine. I interviewed with a company that expected testers to have rather basic skills. But I can also clearly see that expectations vary depending on a company and its processes. I wonder if the real question isn't how to fix processes, but how many kinds of tester positions there are out there? Perhaps the industry needs more words for the types of testers. For example, we already have a word for usability testers: Usability Engineers. They have testing-like abilities, and the sciences around their craft are rather well developed. There are other skill sets, too.

    So, that's a long way of asking: what expectations do you see at your company, and why do you think they apply broadly to most other testers/QA folks? And regarding capabilities: aren't those fairly fluid? Don't testers change their abilities according to demand, or leave for another career? I suppose supply (available capabilities) also influences the craft: perhaps you can comment on this further?

    "No amount of testing will make a bad product good.": I agree. Testing doesn't change a product, but it does give managers and developers information they need to decide what to do with the product (throw it out, revise it, or live with it because it kinda works for now, etc.)

    "Fixing bugs without fixing process won't help a team improve": so are you proposing that testers have a responsibility to help developers improve? Could we also say developers have a responsibility to help testers improve, and tell them how to do their jobs better? I think there's pros and cons, and humility is a prerequisite. Perhaps this is why Bach's satisfice model stresses developer relations? I don't know.

    ReplyDelete
  6. I have longer posts still to come but I will claim two things. First, that in all but the most exceptional cases, the decision to release a product on a schedule has been made before testing starts and test results don't actually matter. Second, if we can't really make the product better, and we can't really stop or delay the release, then we better help the team develop software better or we have no role at all.

    ReplyDelete
  7. Do test results have to affect the current release to matter? Do you really think test results don't matter except in unusual circumstances? If so, does that mean that managers don't value quality, or that the information they're getting about quality isn't valuable (timely, accurate, helpful, etc...)? Do your managers actually disregard quality completely, or is it a graded approach?

    I suspect most managers care a great deal about software qualities--as do the CEO's/directors/investors they work for. In my experience, managers do lose sleep over their work and their jobs--even if it's not visible to individual testers. Maybe it's different for you. I don't expect my managers to fix every bug--successful products have bugs, while perfectionism leads to vaporware. However, in my experience my testing has been done with enough time to do something about my findings. Perhaps your experience is different? I wonder why?

    I don't think testers have ever been able to stop or delay releases, generally. That's a manager's job.

    If managers are releasing despite the information testers are providing, maybe testers aren't providing the right information, or at the right time? Perhaps they don't have the ability to do so--which itself is a test result, right?

    I hope my questions are useful. I haven't had the same experience as you, or worked for your managers. I have seen managers avoid testing, but those I've tested for seem to value the information I provide. Sometimes I feel more like I work for developers than managers--actually, most of the time that's true.

    ReplyDelete