Grey Swan Defocusing

My continued musings on The Black Swan by Nassim Taleb, but this time with my software tester hat on...

All software bugs are not created equal. A small, hopefully very small, number of them have extreme, possibly even catastrophic, impact to a system’s stakeholders. We don’t see it coming, yet later claim that all the signs were there. These are Black Swan bugs.

By definition, the Black Swan bug escapes to the field. My only defense is preparation for the consequences. However, to abuse the metaphor, not all non-white swans are truly black. I may be able to avoid unnecessary damage to my customer, and to my reputation, by dedicating a small portion of my testing to hunting grey swans. When it’s time to take a step back from my testing, perhaps I can translate Taleb’s contributing factors into defocusing heuristics...

My Grey Swan Defocusing Checklist

Confirmation Bias - Are we confirming a specification, claim, example or observation without questioning or exploring it?

• Try contradictory patterns of test data.
• Try drastically different patterns of test data.

Narrative Fallacy - Are we buying a story because it’s convenient?

• Skip and reorder steps in common sequences.
• Attempt to break relationships.

Ludic Fallacy - Are we assuming that everyone plays by the rules?

• How might internal entities circumvent the rules? (business rules, operational rules, project rules)
• What forces might be exerted on the system by external entities?
• Are we trusting a tool or environment that’s hiding something from us?

Beware the Scalable - Are we underestimating bug severity?

• Repeat errors or situations that are expensive to process.
• Look for ‘Perfect Storms’ - combinations or sequences of events that cause severe problems.
• Exhaust resources - what happens when that log fills up?

Silent Evidence - Are we ignoring, or even discarding, pertinent information?

• Look at losers - discarded test failures, discarded test cases, especially controversial ones.
• Have we glossed over something that might cause a problem?
• What are we not observing?

WOPR15

I recently attended WOPR15 (WOPR = Workshop on Performance and Reliability, http://performance-workshop.org). The theme of this workshop was “Building Performance Test Scenarios with a Context-Driven Approach.” It’s amazing what you learn when a couple dozen or so bright people get together to share experiences and vigorously discuss and debate ideas. This blog will fall woefully short of portraying the experience effectively, much like looking at a picture of the Grand Canyon, but here goes...

I have to applaud the content owner, Michael Bonar, and the WOPR organizers on selecting a line-up of presenters that came at this challenging theme from a variety of creative, sometimes non-intuitive, angles. This made it even more interesting to see the emergence of patterns, or sub-themes, across these disparate stories.

From my perspective, the most compelling of these sub-themes was “Testing in Production.” Companies like FaceBook, eBay, Google and Microsoft support operations of such massive scale that it is not feasible, or cost effective, to support production-scale test environments. Instead, they invest in processes and tools that allow them to use their production systems for testing.

Another prevalent sub-theme was “Just get started.” There seems to be a consensus among these leading practitioners that waiting for all questions to be answered, or all forms to be filled out, is a trap (frequently self-imposed) that hand-cuffs testers and puts them under extreme pressure later. Start exercising the system as soon as you can. Maximize your ability to plan, design, test and report concurrently.

Yet another significant theme touched upon by several of the presenters was “Have a plan, but adjust aggressively.” Frequently something happens in the midst of a testing effort that will lead you away from the plan if you pursue it. Good! That’s why we test! We are finding potential problems – those problems don’t always respect your plans.

Some other nuggets:

• Luck and intuition may play a role in the success of your testing effort, but an exploratory approach, executed by skilled testers and test managers, positions you to harvest that luck and intuition. (My take on Jon Bach’s experience report (ER)) 
• Great testing can become a selling point for your product. (from Paul Holland’s ER) 
• What could you do if you were not afraid? (A brave attitude toward development and deployment - from Goranka Bjedov’s ER) 
• Focus on wait times and queue lengths when looking for problems, not so much on CPU and memory utilization. (also for Goranka’s ER) 

Like I said before, I cannot do this workshop justice in a short blog entry. There were several other presenters and many more interesting points of discussion. I continue to derive tremendous value from this format. This is focused experience sharing, discussion and debate of our practice at its finest. The next WOPR, which is WOPR16, will be hosted by my employer, Progressive, and I am the content owner. The theme is “The Intersection of Performance and Functional Testing.” A call for proposals will go out in January. The conference dates are April 28 – 30, 2011.