Grey Swan Defocusing

My continued musings on The Black Swan by Nassim Taleb, but this time with my software tester hat on...

All software bugs are not created equal. A small, hopefully very small, number of them have extreme, possibly even catastrophic, impact to a system’s stakeholders. We don’t see it coming, yet later claim that all the signs were there. These are Black Swan bugs.

By definition, the Black Swan bug escapes to the field. My only defense is preparation for the consequences. However, to abuse the metaphor, not all non-white swans are truly black. I may be able to avoid unnecessary damage to my customer, and to my reputation, by dedicating a small portion of my testing to hunting grey swans. When it’s time to take a step back from my testing, perhaps I can translate Taleb’s contributing factors into defocusing heuristics...

My Grey Swan Defocusing Checklist

Confirmation Bias - Are we confirming a specification, claim, example or observation without questioning or exploring it?

• Try contradictory patterns of test data.
• Try drastically different patterns of test data.

Narrative Fallacy - Are we buying a story because it’s convenient?

• Skip and reorder steps in common sequences.
• Attempt to break relationships.

Ludic Fallacy - Are we assuming that everyone plays by the rules?

• How might internal entities circumvent the rules? (business rules, operational rules, project rules)
• What forces might be exerted on the system by external entities?
• Are we trusting a tool or environment that’s hiding something from us?

Beware the Scalable - Are we underestimating bug severity?

• Repeat errors or situations that are expensive to process.
• Look for ‘Perfect Storms’ - combinations or sequences of events that cause severe problems.
• Exhaust resources - what happens when that log fills up?

Silent Evidence - Are we ignoring, or even discarding, pertinent information?

• Look at losers - discarded test failures, discarded test cases, especially controversial ones.
• Have we glossed over something that might cause a problem?
• What are we not observing?