Like a Drunk, Maybe We’re Looking at the Wrong Data

 The proverbial joke: a drunk is looking for car keys under the lamp post; not because the keys were lost there but because the light is better. Similarly, there’s a story out of WWII about a statistician analyzing data on flak damage to bombers with the purpose of determining where to reinforce the armor plating on the plane. The Allies couldn’t add armor everywhere; the bombers would be heavier and consume more fuel and have a smaller bomb payload. After collecting the data from returning bombers and comparing this information to surface area and corresponding ratios, Abraham Wald reported that some areas are disproportionally damaged. However, in a key insight, Wald advised that these were the wrong areas to reinforce. These planes survived damage to those areas. He didn’t have data on the planes that did not return. Where were the lethal hits on those aircraft? He coined this bias towards available information: survivorship bias. We look at the successful companies, for example, and determine they were resilient to certain problems. But perhaps all companies are resilient to certain problems. We don’t know what damage—decisions, policies, behaviors, circumstances—occurred that finally did the unsuccessful companies in.

Dan Simons and Chris Chabris, the co-authors of Invisible Gorilla—which pointed out problems with our attention/focus and our memories—have given us a new book, Nobody’s Fool: Why We Get Taken In and What We Can Do About It. This survivorship bias shows up everywhere and too often consultants sell us the methods that worked for Google, Meta, Apple, IBM, GE, 3M, Walgreen’s… Even Jim Collins and his comparative studies still can’t report on the thousands of companies that go out of business and the reasons why.

I’ve been in companies that really couldn’t answer the question because the information was missing. I’ve seen leaders tempted to use the available data to answer a different question assuming “it’s close enough” or “it’s the metric everyone else uses.” Like overall profitability instead of product line/service line specific. Or applying the same overhead rate for all activities even though resource consumption is lower on legacy products/services.This can be a problem. And proverbially has you looking for car keys under the lamp post, when they’re in the middle of the block. Close enough?