Saturday, January 06, 2007

Puzzles and Mysteries

Malcolm Gladwell wrote a great piece in the New Yorker: OPEN SECRETS: Enron, intelligence, and the perils of too much information. It's being blogged all over, but since I'm still thinking about it 12 hours after reading, I'm posting too.

He distinguishes between puzzles and mysteries, and the role of data in both.

The national-security expert Gregory Treverton has famously made a distinction between puzzles and mysteries. Osama bin Laden’s whereabouts are a puzzle. We can’t find him because we don’t have enough information. The key to the puzzle will probably come from someone close to bin Laden, and until we can find that source bin Laden will remain at large.

The problem of what would happen in Iraq after the toppling of Saddam Hussein was, by contrast, a mystery. It wasn’t a question that had a simple, factual answer. Mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have too little information but that we have too much. The C.I.A. had a position on what a post-invasion Iraq would look like, and so did the Pentagon and the State Department and Colin Powell and Dick Cheney and any number of political scientists and journalists and think-tank fellows. For that matter, so did every cabdriver in Baghdad.

Adding data to a mystery doesn't make it less of a mystery, but smart analysts can bring experience to bear that will point towards possible answers that are more or less likely to be true.

The Enron disaster qualifies as a mystery, and Gladwell's summary of the confused and complex financial details is disquieting. Even financial experts couldn't make heads or tails out of what was going on with the money. I'm unsettled by how easily corporate environments breed mysteries requiring complex analysis both of their internal and external behavior; and how hard it can be to understand and correct corporate mismanagement. People lose their livelihoods all the time now-- how many other Enrons are actually happening with no one noticing?

As a side note, as I read the recap of the Woodward and Bernstein investigation I actually thought, "This is surely going on all the time now. Would anyone be shocked at this in this political era?"


Celestial Z Weasel said...

Enron... I have two books about Enron (corporate failures are one of my hobbies :-)). Obviously the old saw about hindsight being 20:20(or 6:6 as we should say in Europe :-)) applies here but it does strike me that, whilst there were subtleties, there were also some very obvious things - notably
i) the insanity of using mark to market accounting for long term energy contracts
ii) the fact that there were some divisions that clearly were bleeding money from all orifices (the bandwidth trading and water businesses).

However, many things about the way the world works are mysteries (in this sense) to me. The global economy and the old chestnut of 'British house prices' for two.

This comes back to my old hobby-horse about journalists being unable to comprehend the complex technological world in which we live, so people can get away with practically anything.

Celestial Z Weasel said...

Hmm. As you can probably guess I wrote that reply without reading the article (which is a fine article I think), which probably makes it more interesting... he mentions the mark to market accounting and the SPVs. I mentioned the mark to market accounting and the other businesses apart from oil trading. I think the SPVs were probably genuinely too difficult for observers to understand - I don't really know enough to say whether they would have looked suspect to an averagely skilled observer.

However, the businesses that were obviously going nowhere would, I imagine, have been easier to spot.

Talking of over-hyped companies, have you read Clay Shirky on 2nd Life? :-) see

Lynn said...

Thanks for the link... I've been feeling too jaded since my MUDding days to look at Second Life, because I have such a "been there done that" thing about online community. What a jerk I am. But I thoroughly enjoyed his skepticism and the comment thread on it, especially mark pesce's appearance saying "amen brother" as the inventor of VRML. It was like a blast from the critical past.

Grumpy said...

Well, I don't know... The difference between "puzzles" and "mysteries" as described in the article seems arbitrary and artificial. In most puzzles you have all the information you need, you just have to rearrange it (like in logical puzzles) or retrieve the "missing" pieces from somewhere deep in your memory. In general, you don't have to go to the library to solve a puzzle, there has to be an "I knew it!" or "I knew I knew it!" moment, otherwise it's no fun. And mysteries are usually unsolvable, like the mystery of God, life and death and such.

From another perspective, whether the information was withheld or not, it's not as simple as that, you can lie in many ways, you can lie by telling something that is not true, by not telling something that must be told, by telling too much so that most important pieces (whether true or not) are drowned, etc.

I've read a book about Enron, and these guys definitely knew they were pushing it, one of them (Lou Pai) was wise enough to quit early, Skilling also quit, but too late to escape. Fastow was a swindler and a thief (yes, he was making tens of millions on these SPE deals, he and his wife), Skilling lied to auditors, Lay lied to employees (telling them to hold and even buy Enron stock as he was selling it), Fastow plead guilty and the latter two were convicted for multiple counts of securities fraud. Some mystery... About these students from MIT, yes, there were warnings, and even much more serious than from a bunch of undergraduates, but in late 90-s everybody was crazy about this amazing new economy for which classical approaches don't work, all these dot-com bubbles grew (both in size and in stock) without making any profit for years and everybody was happy, well, until the bottom fell off... And Enron was a leader in this "new economy".

In all, this particular article strikes me like a revisionist version and quite close to whitewash as far as Enron is concerned. And about investment community and why they didn't ring the bell earlier, well, long ago John Maynard Keynes compared stock market to a beauty contest, where it's rational to follow judgements not of your own, but those of an average participant. That is, as long as average investment Joe expects shares of X to grow, they will grow. So for a "smart" investor there is a choice, either to opt-out early and see all these fools making money while he doesn't or follow the crowd and try to catch that peak, after which everything collapses and that's a very risky strategy but tempting nonetheless.

Lynn said...

Thank you, Grumpy, that was an excellent comment... One of the reasons I've been puzzling over the Gladwell article is that I'm also not certain I buy the difference between puzzles and mysteries. After all, even in puzzles where a little more data helps solve it, the interpretation of the data and the understanding of what KIND of data are still important, which blurs the distinction.

Good point about being a revisionist take on Enron, too. I would certainly like to believe it's more black and white than Gladwell chose to paint it. I think I'll read more on the topic myself (apart from the public email archive :-)