Can Usability Scale Up?

At last week's CHI conference, there was a panel "debate" on usability that I didn't think entirely effective, in that it was Jared Spool being provocative and little interesting disagreement from Eric Schaffer of Human Factors International. But I did find it challenging and thought-provoking to hear Jared complain (yet again) about this stuff, so I'm recapping here.

Jared's contention is that usability departments don't scale -- hiring more usability professionals doesn't make for more usable products. He trots out dubious statistics about big name companies (MS) employing huge numbers of usability folks and still having very poor websites; and hotshot firms like Amazon with few or no usability staff producing famously usable, standards setting sites.

His points come down to these, from my summary notes:

1. More usability staff doesn't necessarily (or often?) equal better usability. (But his numbers from various companies were faulty, and people called him on this.)
2. Good usability often means good design culture, not good testing culture. More people doesn't mean good design.
3. ROI isn't calculated and monitored by usability departments; this is dangerous in riffing climates.
4. Are we an engineering discipline or a craft? We want to claim both and don't provide the mechanism and support for either for our profession. "Engineering" means standard skillsets and repeatable results; "craft" means some people are just better than others and portfolios matter.
5. Process quality in usability is important.
6. When usability is effective, are we effective because of our methods and skills and training, or because a company that hires us is thinking about new things and considering them at the same time and could do it just as well without us: customer needs and feedback, firm marketing goals, good beta testing feedback, design process, quality.

Item 2 is his most unexamined, unsupported point and yet the one I also agree with most. He's flippant about what "good design culture" entails and how it works. Yes, evaluation and design aren't the same thing, but they also aren't simply separable either. And good design itself is a complex and difficult issue in environments where employees have diverse skill sets, engineering compromises are required to make deadlines and marketing demands the impossible -- in short, where multiple stakeholders are responsible for implementing the many parts of a large system (who often disagree on the goals or the "design" to achieve them). Design is hard to do well, even in a company with supporting usability staff and cooperative players working in development and management. The cats all need to get along.

Item 4: He's just correct about this. But since design and evaluation are often practiced by the same professional, it's not clear we need to be either an engineering discipline or a craft and frankly we often have to be both. Not everyone is going to be good at both activities, though. And as an audience member said, it's certainly true that some engineers are better than others, and it's also true that at the architectural level of engineering, it's really design that's going on.

Item 6 is disturbing. I worry about this regularly, especially faced with teams that don't really need much help, for whom I'm kind of an expensive waste of company money. Where's the value add? At least I don't think my stone is magic, when it comes to testing products. Jared told the stone soup fairy tale, in which the soldier with the stone knows full well it's the vegetables that make it good, but coerces the townspeople into cooking, and then he moves on with his stone to the next town. Jared thinks usability engineers think they're really making good soup because of their magic stones, and aren't seeing the vegetables too.