Data and Design, Part Two: Context and Moderation

A few days ago I wrote a post that began a discussion about the relationship between data and design, and I'd like to expand on that theme with this post.

I touched on both the importance of data and its limitations, and here I'd like to provide some more context for that discussion by bringing in some other voices. In June, Patrick Lynch, a web designer for Yale University and the author of the Web Style guide, wrote a piece for A List Apart called "Visual Decision Making." In the article, Lynch primarily focuses on new research suggesting that aesthetics play a larger role in user experience than commonly thought. I tend to agree with his point of view. Here's an excerpt:

"Research confirms that users make aesthetic decisions about the overall visual impression of web pages in as little as 50 milliseconds (1/20th of a second). [Sources here: 4, 5] These instant visceral reactions to web pages happen in virtually all users, are consistent over visit length, and strongly influence the user’s sense of trust in the information. In short, users have made fundamental, consistent, and lasting aesthetic decisions about the credibility and authority of sites before major eyetracking events begin."

Since most usability testing focuses on task completion, click streams, and navigational ease, this kind of qualitative response is nearly impossible to capture. It can, however, play an important role in the overall user experience. This isn't to dismiss the usability itself, but instead to place it within context - even sites with great usability metrics may fail to create a sense of enjoyment or engagement. Depending on the type of site or application, this may be of little importance - a company intranet doesn't need to make people feel warm and fuzzy - but it might also be the difference between people "getting it" and people "loving it." The bottom line from Lynch: "You should never ignore solid user experience data, but mountains of data won’t auto-magically build you a successful site." (Brownie points for anyone who defines "auto-magically" in the comments).

Douglass Bowman, a driving force behind the design for Google Calendar, Blogger, and Wired Magazine's web presence, among many others, eventually butted heads with the data monster too many times and decided it was time to leave. On his departure in March of last year, he wrote:

"When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions."

Does this sound a little like a whiny designer? Yes, it certainly does. And when you have as many users as Google does, it gets much harder to step outside the box, for fear of the masses coming back with a vengeance. But it can be done (look no further than Facebook, whose first major redesign caused a venerable digital ruckus) and sometimes the effects are quite positive. If Facebook had relied solely on user data when testing the first iteration of its stream-style design, I'm quite sure it would have been abandoned on the drawing board. Again, the point here is not that data isn't a hugely important factor in designing interfaces, but that it can be overdone. In the case of Google, for instance, I would argue that a little bit of aestheticism would go a long way. Gmail has done this pretty well with themes, but other products (ahem, Reader) remain strange, complex and often confusing beasts.

The purpose of this post, however, isn't to throw data under the bus. It's to help us try to understand what data will be most useful, how to harvest them, and how to put them to use. Here I turn to Allison J. Head, the principal and founder of usability research firm Head and Associates , and what she's called "emblematic measures." In short, she defines these as the measures that are easily understood with complex analysis. A primary example on an emblematic measure, she says, is bounce rate: "Everyone (from the CEO on down) gets this metric right away and can understand if it is good or bad." These are the metrics, she argues that ought to carry the most weight. When constructed correctly they are simple, easy to understand, and easy to analyze. Two other measures she discusses are Site Abandonment Measure (SAM), the percentage of people who give up on a task, and Site Abandonment Rate (SAR), a more traditional which measures the rate of users that end their sessions from specific pages - a setup guide or a shopping cart screen, for example.

These measures, she argues, are the right ones to track because they help point to large problems that have binary metrics: users either completed the task, or they didn't. Simple game. Smaller stuff, she admits, is harder to measure, as much because of the presence of data as the absence: "With so much of our research focused on striving for accurate representations of something as amorphous, varied, and hotly debated as user behavior, we are a profession usually awash in data, practicing a less-than-perfect science."

To make a convoluted analogy, the world of web design must rely on creationism and evolution simultaneously, making judgements that are scientific, artistic, and emotional at the same time. In the third installment of this series, I'll lean more toward the left-brained analysis and focus on how data can reveal insight that might never have been found otherwise.