Data and Design, Part Two: Context and Moderation

A few days ago I wrote a post that began a discussion about the relationship between data and design, and I'd like to expand on that theme with this post.

I touched on both the importance of data and its limitations, and here I'd like to provide some more context for that discussion by bringing in some other voices. In June, Patrick Lynch, a web designer for Yale University and the author of the Web Style guide, wrote a piece for A List Apart called "Visual Decision Making." In the article, Lynch primarily focuses on new research suggesting that aesthetics play a larger role in user experience than commonly thought. I tend to agree with his point of view. Here's an excerpt:

"Research confirms that users make aesthetic decisions about the overall visual impression of web pages in as little as 50 milliseconds (1/20th of a second). [Sources here: 4, 5] These instant visceral reactions to web pages happen in virtually all users, are consistent over visit length, and strongly influence the user’s sense of trust in the information. In short, users have made fundamental, consistent, and lasting aesthetic decisions about the credibility and authority of sites before major eyetracking events begin."

Since most usability testing focuses on task completion, click streams, and navigational ease, this kind of qualitative response is nearly impossible to capture. It can, however, play an important role in the overall user experience. This isn't to dismiss the usability itself, but instead to place it within context - even sites with great usability metrics may fail to create a sense of enjoyment or engagement. Depending on the type of site or application, this may be of little importance - a company intranet doesn't need to make people feel warm and fuzzy - but it might also be the difference between people "getting it" and people "loving it." The bottom line from Lynch: "You should never ignore solid user experience data, but mountains of data won’t auto-magically build you a successful site." (Brownie points for anyone who defines "auto-magically" in the comments).

Douglass Bowman, a driving force behind the design for Google Calendar, Blogger, and Wired Magazine's web presence, among many others, eventually butted heads with the data monster too many times and decided it was time to leave. On his departure in March of last year, he wrote:

"When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions."

Does this sound a little like a whiny designer? Yes, it certainly does. And when you have as many users as Google does, it gets much harder to step outside the box, for fear of the masses coming back with a vengeance. But it can be done (look no further than Facebook, whose first major redesign caused a venerable digital ruckus) and sometimes the effects are quite positive. If Facebook had relied solely on user data when testing the first iteration of its stream-style design, I'm quite sure it would have been abandoned on the drawing board. Again, the point here is not that data isn't a hugely important factor in designing interfaces, but that it can be overdone. In the case of Google, for instance, I would argue that a little bit of aestheticism would go a long way. Gmail has done this pretty well with themes, but other products (ahem, Reader) remain strange, complex and often confusing beasts.

The purpose of this post, however, isn't to throw data under the bus. It's to help us try to understand what data will be most useful, how to harvest them, and how to put them to use. Here I turn to Allison J. Head, the principal and founder of usability research firm Head and Associates , and what she's called "emblematic measures." In short, she defines these as the measures that are easily understood with complex analysis. A primary example on an emblematic measure, she says, is bounce rate: "Everyone (from the CEO on down) gets this metric right away and can understand if it is good or bad." These are the metrics, she argues that ought to carry the most weight. When constructed correctly they are simple, easy to understand, and easy to analyze. Two other measures she discusses are Site Abandonment Measure (SAM), the percentage of people who give up on a task, and Site Abandonment Rate (SAR), a more traditional which measures the rate of users that end their sessions from specific pages - a setup guide or a shopping cart screen, for example.

These measures, she argues, are the right ones to track because they help point to large problems that have binary metrics: users either completed the task, or they didn't. Simple game. Smaller stuff, she admits, is harder to measure, as much because of the presence of data as the absence: "With so much of our research focused on striving for accurate representations of something as amorphous, varied, and hotly debated as user behavior, we are a profession usually awash in data, practicing a less-than-perfect science."

To make a convoluted analogy, the world of web design must rely on creationism and evolution simultaneously, making judgements that are scientific, artistic, and emotional at the same time. In the third installment of this series, I'll lean more toward the left-brained analysis and focus on how data can reveal insight that might never have been found otherwise.

The Symbiosis of Data and Design

Last Thursday, we had a great user testing session where we got to meet a lot of our users, registered a few new ones, and generally had a good time getting to know the people that represent our awesome user base. These people are insightful, extremely bright, and just plain fun to be around. I'd like to thank them for making it over to our humble headquarters here in Boston, and I hope that some more of you will join us at our next session on the November 19th (email me if you'd like to come).

While we had a great time just interacting casually with our users after the primary testing was over, the real value of the event revealed itself after they'd left - and left behind great data for us. It's with this data that I'd like to begin a discussion that will span a few posts, focusing on the relationship between research data and applied product design.

There are all different kinds of data - behavioral, demographic, preferential, etc. - and each of these has incredible value for us. What products are people using? How are they using them? How are they using our product and what do they think about it? How do they perceive themselves as users? The answers to these questions give us a window into the hearts and minds of the people that make up our market. They help us understand the gap (hopefully small) between what our customers expect and what they feel they're currently getting.

Learning all of this is great, but products aren't built from thoughts and opinions and amalgamated data charts. Products are built by a few people who make a lot of small decisions about how things should look, feel, and work. Accurate, well-constructed data is important in the design process not as definitive statements about what exactly will work. Instead, data is a tool that can be used to clearly define those approaches that won't work and to narrow the breadth of potential design solutions.

So how can we use the data we gather to inform design decisions? There are lots of strategies out there, but here are a few basic tactics we try to use when constructing test sessions to gather usable data:

1) Test themes, not specifics. While there are certainly times we want to get feedback on really small features or design elements, it's usually easier to apply data when testing two or more different themes with distinct differences, where elements are arranged in a fashion that's clearly different to a user's eye. If design options are too similar, the feedback can be muddled, whereas wide variations often produce stronger, more passionate responses.

2) Numbers are useful, but words are honest. When we think data, we think numbers. But qualitative data - words, usually - can be just as important, and are often easier to read than numbers. Especially when dealing with small sample sizes (as we often are), it's possible to get numbers that can be misleading when taken alone. In addition, there are stigmas that come with answering survey questions, even if users know their data will be anonymous. On the other hand, getting people to talk through their experiences as they use the product or take the test can reveal all kinds of difficulties, frustrations, and gratifications that don't shine through from a page of statistics. Usability expert Jakob Nielsen has long advocated for the power of qualitative data, especially during a more agile, iterative design process.

3) Ask what matters. There are some parts of a new design that the designer or team has decided are essential features because of their design requirements or other objectives. If an element, such as an action button, has been included and placed somewhere for a very specific reason, it may not make sense to test it. (Of course, if users respond negatively to the element without any prompt; it's probably a good idea to re-think this decision). Ask questions about the elements you're not dead-set on, gather the data, and then make decisions. Relying comprehensively on the data, especially if it comes from a wide range of users, lends itself to a loss of consistency and a common aesthetic.

In the end, it comes down to doing the research, looking at the numbers, and trusting your team's intuition. More data is always better, but finding ways to use and qualify that data is essential to keeping it from becoming overwhelming, especially in a small team without dedicated usability people.

I'll continue with a few more posts on this theme, highlighting some examples of design influenced by data and the musings of some more-established sources on the matter. If you have any thoughts on usability testing, design, or the role of data in the design process, leave a comment!