Tag: empathy

Posted on

Solving Problems with User Research, Best Practices, and A/B Testing

What can I do to persuade more people to buy your product online? I tackled this question for 5 years as I ran A/B tests for diverse clients.

I remember one test idea that everyone on the team loved. The client said “That’s the one. That one’s totally going to win.” Well, it didn’t.

The fact is, most A/B test ideas don’t win.

In fact, interpretation is tough, because there are so many sources of uncertainty: What do we want to improve first? Which of a hundred implementations is a valid test of our hypothesis about the problem? If our implementation does better, how statistically reliable is the result?

Is our hypothesis about the users actually true? Did our idea lose, because our hypothesis is false or because of our implementation? If the idea wins, does that support our hypothesis, or did it win for some completely unrelated reason?

Even if we accept everything about the result in the most optimistic way, is there a bigger problem we don’t even know about? Are we inflating the tires while the car is on fire? 

If you take anything away from this, take this analogy: inflating your car tires while the car is on fire will not solve your real problem.

I believe the most effective means of selling a product and building a reputable brand is to show how the product meets the customer’s needs. This means we have to know what the customer’s problem is. We have to talk to them.

Then if we run an A/B test and lose, we won’t be back to square one. We’ll know our hypothesis is based in reality and keep trying to solve the problem.

Emulating Competitors

“I heard lots of people found gold in this area. I say we start digging there!”

That actually is a smart strategy: knowing about others’ successes helps define the opportunity. That’s how a gold rush happens.

This is why A/B testing blogs are dominated by patterns and best practices. So-and-so gained 50% in sales by removing a form field… that sort of thing. Now don’t get me wrong: you should be doing a lot of those things. Improve your value proposition. Ensure your buttons are noticed. Don’t use tiny fonts that are hard to read. You don’t need to test anything to improve, especially if you focus on obvious usability issues.

So what’s the problem? Well, let’s go back to the gold analogy. Lots of people went broke. They didn’t find any gold where others had or they didn’t find enough:

“The actual reason that so many people walked away from the rush penniless is that they couldn’t find enough gold to stay ahead of their costs.” ~ Tyler Crowe Sept. 27, 2014 in USAToday

You could be doing a lot of great things, just not doing the RIGHT things.

The good thing is many people do some research. The problem is not enough of it or directly enough. They are still digging in the wrong place.

“If I had only one hour to solve a problem, I would spend up to two-thirds of that hour in attempting to define what the problem is.” ~ An unknown Yale professor, wrongly attributed to Einstein.

Think about this for a moment: How can you sell something to anyone when you’ve never talked to them or listened to what they have to say?

Product owners often believe they know their customers, but assumptions usually outnumber verifiable facts. Watching session playback can hint at problems. Google Analytics gives a funnel breakdown, but it doesn’t give much insight into a customer’s mind. It’s like trying to diagnose the cause of indigestion without being able to ask the patient what they had for dinner or if they have other more serious health complaints.

The problem is it’s all impersonal, there’s no empathy. There’s no “Oh man, that sucks, I see how that is a problem for you”. It’s more like “Maybe people would like a screenshot there. I guess that might be helpful to somebody”.

Real empathy spurs action. When you can place yourself in your customer’s situation, you know how to go about helping them. If your solution doesn’t work, you can try again, because you know the problem is real rather than a figment of your imagination.

A Pattern Is A Solution To A Problem

Therapist: “Wait, don’t tell me your problem. Let me just list all the advice that has helped my other patients.”

Let’s say some type of visual change has worked on 10 different sites. Let’s call it a pattern.

A pattern works, because it solves some problem. So choosing from a library of patterns is choosing the problem you have. You don’t chose Tylenol unless you have a headache or fever. You don’t chose Maalox unless you have indigestion.

If you know what YOUR problem is, you can choose the right patterns to solve it.

If you don’t know the problem, you won’t get far choosing a pattern because it’s popular, because of how strongly it worked or how many people it has worked for. That’s like taking a medication you’ve never heard of and seeing what it does for you.

Pattern libraries are great for when you have a problem and want a quick, time-tested way to solve it:

Research Uncovers The Problem: A Short Story

Say you’re a shoe brand. You decide to reach out to people who are on your mailing list but haven’t purchased yet.

So you send out a survey. Within the first day, it becomes clear that many people are avoiding buying your shoes, because they’re not sure about sizing.

You’re shocked, but you shouldn’t be. User research insights are often surprising.

It’s just that you thought you anticipated this by posting precise measurements, a great return policy, and glowing testimonials. If anything, you thought people would mention the price, but no one so far mentioned price.

That’s a big deal for your product strategy. You need to build trust. So you set aside your plans for a full redesign (those fancy carousels on your competitor’s site sure are tempting). You set aside A/B test ideas about the font size of prices, removing fields, and so on.

You tackle the big problem. You do some research and come up with solutions:

  • match sizing to a set of well known brands
  • provide a printable foot template
  • allow people to order two sizes and return one
  • mail out a mock plastic “shoe” free of charge, and so on…

You ask a couple of people to come to the office and try some of your solutions.

Your user testing methodology is simple: First people pick their size based on either the sizing chart or template. Then they see if the real shoe fits.

Result? The matched sizing and the foot template were very effective in predicting fit. In user testing, the initial template didn’t work so well, because it’s hard to place a 3D foot in perfect position on a 2D printout. So, you come up with a template that folds up at the back and front, simulating a shoe. The users liked that much better. In fact, you start working on a cardboard model you can mail cheaply to anyone who requests it.

Now you’re off to testing it in the real world!

You design 2 different foot sizing comparisons, one pretty one with photos of top 3 brands and one long, plain table with 20 different brands. You also create an alternative page that links to the downloadable foot template.

You A/B test these variants over 2 weeks and pick the one that works.

(Then you go back to your research and find the next problem.)

You may also like this post about patterns: Compact Navigation Patterns .

If you want to uncover the biggest problems for your customers, I’m happy to help.

Posted on

The User Story In Context And Time

Meaning is not in the words — it’s in the total situation. – Ronald Langacker

To know if we created a great product, we need to test the User Experience beyond the screen:

Level of Test Types of Stories
Individual screens

Short-term usability stories about UI-level problems.

“I wanted to buy the product but couldn’t find a Buy button.”
Flow in context

User Experience stories that show whether the product can successfully do the job it was hired for.

“I avoided using the medical software, because it forced me to turn away from my patient.”
Usage over time

Full User Experience stories that show how well the product works as the details of the job change over time.

“The gorgeous curved screen design that I loved at first caused the screen to break a few months later, which cost me $400 to repair.”

Visually polished and usability-tested screens can still lead to a failed product experience in the long run.

Case Study: Inventory System in Fallout 4 Game

The inventory gets long and hard to manage as you pick up tons of items. For example, it’s hard to compare weapons, because you can only see stats for one at a time:

But these sorts of usability-level issues are easiest to fix. For example, as a work around, this user prefixed the weapon names with useful stats. This makes it possible to compare items at a glance.

However, fixing screen-level problems is small potatoes in comparison to the larger issues that hurt the game. Here are user stories that evaluate the inventory system at different levels:

Example: Choosing the best apparel Screen Level Context & Time Levels
Problem I can apply clothing in inventory mode A by clicking it, but how do I apply apparel while in inventory mode B? Clicking in mode B sells apparel instead. What kind of apparel is going to keep me safe?
What happened? My workaround is to exit the trade dialogue and go into inventory mode A. I then use the body chart there to see what I’m wearing now and which new apparel is superior. I click the right apparel to apply it and go back into the trade dialogue and click the apparel I’m no longer wearing to sell it. However, I sometimes nearly sell an item by mistake when I reflexively click it to apply it.

I’ve spent hours wondering which items to carry, figuring out where to stash inventory when I had too much to carry, comparing items, and agonizing over which to sell. In the end, I mastered the inventory UI, but my overall UX was poor.

I expected my diligence to pay off, but that optimal weapon I handpicked still couldn’t kill the next enemy and despite all the hoarding and trading I still couldn’t afford the best apparel.

At times I was paralyzed, even quit the game, when I found something important and had to figure out what to drop to make room. I’ve found myself not wanting to go into a new building, because finding new things had become a burden.

You can clearly see which user stories affects the User Experience more profoundly. 

There are issues of both clarity and meaning. Should I use the gun with 100 accuracy & 30 damage or the gun with 200 accuracy & 10 damage? What’s the difference between “accuracy” and “range”? Is a “fire rate” of 6 good or bad? These questions are frustrating, but the bigger UX problem is why any of this matters in the first place.

How does a feature translate into outcomes the user cares about?

If we can’t answer that, we have more than just a usability problem.

Case Study: Context for Medical Software

There’s a great story about software failure in Clayton Christensen’s Competing Against Luck:

We’d designed a terrific software system that we thought would help this doctor get his job done, but he was choosing to ‘hire’ a piece of paper and pen instead…

Why? The design team overlooked the situational and emotional context:

“As [Dr. Holmstrom] began to discuss Dunn’s prognosis, he grabbed a piece of paper to sketch out, crudely, what was wrong with Dunn’s knee and what they could do to fix it. This was comforting, but puzzling. Dunn knew there was state-of-the-art software in that computer just over Holmstrom’s shoulder to help him record and communicate his diagnosis during an examination. But the doctor didn’t choose to use it. “Why aren’t you typing this into the computer?” Dunn asked.

…The doctor then explained that not only would typing the information into the computer take him too much time, but it would also cause him to have to turn away from his patient, even just for a few moments, when he was delivering a diagnosis. He didn’t want his patients to have that experience. The doctor wanted to maintain eye contact, to keep the patient at ease, to assure him that he was in good hands…”

Case Study: Samsung Galaxy Edge

The Samsung S7 Edge phone was very slick with its curved edge. But it turned out that this design choice makes the phone hard to protect. Flat screens allow protective cases with higher sides that rise over the screen. Cases for curved screens rise just barely. Even with a high quality case, this screen cracked along the curved edge (and it happened twice)!

If we look beyond the first experience and aesthetic factors, we see a very different User Experience story.

The cost of repair was $400 the first time. The second time, I had to replace the perfectly functional phone, which didn’t suit my ecological values.

Sadly, Samsung appears to have standardized this design. I suspect it’s even financially lucrative, due to demand for pricey replacement parts or replacement devices.

Case Study: Fitbit Dashboard

In The Big Book of Dashboards, Steve Wexler describes how his experience with his Fitbit changed over time:

“After a while, I came to know everything the dashboard was going to tell me. I no longer needed to look at the dashboard to know how many steps I’d taken. The dashboard had educated me to make a good estimate without needing to look at it. Step count had, in other words, become a commodity fact. I’d changed my lifestyle, and the dashboard became redundant.

Now my questions were changing: What were my best and worst days ever? How did my daily activity change according to weather, mood, work commitments, and so on? Fitbit’s dashboard didn’t answer those questions. It was telling the same story it had been telling on the first day I used it, instead of offering new insights. My goals and related questions were changing, but Fitbit’s dashboard didn’t. After a year, my Fitbit strap broke, and I decided not to buy a replacement. Why? Because the dashboard had become a dead end. It hadn’t changed in line with my needs.”

So there wasn’t anything wrong with the dashboard in 2 dimensions, but its usefulness wasn’t constant along the dimension of time.

How to Uncover the User Experience Story

Time is a necessary dimension of user testing. Retrospective interviews and delayed customer feedback help reveal the total story. You want to know how the customer enjoyed the shopping experience or their first time playing a game. Then you should follow up to see how they feel weeks later.

You can get more insight into how the story unfolds through something like a diary study, which allows users to keep track of their usage at their own pace. They can capture one-off occurrences and experiences that might at first seem insignificant and would get lost otherwise.

Posted on

15 Jobs-To-Be-Done Interview Techniques

Here are 15 techniques I extracted from the Jobs-To-Be-Done interview Bob Moesta’s team did with a camera customer (link at bottom):

Set expectations

Give an introduction to how long the interview’s going to take and what sorts of things you’re interested in. For example, “even minor details may be important”.

Ask specific details to jot the customer’s memory

Don’t just ask what the customer bought but why that model, which store, what day, what time of day, where they in a rush…

Use humor to put the customer at ease

Intentionally or not, early in the interview the whole team had a good laugh about something the customer said. I think it did a lot to dull the edge of formality.

Discuss pre-purchase experiences

Ask what the customer used before they bought the product and what they would use without it. Dig into any “I wish I had it now” moments prior to the purchase.

Go back to the trigger

Walked back to what triggered the customer to even start thinking about buying the product and to a time before they ever considered it.

Get detailed about use

Interviewers and the customer talked about how she held the camera, which hand, in which situations she used it, which settings she used, and advantages/disadvantages of the alternatives. You want the customer to remember and imagine the product in their hands. Things like the weight or texture of the product could impact the user experience. Dismiss nothing.

Talk about lifestyle impact

Dig into ways in which the product impacted the customer lifestyle, things they were/are able or unable to do. For example, they talked about how taking pictures without the camera affected the way she presented her trip photos to her sister. Focus on the “use” rather than the specific “thing”. For example, you can ask “do you like this feature”, but then you want to move to “what does this feature mean to you in terms of what you’re able to do, how it affects your lifestyle, your future decisions”.

Explore product constraints

Talked about how other decisions and products impacted the decision. For example, the size of the bag that has to fit the camera and avoiding the slippery slope of requiring additional accessories.

Ask about alternatives

Products don’t exist in isolation. The customer had several other solutions, which serve different, specific purposes. Figure out whether the new product will replace or complement other products.

Point out inconsistencies, such as delays

Interviewers pointed out that the customer waited a long time to buy the product from the initial trigger to making the call after a trip. They asked “Why did you wait so long?”

Talk about the influence of other people

Ask about advice other people gave the customer or how other people may be affected by the decision.

Don’t put words in their mouth

In digesting and summarizing back to the customer, it’s easy to inject your own conclusions and words. Try to elicit attitudes and conclusions from the customer. Lead them to it but don’t do it for them (a related technique is to start talking and then leave a pregnant pause, so the customer can complete the thought). In one clear case in the camera interview, the interviewers asked a leading question but then prompty noticed this and corrected themselves, saying “Don’t use his words”.

Talk about the outcome

Asked open ended questions about whether the customer was happy with their purchase and in what ways. Ask about specific post-purchase moments when the customer felt “I am glad I have it right now”, but focus on how the situation is affected not the product itself.

Here are some additional I considered after listening  to the interview:

Avoid fallacy of the single cause

Don’t push the conversation towards a single cause (see Fallacy of the single cause). Rather than engage in cause reductionism, accept there may be multiple, complex causes.

Let’s say you pose the question: “Joe said that, and so you decided to buy X?” The simple narrative may be intuitive, causing the subject to be persuaded that “Yes, I guess that is why I decided to buy X”. In reality, the events may be true (Joe did say that), but in reality may be unconnected. In these cases, it’s important to point out inconsistencies rather than seek confirmation. For example, in the camera interview the interviewer rightly pointed out an inconsistency: “Why did you wait so long to buy X after he said that?” They also often asked “What didn’t you…” Work together to uncover the truth.

Beware planting false memories

Do not reflect back your own sentiments or ideas to the interviewee when clarifying. For example, asking people to confirm something they did not literally say may cause them to confirm a causal relationship that did not happen (other cognitive biases may aid this: pleasing the interviewer, tendency to fall for reductionism). It may plant a subtle attitude that might then be amplified through the course of the interview. Also be careful with “because” statements, as there is some evidence that we are biased to accept such explanations even when they are irrational (see The Power Of The Word Because).

More on possibility of implanting false memories Video 1 and Video 2.

Listen to the interview for yourself.