Meaning is not in the words — it’s in the total situation. – Ronald Langacker
To know if we created a great product, we need to test the User Experience beyond the screen:
| Level of Test | Types of Stories |
|---|---|
| Individual screens |
Short-term usability stories about UI-level problems. “I wanted to buy the product but couldn’t find a Buy button.” |
| Flow in context |
User Experience stories that show whether the product can successfully do the job it was hired for. “I avoided using the medical software, because it forced me to turn away from my patient.” |
| Usage over time |
Full User Experience stories that show how well the product works as the details of the job change over time. “The gorgeous curved screen design that I loved at first caused the screen to break a few months later, which cost me $400 to repair.” |
Visually polished and usability-tested screens can still lead to a failed product experience in the long run.
Case Study: Inventory System in Fallout 4 Game
The inventory gets long and hard to manage as you pick up tons of items. For example, it’s hard to compare weapons, because you can only see stats for one at a time:
But these sorts of usability-level issues are easiest to fix. For example, as a work around, this user prefixed the weapon names with useful stats. This makes it possible to compare items at a glance.
However, fixing screen-level problems is small potatoes in comparison to the larger issues that hurt the game. Here are user stories that evaluate the inventory system at different levels:
| Example: Choosing the best apparel | Screen Level | Context & Time Levels |
| Problem | I can apply clothing in inventory mode A by clicking it, but how do I apply apparel while in inventory mode B? Clicking in mode B sells apparel instead. | What kind of apparel is going to keep me safe? |
| What happened? | My workaround is to exit the trade dialogue and go into inventory mode A. I then use the body chart there to see what I’m wearing now and which new apparel is superior. I click the right apparel to apply it and go back into the trade dialogue and click the apparel I’m no longer wearing to sell it. However, I sometimes nearly sell an item by mistake when I reflexively click it to apply it. |
I’ve spent hours wondering which items to carry, figuring out where to stash inventory when I had too much to carry, comparing items, and agonizing over which to sell. In the end, I mastered the inventory UI, but my overall UX was poor. I expected my diligence to pay off, but that optimal weapon I handpicked still couldn’t kill the next enemy and despite all the hoarding and trading I still couldn’t afford the best apparel. At times I was paralyzed, even quit the game, when I found something important and had to figure out what to drop to make room. I’ve found myself not wanting to go into a new building, because finding new things had become a burden. |
You can clearly see which user stories affects the User Experience more profoundly.
There are issues of both clarity and meaning. Should I use the gun with 100 accuracy & 30 damage or the gun with 200 accuracy & 10 damage? What’s the difference between “accuracy” and “range”? Is a “fire rate” of 6 good or bad? These questions are frustrating, but the bigger UX problem is why any of this matters in the first place.
How does a feature translate into outcomes the user cares about?
If we can’t answer that, we have more than just a usability problem.
Case Study: Context for Medical Software
There’s a great story about software failure in Clayton Christensen’s Competing Against Luck:
We’d designed a terrific software system that we thought would help this doctor get his job done, but he was choosing to ‘hire’ a piece of paper and pen instead…
Why? The design team overlooked the situational and emotional context:
“As [Dr. Holmstrom] began to discuss Dunn’s prognosis, he grabbed a piece of paper to sketch out, crudely, what was wrong with Dunn’s knee and what they could do to fix it. This was comforting, but puzzling. Dunn knew there was state-of-the-art software in that computer just over Holmstrom’s shoulder to help him record and communicate his diagnosis during an examination. But the doctor didn’t choose to use it. “Why aren’t you typing this into the computer?” Dunn asked.
…The doctor then explained that not only would typing the information into the computer take him too much time, but it would also cause him to have to turn away from his patient, even just for a few moments, when he was delivering a diagnosis. He didn’t want his patients to have that experience. The doctor wanted to maintain eye contact, to keep the patient at ease, to assure him that he was in good hands…”
Case Study: Samsung Galaxy Edge
The Samsung S7 Edge phone was very slick with its curved edge. But it turned out that this design choice makes the phone hard to protect. Flat screens allow protective cases with higher sides that rise over the screen. Cases for curved screens rise just barely. Even with a high quality case, this screen cracked along the curved edge (and it happened twice)!
If we look beyond the first experience and aesthetic factors, we see a very different User Experience story.
The cost of repair was $400 the first time. The second time, I had to replace the perfectly functional phone, which didn’t suit my ecological values.
Sadly, Samsung appears to have standardized this design. I suspect it’s even financially lucrative, due to demand for pricey replacement parts or replacement devices.
Case Study: Fitbit Dashboard
In The Big Book of Dashboards, Steve Wexler describes how his experience with his Fitbit changed over time:
“After a while, I came to know everything the dashboard was going to tell me. I no longer needed to look at the dashboard to know how many steps I’d taken. The dashboard had educated me to make a good estimate without needing to look at it. Step count had, in other words, become a commodity fact. I’d changed my lifestyle, and the dashboard became redundant.
Now my questions were changing: What were my best and worst days ever? How did my daily activity change according to weather, mood, work commitments, and so on? Fitbit’s dashboard didn’t answer those questions. It was telling the same story it had been telling on the first day I used it, instead of offering new insights. My goals and related questions were changing, but Fitbit’s dashboard didn’t. After a year, my Fitbit strap broke, and I decided not to buy a replacement. Why? Because the dashboard had become a dead end. It hadn’t changed in line with my needs.”
So there wasn’t anything wrong with the dashboard in 2 dimensions, but its usefulness wasn’t constant along the dimension of time.
How to Uncover the User Experience Story
Time is a necessary dimension of user testing. Retrospective interviews and delayed customer feedback help reveal the total story. You want to know how the customer enjoyed the shopping experience or their first time playing a game. Then you should follow up to see how they feel weeks later.
You can get more insight into how the story unfolds through something like a diary study, which allows users to keep track of their usage at their own pace. They can capture one-off occurrences and experiences that might at first seem insignificant and would get lost otherwise.