Usability Testing: Monitoring User Mood and Identifying User “Lies”

We have been entrusted by the Research Institute of China Mobile (CMRI) to conduct usability testing for their newly developed conceptual mobile homescreen. This is a continued effort of a prior joint project, in which our task was to gather needed information about mobile homescreens and user needs and provide a research report to support CMRI’s development of the above mentioned product. The purpose of the testing project is to identify user preferences, responses and potential needs for the concept product and provide support to the CMRI development team in improving the concept homescreen.

The testing team recruited the former in-depth interviewing team members; their prior understanding and knowledge of homescreen made the work easier to start. In fact testing user recruitment was completed within one day. Testing materials were presented through flash, and cards were added as supplements, to facilitate and enhance user understanding of the UIUE design features of the tested homescreen. Preliminary tests were conducted with actual representative users of the product, providing opportunities to understand and practice scheduling testing sessions and sequencing testing components and tasks; further revision was done of the usability testing plan based on insights from these pre-tests, too.

Here I would like to discuss two findings from this usability testing. Their implications may extend beyond to work for any type of user testing.

1. Monitor user moods.

This observation has come from the last testing user, who was annoyed when a security guard asked him to get registered for entry into the testing laboratory. The guard was just doing his job; and the user was supposed to do as he was required.

Let me explain this here: in recruiting testing users, we will inform them of testing environment, facilities and mode, and inquire their opinions too. If the user disagrees, he/or she will not be recruited for the test.

So as a recruit, this user should have been informed the rule of registration for the lab. Then, right when the test was about to start, this annoyed user started to complain about the “indignity” of registration, claiming such requirement was a humiliation to him, and that it was the source of all existing evils in social systems and human equality!

At first, we explained with great patience, wishing this emotional user could display understanding, only to find he turned increasingly sensational. Apparently this user was not in the right mood for a usability test or rational thinking. Even if we started the test immediately, it would not mean much. Decidedly we postponed his test. We had talks with him. We tried our best to calm him. Eventually, our patience and sincerity won him over; about 10 or more minutes later, this user calmed down and expressed his understanding of our work. Finally, the whole test wound up smoothly.

Being open and sincere is an important technique, or rather an attitude, to conduct user testing successfully. The case above is a good illustrate of this point: being open and sincere in the testing requires the interviewer listen attentively, interact dynamically, switch topics like a diplomat, always alert to the subtleties in user responses. In fact, this technique should be extended into each and every stage of user testing, including inviting users through phone calls, receiving users at the entry to testing labs, their leave-taking after tests, etc. The reason is simple: each and any of these stages can have impact on the testing user’s cognition, mood, attitude, psychology of thinking, and eventually the quality and value of the whole testing.

2. Identify user “lies”.

User responses in interviews fall into three broad categories: complimentary, critical and neutral.  Users of the first category of responses can always find strengths of tested products, and are often very generous with compliments, marking the majority of the tested products and near-100 points. Even when they do report some problem, don’t expect them to be frank and direct. They are the cautious type, often very cautious in identifying the problem, and even more so in claiming it a design defect of the product itself. Instead, they will put the blame on themselves, claiming they are not familiar with such products, or not skilled in using them, or even not intelligent as users.

 

Unlike the complimentary type, users of the critical type tend to present themselves as very experienced, always citing examples, comparing and contrasting, using jargons in their reports. Their comments are full of criticism, direct and very sharp, sometimes even exaggerated. No words like “good” can be found in their rankings. They use “just so-so” for “good”, “poor” for “just so-so”, and the truly “poor” will become unbearable to them!

The neutral type of users are more like their true selves in testing. They usually call a spade a spade, and honestly report how they feel about a specific tested product.

This categorization reveals one plain fact: in testing, for one reason or another, many users may not report their true feelings and responses. Then, how to tell whether the testing user is being insincere in reporting becomes a required technique for us test-conductors. This means we should be able to detect and interpret users’ body language. One rule of thumb from this testing, and a very effective way, is to watch the testing user’s hand gestures and facial expressions, point out the inconsistency between his/her verbal and physical responses and then work with him/her to find out the true reasons behind.

Take a 39-year-old female user for example. She was very representative of the complimentary type, always providing full marks for whatever subjective questions faced by her, claiming now and then “This is marvelous!”, “It is so convenient!”. But when I asked her to accomplish one related testing task, she failed after a very long time of working on it, and explained it was because she knew too little of mobile phones. Of course, this was not true reasons. Later on, I worked together with her through all the procedures involved in the task, explained to her what is interactive mode and eventually identified the designs that fell short of her expectations.

This time, let’s take a 26-year-old male user, a genuine trendsetter, for example. He belonged to the critical type. He breezed through each and every testing task assigned to him, but always scored very low for all designs, insisting they simply lacked creativity. He then actually compared the tested products with a diverse range of homescreens to illustrate his point. After in-depth interviewing, we found he is a loyal user of Apple; products other than Apple will make him uncomfortable. The fact was: his first responses were not true. They were just some flow of first impressions ungrounded by earnest thinking and/or manipulating of the tested products.

How to interpret body language is a sophisticated knowledge. We need to study instances of body language closely in daily interactions, make generalizations about them, and read theories on the subject, so that we can use the knowledge for user testing purposes, to elicit and gather reliable user information to produce useful reports for product development and improvement.

 

Tiekuaier

User Researcher from Peopeo

http://www.peopeo.de