Three Tips for Effective Usability Testing
“eWalk” is an application for WAP phones of China Mobile. Users need to log in through the application in order to visit the Wlan of China Mobile. To provide its users with access to more user-friendly online services, the Research Institute of China Mobile (CMRI) has improved the design of “eWalk”, and entrusted us to conduct a series of usability tests on the conceptual product with representative participants from diverse age groups and occupational backgrounds. The goal of the project was to develop a thorough understanding of the target audience(s), including the potential challenges and uncertainties they may face during their online use of the new application and their expectations for improvement in its design.
The work was required to be performed within three days. This was a big challenge, and a perfect opportunity for us to display our best grace under pressure, too. Promptly we built a responsible testing team with experienced user researchers, did all the needed homework, including recruiting testing participants, and drafting testing plan and timeline, and test scripts for usability testing. Prior to the formal testing, a diagnostic test was conducted with representatives from the actual user population for eWalk, gathering data of potential challenges and uncertainties faced by the users.
As the leader of the testing project, altogether I conducted one-on-one usability tests with eight users of the new eWalk within three days, each averaging about one and a half hours. To be frank, tests of such intensity are indeed a big challenge, both physical and mental, for any performer with all those designed tasks. Fortunately, I have done my homework, and gained valuable experience from the preliminary tests. So, the formal tests have worked out right for me. Here are my observations from the tests.
1. Maintain the best state before performing usability tests.
In performing usability tests, when I faced each and every user for the new eWalk, I was always in a highly alert status, ready to answer the user’s questioning at any possible point, and display procedure pages for specific operations accordingly. In order to guarantee adequate amount of energy for best testing effect, I would speed up my preparation for the upcoming session right at the conclusion of the previous session, and then seize every minute to relax. Between the sessions, I would breathe in droughts of fresh air, or look far into the sky, refueling my tired body and mind.
I have long known this from my testing experiences: the performer’s moods and state of mind have great impact on the whole testing process. Alert and dynamic, a performer can swiftly follow the test user’s pace of operating and reasoning, capable of co-textual dialogues with the user and quick to detect and capture the fleeting bits /fragments of valuable information. In contrast, a dull floppy performer will have trouble following the user, or even lose control of the test itself!
Perhaps some of you will say: we can always refer to the videos for user information after the tests. Yeah, that will work for you in some way, but you can neither recover the time wasted on the extra work, nor the most valuable but missing information that can be obtained only through the onsite face-to-face interaction between you and your testee.
Likewise, the testing user can be influenced by the performer’s moods. If the performer is cheerful and encouraging, the user will respond in a more positive way, showing greater willingness to voice his/her preferences. Otherwise, the user will feel dull, inefficient, and unrespected, always responding in a more negative way.
2. Don’t expect the user to do all testing tasks in the order specified in the testing plan.
In usability tests, the users were asked to accomplish some designated tasks,—in fact, a total of 10— in the order specified in the testing plan. But the problem was: they would not operate as instructed! As I noticed from diagnostic tests, in most cases, the testing user will feel greatly tempted to explore buttons of an unfamiliar interface; often when we expect him to click “confirm”, very probably he will click “search” instead to enter a different interface, where some problems are designed for later tasks only.
When this occurred, many users did not correct their operation mistakes immediately, as they were supposed to. Rather, they were attracted by the new thing; they started to think hard about the new problems! At this moment, I would not force my testing user to do as I told them to; I would let him continue his adventure, my thinking following his, discussing the interface with him, eliciting comments from him, exchanging insights now and then.
I understood that such unplanned responses may very well interrupt the designated order for the tens of cards we prepared for the test, and increase the difficulty of after-test data-analyzing, since the testing plan would not work for me then; and worse still, I may very likely follow the user’s thinking too closely as to forget my original purpose!
Despite all those possible risks, still I will stick to my own way: try not to interrupt the user in his exploration, not to mention force him out back to where he should stay unless what he is doing goes too far away from the testing theme. I do believe, when a testing user starts to think without being urged, he is actively involved, his mind working hard to structure some important information he strongly wishes to convey. Any untactful interruption at the point will only cause him to forget the important information he has in mind then or discourage his initiative as to affect his performance in the upcoming tests.
3. Be patient with the testing user; interrupt the user’s answering with tact.
I have an interesting observation from the usability tests of the new eWalk: when they saw a UI or operational process, almost all users would make some comments, like “the color design of the interface is poor”, “I didn’t expect I had to press this button. This is so very inconvenient”, “This process is very complicated, more so than the other application…”, “you can refer to X’s design to improve this”, etc. when I heard these critical remarks, I would not ask questions right then and there. Instead, I would first listen patiently to the users’ free remarks.
True, sometimes I might feel confused somewhere in the middle of their discussion, and could not probe further without the necessary knowledge. If that’s the case, I would slip in, repeat what they had said and then inquire in a polite tone, “Excuse me, you just mentioned #￥%#￥#, but I don’t quite understand. Could you please explain it for me? ” so that the user will not forget what he said after being interrupted. Partial repetition helps remind him of some clues and quickly return to where he has been cut.
The eWalk project has proceeded very smoothly. Almost each and every testing user has contributed some important information, so our final testing report turns out perfect and well-accepted among customers. Of course, efforts to produce the report are strenuous; often because we did not interrupt users’ thinking and reasoning during the testing, we had to examine thoroughly all the testing videos, audio recordings and written notes, just to confirm a user response to certain task. For the valuable information, for the improvement of eWalk, and for the interest of all users, all we do are worthwhile!
Tiekuaier
User Researcher from Peopeo
http://www.peopeo.de
