1

I'm planning some usability testing at work and have a few questions that I can't seem to find good answers for anywhere else.

Firstly I'm relatively new to usability testing and have never planned or conducted it in a professional setting however the client doesn't have a huge budget for this so we're taking a diy approach.

I've been influenced by Steve Krug's 'Rocket Surgery made Easy' and would like to use a script similar to his 'Usability Test Script' http://www.sensible.com/downloads-rsme.html

The problem is that we'd like to ask a lot more interview style questions beforehand to inform the development of our user personas however this is starting to feel like a different exercise and I'm wondering wether it's a good idea to combine design research and usability testing in the same session?

Another concern is that the production team want to conduct the usability tests themselves having been provided with a script because of concerns about impartiality. Is this a better alternative to having the same UX team that will be working on the website conducting the session?

Let me know your thoughts,

Many thanks

2 Answers 2

3

The problem is that we'd like to ask a lot more interview style questions beforehand to inform the development of our user personas however this is starting to feel like a different exercise and I'm wondering wether it's a good idea to combine design research and usability testing in the same session?

It's not uncommon to ask the user pre and post interview questions to both establish an audience baseline and qualitative feedback on system usability. However, these are more related to the experience of the test rather than exploring facets of different users of your system. It's recommended that when doing persona interviews, patterns begin to emerge after 25-30 users. You probably don't to need run a usability test on that many users, and while you probably will get some valuable persona information from 5-6 users of the usability test, it won't be enough to look for patterns to fuel a cluster analysis.

Another challenge will be managing the users time and attention. For each user you'll have realistically about 30-45 minutes of quality attention from the user. A persona interview is typically more conversational in nature and could quickly and easily take up the hour scheduled for the usability test. A usability test will be driven by the task at hand. Some participant may have a problem efficiently switching modes of thought. That said, I think these efforts should be split so that you can get the most out of each effort. Whatever you decide to do consider having a second person in the room with you facilitating the note taking. As moderator, it can be challenging to keep the interview/test moving while observing and taking notes. The second person will also serve to catch additional insights that you may have otherwise missed.

Another concern is that the production team want to conduct the usability tests themselves having been provided with a script because of concerns about impartiality. Is this a better alternative to having the same UX team that will be working on the website conducting the session?

This depends on whether you can remain impartial and which team has more training/insight in how to conduct a usability test. Impartiality is of upmost importance. You don't want to be leading the user or influencing the results in any way. In my experience UX teams have greater experience/interest in conducting usability testing but are challenged, understandably, by internal biases. To quote a passage from UXMatters.

The fact that you’ve taken a stand and created a design in the first place means you’ve articulated your design hypothesis and instantiated your hypothesis in the form and function of your design. It is going to be difficult to keep yourself from wanting to confirm your design hypothesis, because you’re hardwired to preferentially seek out confirming rather contradictory evidence. Even though you’re able to criticize your own designs and recognize that a fundamentally sound design needs some adjustment, confirmatory bias makes it hard for you to realize that your design is the wrong approach entirely.

http://www.uxmatters.com/mt/archives/2009/09/testing-your-own-designs-bad-idea.php

In a perfect world we would have a third party test our designs, or a UX researcher embedded in the UX team for this very purpose. However, with organizations downsizing UX resources we are often faced with both the need to design and validate. In your case you have a team who may have less internal bias willing to perform the testing. If you feel that can remain unbiased, and conduct the interview effectively I'd say go for it. You can alway double as the note taker and participate in the interview observations/feedback.

2
  • 1
    A good answer - though clearly DIY testing isn't going produce the same quality of results as hiring a professional to do the research.
    – PhillipW
    Commented Aug 28, 2015 at 17:16
  • Thanks for your thorough answer Mark - you make a particularly good point about the need to manage the participants attention as well as their time. To give a bit more context we didn't actually design the site but are responsible for the redesign so impartiality may not be as big an issue as it might be, however, we have already made some recommendations based on an evaluation of the existing site so need to be conscious of a bias towards validating those! Thanks for your help
    – trkinnear
    Commented Aug 29, 2015 at 7:53
0

In the past, I've had my project manager conduct usability tests with users (who represent the target audience). The PM is familiar enough with the project to know how it works and is able to guide other people through the process of using the site/product, but also removed enough from it to stay objective - because they haven't been as deeply involved as a designer or developer (so you don't get a defensive reaction if a user is stumped or makes an honest comment about the product/site not working or being confusing etc).

Also, you mentioned budget... our user tests were usually conducted among relatives, friends, friends' kids (depending on the target audience) of anyone who worked there. We'd usually provide a few snacks and drinks etc as compensation :)

I think user stories (for informing design research) is something of a different beast, although you can certainly base them on people you know, but I think you'll want to do that at the beginning of the project, where, in my experience, the usability testing has come towards the end of the project (when there is something to interact with)

I hope I've understood your questions, and hope that helps!

3
  • Cheers binky! The reason for wanting to conduct design research and usability testing in the same session is firstly down to the fact that the project is a redesign (which I neglected to mention previously) - the client doesn't yet have any design personas to guide the project, only marketing personas based on demographic information of their customers, so we need to create these as well as test the current site to see what is and isn't working. Secondly it's a matter of resources as we only have one session with users before the redesign begins so want to get as much out of it as we can
    – trkinnear
    Commented Aug 29, 2015 at 8:13
  • that said there is definitely a trade off here as we may risk overloading our participants by trying to cover too much in the sessions!
    – trkinnear
    Commented Aug 29, 2015 at 8:13
  • Well, something you might want to try is maybe going over your existing sites and the sites of some competitors. You can get comments on what's working and what's not - if you have anything done for the redesign, like wireframes, mockups, you can run these things past the users too. I think it will still be very helpful to guide the process!
    – binky
    Commented Aug 31, 2015 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.