2

I am part of a small team of UX designers assigned to a identify and implement positive change to an integrated suite of 12 products.

I am looking for an online tool that would allow me to enter user testing results across a product map with the intent to identify areas that have the highest number of pain points. The user base would be large (100-500) to start, so I am ok with automated tools, although I prefer in-person testing and entering the results after the fact.

The output should be similar to a journey map. However as these products are integrated I'd like to be able to compare results at the platform, products and task level.

I have done my research. However, most companies do not show the product in a detailed way, and if I do go down this route I can't afford to pick a product that doesn't complete the required tasks. Which is another part of this challenge, the budge may be minimal or even $0.

I'm also not convinced this is the correct path to go down, it's just my working theory. I know we are supposed to ask clear questions, so I'll break it down and look forward to the communities response.

  • Is this a valuable exercise? "Yes" or "No" and why.
  • If "yes", what is the most cost effective tool to enter and share results in a journey map format?
  • If "no", is there a better way?
  • While none of the answers addressed the full scope of the question I appreciate the effort. – Johnny UX Jan 30 '18 at 17:36
4
+200

If you're looking for an almost free tool, I need to tell you that Google Sheets is the best option.

Here's how I set it up. It's called a "Rolling list of Observations"

Columns U1, U2 etc are representing individual users.You start with the U1, note the observation(let's call it O1) in the observation column, and mark "1" for this particular user(U1).

Now when doing the testing with U2, check wether the obserations are valid for this new user too, if yes, mark "1" else just mark a "0".

The best part of this method would be, that you just need to find the total in order to get the frequency.

For ex. The yellow column, represents how many users faced a particular observation. You could sort the sheet according to this total and get a list of observations which are most important.

You could further add columns to classify observations (like add a column for "product name" / "step in the flow") so that you could filter / sort easily later

enter image description here

1

User testing in order to get a view of how to improve a product is the way to do it. IMHO there is no better way.

BUT... In the first place it all depends on what 'user testing results' you are going to collect and how you would actually get hold of the data from your user population. My preference would be to write a PHP script if you are going to automate this process but doing things by hand is a VERY useful exercise if you can afford the time.

From what you write it seems as if you have identified a number of points in each of the 12 products and will be using some kind of rating scale to assess ease of use / pain at each of these points. So you need something really very simple like a SMEQ rating scale. Will each product have the same points, so you can meaningfully compare between products (eg login, print, database access, yada)?

In which case you need nothing more complicated than a spreadsheet IMHO! Same format of columns (1 column per point), each participant to their own row, one page per product. In each cell you put the SMEQ or difficulty / pain score. You can derive any number of fancy stats from this kind of arrangement, have fun with the numbers!

However, if you have a different set of points in each product and each product does different things so direct point-by-point comparison is not possible, then you would be best advised to use an overall quantified questionnaire (I blush to recommend sumi.uxp.ie but I do own this tool having worked on it for over 20 years!) and to ask each user some 'critical incident' type questions: pain points and delight points come out spontaneously from user comments. This will allow you to compare user experience with products that do different things as well as give you some idea of where the pain lies with each product. You can use SUMI online to get a big data set, or in the lab in conjunction with more intensive personal evaluations. I always encouraged my students to do both.

  • For the first phase I intend to break each product down into tasks. Then test the tasks on either a pass fail basis. The criteria will be wether or not users can complete the task without assistance. The users would be able to enter comments as either pain points or positives. I hadn't considered using a SMEQ score, but I think it may work better. I had considered using a spreadsheet however, I do want the final result to be sharable and in the version of a task map or workflow. I did review your link, I don't think the sample results is exactly what I was looking for. – Johnny UX Jan 16 '18 at 16:24
1

This will probably only partially answer your question, but here are my two cents. I don't think you need that many test subjects. NNG recommends around five. Not saying you should stick to this low number, but you may consider targeting way less than 500.

I found about this tool for measuring usability in one of Eric Reiss's seminars. It is a nice and structured way to collect, analyze and share your findings. Again, not saying you should use the exact same process, but I believe you can find some parts useful.

Eric Reiss - Users, experience, and beyond slides 50 to 60

Hope this helps, cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.