Analysis Device Guide for a Client

Question

I hope falls into UX.

We've just completed some analysis work for a client for a new mobile application. Part of that analysis work was to provide user experience analysis from their preferred device type, a 10" tablet, to the users preferred device type, a 8" tablet.

We have submitted our report back to the client including our findings of the differences between the two devices, however we have kept our analysis to simple facts, i.e. loading an example app onto both devices and giving the size and percentage reduction in elements, grids etc.

The client has come back and isn't generally pleased with this factual breakdown and wants us to give a "subjective view" on which device is more usable.

I'm reluctant to do so.

Is it the job of the UX analyst to provide subjective opinions like this or should we stick to what we know? In my mind, how can we make an informed decision without loading a prototype onto both devices and performing an A/B test.

Your experiences, advice and references would be greatly appreciated in how to respond to this client, without committing to advice that could lead to hefty device purchases, that may be wrong.

Answer 1

It is absolutely within the scope of a UX practitioner's skill set to collect qualitative (i.e., subjective) data. As to if it is in the contract's scope, with your client, is different.

In my mind, how can we make an informed decision without loading a prototype onto both devices and performing an A/B test.

The question is unclear as to why this isn't an option. But being able to do so it certainly important for performing many qualitative type tests, but not required by all.

Side bar: Note that putting a new application on two different tablets is not an "A/B Test". An A/B Test requires a "control" and a "treatment". Since the user is unfamiliar with using the application on either tablet size, you have no "control". A/B Tests also produce Quantitative (i.e., not subjective) data.

Above shows many different types of user research that can be performed which fall all around the spectrum. (Image taken from: When to Use Which User-Experience Research Methods).

The green dots represent research methods that do require the actual product to be used. But there are many methods that can be used to gather data which don't necessarily require the product to be in the user's hand.

From the list above, some examples of how to gain quantitative feedback on the differences between an app on two different tablet sizes could include:

• Focus Groups
• Interviews
• Concept Testing
• Desirability Studies

Getting the product into the hands of the user should absolutely be a goal though. If the customer doesn't want that, for some reason, it should become your job to convince them to change their mind. If they're asking for qualitative data, getting users to use the product is a great way to get it.

It does not require the purchase of 20 of each device. Having 1 of each is certainly a key point though. You're testing basic device size, not the device itself. While each device of the same size might be a little different, you're goal should be to present a subjective summary of a "generic 10-inch" and "generic 8-inch" device.

The client should be involved in selecting what the "generic" device is, and pay for it.

In addition to the above examples of user research methods, a breakdown of many other qualitative methods can be found in this article: Qualitative UX Research. Some examples that, again, don't necessarily (though it still helps) require you to have the product in the user's hands:

• Expert Review
• Task Analysis
• Persona Development

Running through a heuristics evaluation on the interface falls inline with an "expert review", among other methods, which gives you insight into potential issues.

Conducting a task analysis can also give you insight into if the user can accomplish their task with a 10" vs. 8" device. Maybe all your users have pockets that can only ever fit an 8" device, and a key element of their task flow is to carry the thing around all day (which they would likely prefer to do in their pocket, vs. their hand).

Persona development gives you the ethnographic landscape of the user base. Helping you to show that 10" displays aren't "hip" and "cool" or that 8" displays are the ones that are "so last year".

It is unclear from the question why you are "reluctant to [perform qualitative user research]" to gain a subjective view from your user base. As "Strengths and Weaknesses of Quantitative and Qualitative Research", from UXMatters, points out:

Qualitative research studies can provide you with details about human behavior, emotion, and personality characteristics that quantitative studies cannot match. Qualitative data includes information about user behaviors, needs, desires, routines, use cases, and a variety of other information that is essential in designing a product that will actually fit into a user’s life.

This is a vital aspect of user experience.

Answer 2

Good UX analysts have solid technical skills. Great UX analysts also have experience

...and experience in almost any field transcends quantitative analysis.

So yes, I think this is absolutely within the scope of UX analysis.

There are many concepts in UX that are better conveyed qualitatively or in principle than by hard analysis. For example:

• This application presents dense data tables, so these will generally be easier for users to work with on larger screens.
• The tablet is being used on construction work sites where weight is a premium and usage is infrequent, so smaller form factors are preferable.

You could certainly quantify these recommendations if you really need to, but they are prima facie evident enough that it'd be a waste of client money to do so.

For contractual work I'd recommend two things:

1. Check your service agreement to make sure you understand what you are liable for. If you are providing subjective then make sure you are legally covered for consequential damages that might arise from a customer following you negligent (in hindsight) speculative advice. You may need to get a carve-out to cover the client's subjective request.

2. Clearly label what is speculative/subjective and note that you are providing this at the specific request of the client. This will help guard against future whodunit claims if your subjective opinions turn out to be wrong.