How to prompt users to give feedback on widget design without biasing their answers?

Question

We have a set of matching widgets we use as a homepage for an app. The user can interact with those widgets causing various notification dialogs. I think these dialogs should match the look and feel of our app/widgets. My bosses have opted to have them match standard unstyled android notifications as they reason that it's easier for the user to understand that they are being prompted to do something.

My bosses also said if any users mention it then we can re-discuss the options.

How can we prompt users to give feedback on such things in user testing without telling them it's because we consider they might need to match and biasing their opinion?

I have a feeling users will not mentions such things. I can't convince the bosses to do AB testing which means we can never get feedback from the other perspective, only from the perspective of people using the standard notifications.

Answer 1

Based on your answer to my clarification question, I would suggest that you start by testing if the users are able to use the dialogs properly as-is. Split your test group in two, based on previous Android experience (or not). Set up scenarios that you think will be "jarring" (as you put it), and see which users struggle. Also set up scenarios that should be easy/straightforward (that also rely on the dialogs) and see which users struggle.

At the end of the tests, ask them (in an open-ended way) if they could suggest any improvements (do not mention the dialogs), both functionally and aesthetically. Then count how many highlight the dialog styling as being out-of-place. Make your case with the data. If the users are able to perform the tasks to your client's satisfaction, then maybe the styling isn't such a major concern.

I suspect that users will grow accustomed to the look and feel of these "system" dialogs, in much the same way that desktop users know what the standard "Open File" and "Print" dialog windows look like for their operating system. Your use-case may be more complex, in which case the tests should yield some meaningful insights.

Answer 2

Why not use UsabilityHub?

Test your designs and mockups on real people

Learn how visitors will interact with your design before launching. Find out what they think, what they remember, and where they make mistakes.

Create a usability test where you pair up your widget design and versus your boss'.

See who gets the better feedback and go from there.

Answer 3

I know this doesn't answer your question but generally you should stick to the system standards when possible. A user, having never seen your app, should know a notification for what it is. If you can embellish it with your app's brand in the process, great. Good luck, the politics in this situation can be tricky, especially when you're proud of your work.