User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
One of our designers has redesigned a page of ours using insights from another piece of research we did months ago.
I want to test old design and new design at the same time to evaluate if the changes are indeed better?
Is it a good idea to set x3 tasks for both prototype to evaluate which is easer to use for participants via success rates and ratings along?
This isn't so dissimilar from A-B Testing, where you test two valid or "workable" options to see which works best for users. It'd probably be a fun thing to test, but there's a very big caveat here.
Long-time users will be used to the old design, and while some will invite change, others may think the old one is better or easier for them, no matter how efficient and perfect the new one may be (and better for them in the long run). People like creating their own systems to work around difficult tasks, and it can be hard to abandon those hacky forms of navigation. So I'd strongly recommend you get a healthy balance between long-time users, recently-added users, and people who've never seen your site before.
I'd also suggest you focus as much as possible on quantifiable results, not just whether someone says they find one thing easier or another (though that's very important). It sounds like you're already thinking that way with success rate, just again, keep in mind that long-time users may be so familiar with the old system that they're successful in spite of its problems.
Of couse you should test it. Clear and consistent navigation is a basic requirement for the user interface. So maybe your changes have a great influence on indices and increase conversion.
Yes, test both ideas.
When testing different versions of the same flow, you'll find the sequence of exposure to designs might play a role in results. To mitigate this, I would suggest splitting the testers into two groups. Take the first group through the old version, followed by the new one and the second group through the new version, followed by the old one.
Yes, this type of test would be a "between-measures, within-group experiment". You are testing two conditions (old design and new design) within the same group of participants.
Aside from getting user ratings, you should set easily measurable tasks such as time-taken for a task, accuracy and success rates, so that you'll be able to reach a conclusion.
It's good practice to test before and after applying UI changes and it's something that's not done often enough in UX.
