3

I want to gather some user feedback about an application process. I was thinking it would be ideal to 1) do real-time usability testing and 2) gather some feedback from people who completed the application in the last few months

I am a bit worried about gathering feedback from folks who completed the application already, since everything would be based off of their memory and their answers may be influenced by whether or not they passed the application. Are there any suggestions on how to best approach gathering this type of "after-the-fact" user insights? I want to make sure I ask questions that will give me reliable answers.

I'm thinking of providing screenshots of the application to jog their memory, and then asking questions like:

  1. Why did you choose to complete our application?
  2. What do you remember liking about the application process?
  3. What do you remember finding confusing about the application process?

What do you guys think about this approach and the reliability of info I might collect from this type of "after-the-fact" interview?
P.S. Is there a name for this type of "user evaluation"? Thanks in advance for anyone who can help!

Edit: I found this article from Nielson Norman. My takeaways are that a post-experience interview might be good for:

  1. Exploring general attitudes
  2. Exploring how users think about a problem
  3. The "critical incident" method is useful for post-experience interviews. Specific instances when something worked well or didn't work well are more vivid in a user's mind and can provide useful details.
  4. Beware of asking leading questions, and make sure questions aren't so specific that you are asking the user to make up an opinion about something that isn't necessarily important.
  • Could you offer the feedback survey before presenting the application results? – Nathan GoFundMonica Arthur Apr 14 '15 at 20:51
1

It is definitely good to consider the customer/user experience as a continuous and evolving metric, so taking samples or measurements at different points in time is the best way to gauge the direction and magnitude of the change. I don't know that there is a specific name or terminology for this, because I don't think you should necessarily see it as a discrete piece of information.

In fact, marketing research is the pre-experience user insight or expectation if you will (if done correctly), and real-time testing is working out how much the actual experience matches with the expectation, and your 'post-experience' insight tells you about first time use versus repeat use experience. So if you structure the testing and measurement in a way that allows the same types of information or metrics to be compared, then you will have a very powerful and persuasive set of user-generated data to make changes to your design.

But getting back to your original question, I think you should have a clear research question in mind, and if it only involves gathering feedback from users 'post-experience' (for example if this is for the training and customer support team) then structure the questions to block out factors relevant to customer expectations. However, as I mentioned before, this type of information makes more sense if you can reflect the whole continuum of the user experience. My advice is not to think about this a a one-off type of research, but rather how it fits in with the rest of the research work and what you want to find out from repeating this type of research.

  • Hi Michael, thanks for your comment! The purpose for gathering insights is to help inform my redesign of the process. I want to understand what's working well and what's not. I agree with you about ideally having a "pre, during, post" spectrum of information. I am planning to gather real-time insights as well by doing a think-aloud usability test (although it might be challenging because the application requires a lot of data input that they might not have on hand). – Lauren Dankiewicz Apr 14 '15 at 21:51
  • Hi Lauren, if you are collecting data and doing research to help inform your redesign of the process then I suggest the first step should actually be going back to the assumptions and design decisions you have made in the first iteration and working out where they influence or impact on the design. This is why the more 'agile + lean + ux' approach looks at trying to develop a minimum viable product that allows you to rapidly evolve and test ideas until you have something reasonably solid, then you can get into more solid data collection. – Michael Lai Apr 14 '15 at 22:09
  • I would agree that would be ideal. Unfortunately, this application was designed by an outside agency a few years ago before I was part of this team. I have talked with the product managers about certain business constraints to try to understand why things may have been designed a certain way. – Lauren Dankiewicz Apr 14 '15 at 22:12
  • I think you might find it interesting that during my google searching I came across a post you made a few years ago! ux.stackexchange.com/questions/44567/… – Lauren Dankiewicz Apr 14 '15 at 22:58
  • @LaurenDankiewicz the post isn't quite that old :p But it is interesting that research is still very much lacking in UX design decisions for a lot of organisations... happy to discuss further if you want help fleshing out some details with your questions :) – Michael Lai Apr 14 '15 at 23:08
0

You're quite right to say that post-experience feedback is based off the user's recollection and is influenced by various factors. Self-reported responses are a valid type of data collection and is used in research settings such as psychology where a participant is asked to recall an experience.

I would rephrase questions 2 and 3 to be more neutral. Users might not be thinking in terms of liking and "confusing" when asked to reflect on the application process.

  1. What did you like about the application process? (This question is open-ended and lets the user respond with their own terms, including neutral or none.)

  2. Was there anything you didn't like about the application process? (Avoid implying that there was something confusing in the process)

  3. What was the most difficult part of the application process? (The user has to decide which step was relatively difficult, even if they found the overall process very easy)

You can use their responses to probe deeper e.g. if someone says they didn't like or dislike anything about the process, you could ask them to compare it with other application processes they have done.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.