How to gather "post-experience" user insights

Question

I want to gather some user feedback about an application process. I was thinking it would be ideal to 1) do real-time usability testing and 2) gather some feedback from people who completed the application in the last few months

I am a bit worried about gathering feedback from folks who completed the application already, since everything would be based off of their memory and their answers may be influenced by whether or not they passed the application. Are there any suggestions on how to best approach gathering this type of "after-the-fact" user insights? I want to make sure I ask questions that will give me reliable answers.

I'm thinking of providing screenshots of the application to jog their memory, and then asking questions like:

1. Why did you choose to complete our application?
2. What do you remember liking about the application process?
3. What do you remember finding confusing about the application process?

What do you guys think about this approach and the reliability of info I might collect from this type of "after-the-fact" interview?
P.S. Is there a name for this type of "user evaluation"? Thanks in advance for anyone who can help!

Edit: I found this article from Nielson Norman. My takeaways are that a post-experience interview might be good for:

1. Exploring general attitudes
2. Exploring how users think about a problem
3. The "critical incident" method is useful for post-experience interviews. Specific instances when something worked well or didn't work well are more vivid in a user's mind and can provide useful details.
4. Beware of asking leading questions, and make sure questions aren't so specific that you are asking the user to make up an opinion about something that isn't necessarily important.

Answer 1

It is definitely good to consider the customer/user experience as a continuous and evolving metric, so taking samples or measurements at different points in time is the best way to gauge the direction and magnitude of the change. I don't know that there is a specific name or terminology for this, because I don't think you should necessarily see it as a discrete piece of information.

In fact, marketing research is the pre-experience user insight or expectation if you will (if done correctly), and real-time testing is working out how much the actual experience matches with the expectation, and your 'post-experience' insight tells you about first time use versus repeat use experience. So if you structure the testing and measurement in a way that allows the same types of information or metrics to be compared, then you will have a very powerful and persuasive set of user-generated data to make changes to your design.

But getting back to your original question, I think you should have a clear research question in mind, and if it only involves gathering feedback from users 'post-experience' (for example if this is for the training and customer support team) then structure the questions to block out factors relevant to customer expectations. However, as I mentioned before, this type of information makes more sense if you can reflect the whole continuum of the user experience. My advice is not to think about this a a one-off type of research, but rather how it fits in with the rest of the research work and what you want to find out from repeating this type of research.

Answer 2

You're quite right to say that post-experience feedback is based off the user's recollection and is influenced by various factors. Self-reported responses are a valid type of data collection and is used in research settings such as psychology where a participant is asked to recall an experience.

I would rephrase questions 2 and 3 to be more neutral. Users might not be thinking in terms of liking and "confusing" when asked to reflect on the application process.

2. What did you like about the application process? (This question is open-ended and lets the user respond with their own terms, including neutral or none.)

3. Was there anything you didn't like about the application process? (Avoid implying that there was something confusing in the process)

4. What was the most difficult part of the application process? (The user has to decide which step was relatively difficult, even if they found the overall process very easy)

You can use their responses to probe deeper e.g. if someone says they didn't like or dislike anything about the process, you could ask them to compare it with other application processes they have done.