How to informally test users’ knowledge of app’s undocumented behavior?

Question

The app I work on has some undocumented but consistent behavior when filtering and sorting a queue.

The formal requirement is that, if the filter returns more than 1000 results, a warning message is displayed and only 1000 results are shown.

As an implementation detail, it so happens that the app in fact displays the first 1000 results (as determined by the sort order the user had specified).

This is not documented anywhere and was never intended as a guarantee; it was just a consequence of the specific implementation. But it has worked this way consistently for many years.

To improve database performance, the devs would like to modify their implementation of this feature in a way such that this would no longer be true; users would get 1000 matching results but there would be no predictable pattern to which 1000 they get.

My current but untested intuition is that users are not aware of the exact current behavior and do not rely on it, so it would not upset anyone if it suddenly changed.

But I’m not really sure how to test this assumption. I don’t want to explain the current behavior to users because I’m trying to find out whether they already know about it; it is easy to say “yes, I knew that” after I’ve already explained it. But I don’t know how to find out if anyone relies on something without explaining what it is!

Answer 1

How to informally test users’ knowledge of app’s undocumented behavior

After the warning, and along with the 1000 partial results, present a somewhat unobtrusive note or easily dismissable popup that asks:

Are these the results you expected? 

Yes No

My current but untested intuition is that users are not aware of the exact current behavior and do not rely on it, so it would not upset anyone if it suddenly changed.

Maybe and no.

Users may be unaware exactly what has changed but if they've used the search often enough they'll have internalized an expected model. And when a model is presented that doesn't match the internalized model they'll feel frustrated, especially if they don't know why.

Answer 2

As I understand the question we want to figure out "what is the best of the two ways to present the data? Is a random return better than a sorted-by-date return?". I'd try to approach it two ways, hopefully one is feasible in your case:

1. Understand why users would use this functionality (i.e. when does it happen that a user will filter in such way that >1000 results appear. What is their goal?). Create a task replicating this goal and test it with a couple of users in a moderated or unmoderated scenario. See what they say about the results returned. You can also measure perceived usability to get quantitative data, if you use the System Usability Scale you can probably get good results for 8 people testing one version and 8 people testing the other.

2. If you have the ability to A/B test, put both versions up 50-50 split for some time until you achieve statistical relevance. Find a couple of metrics to track: for example you can see interaction rate, how many pages the user views (assuming that they can see all data paginated), number of retries etc. Another idea, put a satisfaction scoring near the table "How satisfied are you with the results?". See if any of the two versions have any significant difference in those metric.

Hope it helps :)