What is the role of a test user that you invited to test your app and how do you handle communication with them?

Question

We have a bit of a quarrel in our team around this topic. Here is what happened without going into much detail:

We are developing a new social network and are in the alpha stage right now. To test our app we invited over 100 people from our group of friends and people we know, including a few users via advertising. Every user was invited with the premise that the app is in the alpha stage and that we need them to test our app and give us feedback in the form of bug reports and feature requests.

What led to the quarrel was the fact that one of the team members used the fixed user tagging to tag the user that reported it was not working. He tagged the user in the comments of a "post" and got tagged back with the user confirming it worked now.

Some team members got pissed that this interaction happened.

We are split on what the role of our test users are and what we can and/or should expect of them. Should the communication be one sided or two sided?

Some of us think the users should test the app and if they find a bug or need a feature use the feedback form in the app to report them. And that is all they are supposed to do.

Another part of our team thinks that is not enough and we should be more proactive in how we interact with the test users. Ask them if the bug they reported is fixed or what they think of the new implemented features.

Are there any guidelines for this that are good practice? I couldn't find anything useful for our problem.

Answer 1

The answer to your question is... it depends.

If the goal of your alpha stage testing is to fix bugs and high level usability problems quickly, then there is no issue with an iterative feedback loop, as this will likely allow you to implement changes quicker. However, be aware that if one user says something is an issue or requests a feature - it may not be worth fixing or investing in. Generally speaking, best practice is to hear similar issues or feature requests from 5 - 12 users.

If the goal of your alpha stage testing is to collect information on the most common bugs and high level usability issues or the most frequent feature requests, then it may be more beneficial to leave it one-sided, collect your feedback, then analyze the data to spend your time and resources on the more frequent or highest severity issues.

Getting the feedback is a great first step, now you just have to determine what method is most beneficial for you at this stage.

Answer 2

I think a better process for this might be to define tasks that the test users should try to accomplish and have them rate the difficulty of each and whether they were able to complete it or now. If they encounter a block, have them report whether it was due to a technical bug, or confusing experience.

By approaching the issue this way, you get a more fruitful interaction compared to just letting the users go on bug scavenger mode.

So now you might have 20 users that have failed on a task, and you don't have to interact with all of them, because now your entire sample has gone through similar tasks, and you can narrow down the group that needs to become involved in the feedback process. In short, standardize the test conditions so that you can then reduce the required sample for feedback loops or produce new samples by just having testers go through the same task sequence.

Answer 3

Why do those team members only want testers to use the form for reporting bugs and requesting features? Is it because they don't want bugs to be exposed to the other testers? Also, will this cohort of testers be converted to users when the app moves out of alpha? I can understand how this could potentially get messy down the road if you don't decide on what to do now.

An example of maintaining a continuous feedback loop with user is Uservoice where users can post bugs and requests and see what other users have posted and vote on features. The pros of keeping communication open is so that testers can verify the fixes, just like QAs. The answer depends on how you want to engage this group of testers...

Answer 4

The role of the test user is defined by the objective for doing the user testing (as this can vary depending on the stage of your product/service and the resources available), so there is definitely no hard and fast rule.

However, it is important to have a defined objective to avoid issues with inconsistency of how the testing is performed (if this is important for your metrics/results) and what the next steps will be.

Based on your description of the event, it feels like there could be a number of different things that you want to test with the application. So you could have either grouped the users and have each group undergo a different type of testing (if you believe that one test affects the results of another test), or allowed the entire group to do all of the testing (if you needed the numbers for the type of tests conducted).

Keep in mind that the most important part of user testing is to ensure that you don't set the process up to show the results that you are expecting (i.e. don't introduce unnecessary bias), which is why there is usually an emphasis on the consistency of the process. However, the bigger picture is to try and fill in gaps between what you think or assumed in the design process with a representative group of users.