In a lot of user testing sessions I've been in a common task is to test the effectiveness of a particular icon or navigation control.
The typical scenario:
tester: How would you accomplish task x?
testee: They stare at the screen for a while because they are in 'test mode' and feel like they need to narrow it down to one and only one option before answering.
The failing here, in my opinion, is that this excludes the natural discovery process that many people use when looking at a new app--especially on touch devices.
For example, many apps are heavily icon-driven (instagram, Facebook, Flickr, etc) and even for a visual designer such as myself, I often don't sit there and attempt to decipher each icon before tapping it. I just start tapping and playing with the app.
Is 'playing/discovering' a common technique for users to figure out how to navigate an app?
If so, have you figured out ways to accommodate this in your user testing scenarios?
The situation I'm trying to avoid is one where we do some user testing, and we find out that a particular icon wasn't immediately identifiable by the user when asked. We then consider that a problem, even though I hypothesize that outside of the user test, people would just tap on it then immediately know what the icon is. In other words, this is an issue with the test--not the actual icon. Alas, we then focus time and energy on changing an icon instead of fixing more pressing usability issues.
UPDATE: I clarified the scenario to make it more task-focused rather than feature-focused. The feedback about making sure it is task-focused is valid and appreciated. That said, I think the concern is still there in terms of making sure exploration is encouraged.