4

As product managers, UI/UX specialists, engineers, etc., we often deceive ourselves into believing that when we test a new product feature, we’re testing an idea, when the fact of the matter is, we’re testing a particular instantiation of an idea. If, in accordance with whatever metric we’re using, we validate a product feature, we correctly interpret it as a validation of an idea. However, if a feature underperforms the standard which we’ve set for it, we often wrongfully consider this to be a reflection of the viability of the idea or abstract concept that the feature represents, rather than (correctly) stating that it’s merely that particular implementation of a feature that has failed.

So I'd like to ask: do you consider it to be possible to test an idea, rather than testing an instance of an idea? If so, what are your methods (statistical, analytical, etc.) for doing so? How should the failure of a feature reflect on the idea that the feature represents? At what point, do you consider an idea invalidated, rather than considering a particular instantiation of an idea invalidated?

Improve this question

2 Answers 2

Reset to default
2

I guess the way to test the idea and not the form you take to represent that idea is to test several forms of that idea, and then take conclusions from it.

Creating several solutions and then making A/B testing could reflect if the problem is in the concept itself or in the implementation.

Solutions should be different from each other, not just variations. The more solutions tested the more information obtained. Results could show:

  • An implementation has significantly more success than the others. This would imply that the other implementations didn't let the idea work.

  • The overall is good or bad for your expectations. Either the idea is not working or the solutions tested until then were not correct. The moment you see slightly success on a solution analyse what does that solution have that others don't. If the overall is good then the idea really works regardless of the solutions you tested.

Improve this answer
2
  • I was thinking along the same lines, but what would be a good quantitative threshold, or rule of thumb, to decide that the several instances of an idea that you've tested have given you enough data that you can, for pragmatic purposes, say that you've ascertained enough data to "evaluate the idea?"
    – National Acrobat
    Dec 19, 2016 at 17:37
  • That problem would also happen when testing a feature. It depends on your idea of which threshold values are alright to consider it successful. I edited the answer.
    – Alvaro
    Dec 20, 2016 at 9:08
1

You're probably going to get the question closed as being too contentious, but I have to agree (I think I'm agreeing with you!) that no, it's not possible to test a pure idea.

The reason why it's not possible is probably related to the Sapir-Whorf hypothesis that it's not possible to think about something if we have no words for it, i.e., our words determine our thoughts.

Once we "have an idea", it's expressed in words, and any tests we can create depend on those words. So....

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.