For example, if we have some mockups and we want to test it, using a lab test, to check if they are ok to implement we should:

  • Do only a formative test to identify issues?
  • Or we can also do a summative with the mockups to verify errros, eficiency and effectiveness? Or this summative test should only be after implement? Also it seems that its ok to test with just 5 users to identify problems, that is to make formative tests, but for summative (to measure errors, efficiency and so on) tests, with just 5 users can be enough also?

  • Note, but if we should only do a formative test with the mockups, a formative test should include what? What can we test to identify problems? We can use summative metrics in the formative test?


Should I perform summative testing on mockups?

It doesn't make sense to perform summative testing on mockups. The purpose of summative testing is to confirm empirically that a developed solution works.

As Akendi say:

Summative usability testing is a Quality Assurance (QA) type of test usually performed later in the product development process.

Things like think aloud protocol and hints for the user when they get stuck are left out in this kind of testing. The pass/fail metric is the most important measurement in this test.

In contrast, the purpose of formative testing is to find out why a design is not meeting the user goals. You can then use this qualitative data to refine your mockups.

What should a formative test include?

User Focus describes this very succinctly:

With formative tests, you want participants to think out loud, describe what it is they are trying to do and let you know when they’re confused.

In short, it should include anything which can gain insight from you user which will allow you to further develop your mockup.

Usability Body of Knowledge lists some methods often used in formative testing:

  • Heuristic evaluation
  • User interface inspections
  • Thinking-aloud testing
  • Pluralistic usability walkthrough
  • Cognitive walkthrough

Not the answer you're looking for? Browse other questions tagged or ask your own question.