0

An A/B test done to introduce two new designs, version A and version B, collects three metrics, number of steps to reach page, System Usability Scale (SUS), and one question which uses Likert Scale.

Based on the above details (I made them up), can this be considered a Formative or Summative study?

I have been Googling around and reading in different pages, and PDF books, but there is no clear definition regarding this.

This website, https://www.nngroup.com/articles/formative-vs-summative-evaluations/ by Nielsen Norman Group, gives the following:

Formative: evaluations focus on determining which aspects of the design work well or not, and why. These evaluations occur throughout a redesign and provide information to incrementally improve the interface.

Summative: evaluations describe how well a design performs, often compared to a benchmark such as a prior version of the design or a competitor. Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product.

But, for this made up thing, the users get two test two different designs, i.e. specific selected users, prior to deciding if A or B shall be picked based on their final statistics.

Also read, Formative vs. summative research Quick and dirty versus slow and rigorous usability research.

Improve this question

1 Answer 1

Reset to default
2

I see this as Summative but honestly someone could argue for either case and be correct. Should you be picking your research methods and then figuring out what type of evaluation they are? OR should you be choosing what type of evaluation you need than pick research methods that fit into that.

Considering everything you have is made up I'm assuming you don't have an existing design/interface launched so you should be doing Formative evaluations building up to what will launch. As I said, arguably your methods could be formative but if you added some participatory design and interview style questions it would be much more Formative.

Improve this answer
1
  • Hi Dylan, thx for your reply. I arrived to the same conclusion. Mainly because of the following: click count per participant, SUS per participant, and question that every participant answered via Likert Scale (1-5). Based on this you still arrive to a Summative, also no issues are solved. Moreover, two high-fidelity artifacts i.e. two designs in A/B and one with best metric score is to be selected for deployment to all website users.
    – Alix Blaine
    May 13, 2020 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.