Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I bumped into this document and it suggests the use of (a) Formative and (b) Summative Usability Testing.

To be honest, is the first time that I bumped into this kind of terminology, and I while reading about it, it's not quite clear to me how can I use these tools to assess usability.

Can anyone give me an example of each?

share|improve this question

4 Answers 4

up vote 13 down vote accepted

Short version:

Do Formative Usability Testing at the start of the design phase, testing with paper-prototypes and similar. Do this to discover insights and shape the design direction.

Do Summative Usability Testing during latter half of the development phase, testing with actual working prototypes. Do this to determine metrics (time on task, success rates), which you would then compare test results from improved versions against.


Longer version:

In an article by Jeff Sauro where he discusses whether the two terms are useful he defines them as:

Most usability testing involves finding and fixing problems as part of an iterative design process to make an interface more usable. It is typically called a Formative Usability Evaluation. In contrast, a Summative Usability Evaluation describes the current usability of an interface—as measured by things like task times, completion rates and satisfaction scores. Summative tells you how usable an interface is and formative tells you what isn't usable.

In other words - Formative reports on what is broken, and Summative is for what is working.


Akendi describes Formative and Summative usability testing on their site, and has this to say:

Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process and - if applied early in the development process - provides valuable insights of where users have difficulty reaching their user goals with the technology (website, desktop GUI design, hardware product) or service.

They go on to describe using sketches or paper prototypes for the formative testing to identify if they are heading in the right direction.

And then this…

Summative usability testing is a Quality Assurance (QA) type of test usually performed later in the development process. A similar usability test protocol is used as in formative usability testing but now this setup is used to do formal user acceptance testing before the product is released to the target audience.

… which works on the assumption that the design direction is the right one and that there is potential for a usable solution, and thus the summative usability testing determines whether the execution actually delivers on that promise.


UserFocus have a summary of Formative and Summative usability testing in an article about Common Industry Format for reporting usability tests.

The CIF makes a distinction between "formative" and "summative" usability tests.

Formative tests are carried out:

  • During the development of a product;
  • To mould or improve the product;
  • Virtually anywhere (you don’t need a lab);
  • With the test administrator and the participant co-present.

The outputs from a formative test may include:

  • Participant comments in the form of a "thinking aloud" narrative (for example, attitudes, sources of confusion, reasons for actions);
  • Photographs and highlights videos;
  • Usability problems and suggested fixes.

In contrast, summative tests are carried out:

  • At the end of a development stage;
  • To measure or validate the usability of a product;
  • To answer the question: "How usable is this product";
  • To compare against competitor products or usability metrics;
  • To generate data to support marketing claims about usability;
  • In a usability lab;
  • With the participant working alone.

The outputs from a summative test may include:

  • Statistical measures of usability (for example, success rate, average time to complete a task, number of assists);
  • Reports or white papers.

If you don't do Formative Usability Testing, you might end up designing something that simply will never be a usable solution (e.g. a baby crib made from barb wire and broken glass, vs. one of wood and fabric). If you don't do Summative Usability Testing then you won't find out if your potentially usable solution has any problems or not (e.g. a baby crib made of wood and fabric, but with very awkward latches, lead paint, and nasty nails sticking out everywhere).

share|improve this answer
    
Thanks Erics. Learned something new! –  Tyler Langan Nov 23 '12 at 9:05
    
I think this rather muddles a/ the development stage at which testing is being carried out with b/ the kind of testing (qualitative or quantitative). –  PhillipW Mar 15 '13 at 10:49
    
Yes, it is important to remember that quantitative measures can be used for both formative and summative testing. As can qualitative measures (usually to a lesser extent). While qual methods are more suited to formative testing, and quant methods to summative, don't take them as being synonymous -- you can use quant measures in formative testing, and (to a lesser extent) qual measures in summative. –  Erics Mar 26 '13 at 5:42
    
+1 Well explained note! –  inkmarble Jul 19 '13 at 4:05
    
I have a bit of a problem with the sentence "Do Summative Usability Testing during latter half of the development phase". Properly done, a summative evaluation is time and money consuming and involves complexities like statistical analysis. It is hardly justified before the design has been signed off and implemented (unless one conducts an informal summative evaluation). See this for more. –  Izhaki Jan 15 at 19:32

I wouldn't be too caught up on the terminology because more often than not they are either used incorrectly or misleading. The important take home message is that you need to be doing testing at different phases of the design/development cycle or sprints, and that at different stages you will identify or encounter different usability issues depending on how you do the test and who the test subjects are. In order to try and anticipate all the design issue that may arise, you need to get as close to the representative users as possible. Of course, it is not worth introducing all this testing if it has a major impact on your development path, so it is about deciding what to do, when and how much of it to do while still being able to deliver on time with reasonably good quality.

Examples of formative testing, which will generally be based on loose requirements/guidelines, involve working through things like the layout design, interface design and interaction design either internally or throw it open to external users. Examples of summmative testing, which will generally be based on design and build specifications, involve ticking off each of the functions and features to make sure everything works as per the design intention.

share|improve this answer

The mnemonic rule to remember:
F.irst, For.mative == F.ind O.bjectives to R.epair
S.econd, Sum.mative == S.uccess U.sability M.etrics

share|improve this answer
    
+1 Thats a good short form of the description! –  inkmarble Jul 19 '13 at 4:04

These are just two names to describe the usability testing at two different stages of a project life cycle. Formative testing is more about gaining insights on user perceptual experience and requirements at intial stages, while Summative testing is more about final product-usage experience and satisfaction. Summative testing sums up overall experience offered by a product.

Formative Testing

  • Need finding for the website and product
  • Paper prototype evaluation (card sorting, task analysis)
  • Heuristic and qualitative testing (Observations, Interviews)
  • Testing usability at the end of each sprint
  • Common metrics include, Error free navigation, UI issues, Ease of use, User expectations, Learnability, and efficiency

Summative Testing

  • Site, App or product validation against goals and benchmarks

  • Quantitative Surveys, and Qualitative (User Comments, Sentiment Analysis, Interviews etc)

  • Behavioral Metrics (Web Analytic, Performance, Success rate), Physiological measures (Cognitive load, memory, attention, affective state and engagement), Subjective measures ( Satisfaction, expectations, recommendation, opinion, emotions, and overall user experience)

share|improve this answer
    
Can you back your answer with supporting references/links? –  rk. Jul 18 '13 at 1:38
    
Dear @rk. UX community borrowed these formative and summative methods from teaching and learning methodology. You can find many references in learning and teaching domain, where they use these terms to define assessment at the initial stages and the end. –  Dr_Laghari_UX Jul 18 '13 at 2:10
    
While that is true, the answer needs to be complete in and of itself. And right now it looks like your personal views, which is why I asked for the references. –  rk. Jul 18 '13 at 2:16
    
Yes, you are right. The earlier post included my personal understanding on testing concepts, and this is how I conduct formative and summative testing during product evaluation. –  Dr_Laghari_UX Jul 18 '13 at 15:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.