I bumped into this document and it suggests the use of (a) Formative and (b) Summative Usability Testing.
To be honest, is the first time that I bumped into this kind of terminology, and I while reading about it, it's not quite clear to me how can I use these tools to assess usability.
Can anyone give me an example of each?
Short version:
Do Formative Usability Testing at the start of the design phase, testing with paper-prototypes and similar. Do this to discover insights and shape the design direction.
Do Summative Usability Testing during latter half of the development phase, testing with actual working prototypes. Do this to determine metrics (time on task, success rates), which you would then compare test results from improved versions against.
Longer version:
In an article by Jeff Sauro where he discusses whether the two terms are useful he defines them as:
Most usability testing involves finding and fixing problems as part of an iterative design process to make an interface more usable. It is typically called a Formative Usability Evaluation. In contrast, a Summative Usability Evaluation describes the current usability of an interface—as measured by things like task times, completion rates and satisfaction scores. Summative tells you how usable an interface is and formative tells you what isn't usable.
In other words - Formative reports on what is broken, and Summative is for what is working.
Akendi describes Formative and Summative usability testing on their site, and has this to say:
Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process and - if applied early in the development process - provides valuable insights of where users have difficulty reaching their user goals with the technology (website, desktop GUI design, hardware product) or service.
They go on to describe using sketches or paper prototypes for the formative testing to identify if they are heading in the right direction.
And then this…
Summative usability testing is a Quality Assurance (QA) type of test usually performed later in the development process. A similar usability test protocol is used as in formative usability testing but now this setup is used to do formal user acceptance testing before the product is released to the target audience.
… which works on the assumption that the design direction is the right one and that there is potential for a usable solution, and thus the summative usability testing determines whether the execution actually delivers on that promise.
UserFocus have a summary of Formative and Summative usability testing in an article about Common Industry Format for reporting usability tests.
The CIF makes a distinction between "formative" and "summative" usability tests.
Formative tests are carried out:
- During the development of a product;
- To mould or improve the product;
- Virtually anywhere (you don’t need a lab);
- With the test administrator and the participant co-present.
The outputs from a formative test may include:
- Participant comments in the form of a "thinking aloud" narrative (for example, attitudes, sources of confusion, reasons for actions);
- Photographs and highlights videos;
- Usability problems and suggested fixes.
In contrast, summative tests are carried out:
- At the end of a development stage;
- To measure or validate the usability of a product;
- To answer the question: "How usable is this product";
- To compare against competitor products or usability metrics;
- To generate data to support marketing claims about usability;
- In a usability lab;
- With the participant working alone.
The outputs from a summative test may include:
- Statistical measures of usability (for example, success rate, average time to complete a task, number of assists);
- Reports or white papers.
If you don't do Formative Usability Testing, you might end up designing something that simply will never be a usable solution (e.g. a baby crib made from barb wire and broken glass, vs. one of wood and fabric). If you don't do Summative Usability Testing then you won't find out if your potentially usable solution has any problems or not (e.g. a baby crib made of wood and fabric, but with very awkward latches, lead paint, and nasty nails sticking out everywhere).
The mnemonic rule to remember:
F.irst, For.mative == F.ind O.bjectives to R.epair
S.econd, Sum.mative == S.uccess U.sability M.etrics
These are just two names to describe the usability testing at two different stages of a project life cycle. Formative testing is more about gaining insights on user perceptual experience and requirements at intial stages, while Summative testing is more about final product-usage experience and satisfaction. Summative testing sums up overall experience offered by a product.
Formative Testing
Summative Testing
Site, App or product validation against goals and benchmarks
Quantitative Surveys, and Qualitative (User Comments, Sentiment Analysis, Interviews etc)
Behavioral Metrics (Web Analytic, Performance, Success rate), Physiological measures (Cognitive load, memory, attention, affective state and engagement), Subjective measures ( Satisfaction, expectations, recommendation, opinion, emotions, and overall user experience)
I wouldn't be too caught up on the terminology because more often than not they are either used incorrectly or misleading. The important take home message is that you need to be doing testing at different phases of the design/development cycle or sprints, and that at different stages you will identify or encounter different usability issues depending on how you do the test and who the test subjects are. In order to try and anticipate all the design issue that may arise, you need to get as close to the representative users as possible. Of course, it is not worth introducing all this testing if it has a major impact on your development path, so it is about deciding what to do, when and how much of it to do while still being able to deliver on time with reasonably good quality.
Examples of formative testing, which will generally be based on loose requirements/guidelines, involve working through things like the layout design, interface design and interaction design either internally or throw it open to external users. Examples of summmative testing, which will generally be based on design and build specifications, involve ticking off each of the functions and features to make sure everything works as per the design intention.
Formative or Exploratory study
A formative study is done early in the product development cycle, when a product is still in the initial phase of the product design process, where it is being defined and designed.
The main objective is to examine the effectiveness of the preliminary design concept. Usually, paper prototypes or low fidelity prototypes are used in this phase.
The formative study allows designers to study the assumptions of the users regarding the design (user-oriented approach) and also high-level issues such as the Organisational pattern, navigation etc, findability etc (product or system oriented).
In the formative study, you probe into why the user performed a certain action on the interface. The salient feature of formative study lies in discussion and examination of high-level concepts and the thought process of the users to inform the final design.
Assessment or Summative Test
Assessment or summative test is conducted early or midway into the product development cycle.
The main purpose is to expand on the findings of the exploratory test by evaluating the usability of lower level operations and aspect of the product.
Once the basic conceptual model of the product is solid, formative test seeks to examine and evaluate how effective the concept has been implemented, rather than just exploring the intuitiveness of the product.
