This week we wrote a recommendation to a fictional company about whether they should conduct a formative or summative usability test.
I like this sentence to sum up the differences between the two types of tests: “A formative test reports on what is broken while a summative test examines what is working.”
As I said in the recommendation report, that’s a pretty broad statement. Another way of differentiating the two tests is to say a summative test is usually conducted at the end of a project (“summary”) while a formative test is usually conducted during the creation of the project (“formation”).
Formative tests can be very informal, using rough prototypes. They are trying to find “what is broken,” what the user expects of the product and where those expectations are not lining up with reality.
Summative tests are more formal, using a final or nearly-final prototype. You give users specific tasks to complete and see where they succeed or fail doing that (aka, “what is working”).
Ultimately I recommended doing a formative test, since the fictional situation had a limited timeframe and I thought the final product would have a lot of glitches no matter which way we went. A formative test would catch overall problems — the kinds that are more expensive and time-consuming to fix later. I thought that would give more value to the company.
I can see someone arguing for the opposite recommendation, however.
It made for a good thought experiment.
Note: This blog was written during Usability week 2.