24 Examples of Design Testing

John Spacey, July 10, 2022

Design testing is the process of validating or optimizing a design. This can be pursued at the level of a product, design element or design concept. The following are common methods of design testing with examples.

A/B Testing

A/B testing involves preparing two designs and comparing measurable results such as conversions or click-through.

Benchmarking

Benchmarking is the process of comparing a design to a known best result. For example, comparing the conversion rate of a landing page to the best conversion rate that you have sustained in production. This is a type of A/B testing whereby B is the benchmark.

Multivariate Testing

Testing multiple variables at the same time as opposed to simple A/B testing. For example, asking users to score design alternatives in 22 ways.

Surveys

Collecting multiple variables by asking customers to fill out a survey after using a design.

Interaction Data

Allowing users to use a design so that you can collect multiple variables such as user timings and flow. This can be analyzed using analytics and similar tools.

Parallel Testing

Testing multiple variations of the same design at the same time. For example, testing 22 colors of a product to gauge customer reactions.

Massively Parallel Testing

Testing a large number of variations. For example, testing thousands of link colors in production using millions of users.

Field Testing

Testing a design in the real world. For example, testing a new snowboard design on a ski hill.

Throwaway Prototype

Building a quick, cheap and safe prototype to get a design in front of users or into the field.

Evolutionary Prototype

Building an expensive and refined prototype that captures your vision for a state of the art design. For example, a prototype sports car that has cost many millions of dollars to build and refine.

Proof of Concept

A test to prove that a design is feasible. For example, building a prototype of a computer hardware design to determine if it would overheat under certain conditions.

Design Experiments

Conducting controlled experiments to test a design. For example, subjecting a chair that uses an unconventional material to stress testing to determine its durability.

Open-ended Interview

Interviewing users to gauge their feelings about a design with an open-ended question such as "how did you feel about the product?" This is a preferred interviewing method that allows the customer to tell you what they really think. Difficult to capture as data.

Ladder Interview

A ladder interview starts with an open-ended question and goes to great depth with followup questions based on responses. This can produce much insight but is difficult to analyze because each interview is completely unique.

Close-ended Interview

Asking survey type questions that have few options in an interview. For example, "how would you rate the product color on a scale of 1-5?" This allows for precise data capture but isn't nearly as insightful or useful as an open-ended interview.

Recall Interview

Seeing how much users can remember about a design after using it. Indicates things like brand awareness and product positioning whereby you want a design to stand out and be memorable.

Focus Group

Interviewing users as a group to discuss a design. Tends to suffer from the abilene paradox whereby people adjust what they say to conform to the group such that results are less candid.

User Observation

Watching users as they use your products or services. For example, giving people a smart watch and seeing if they can figure out how to use it intuitively without instruction.

Demographic Study

Performing tests for a cohort based on demographics. For example, an observational study that identifies differences in how different age groups use a remote control for an air conditioner.

Psychographic Study

Performing tests for a cohort based on how people think. For example, testing snowboard graphic designs with a cohort of consumers who like hip hop music.

Design Immersion

Having the design team use the design in real world conditions to prevent naive design. This may resemble empathy training. For example, designers of a wheelchair who use the wheelchair design everywhere they go for a week. This tends to be amazingly fruitful in improving a design.

Test & Iterate

The process of quickly and aggressively testing, changing and testing again.

Optimization Myopia

There is much danger in using data to optimize things whereby you may miss the big picture. For example, spending months changing the color of a product based on a/b testing but missing the fact that the product is completely unusable and unreliable.

Quality Control

Quality control is the process of confirming that an instance of a design conforms to requirements. For example, testing that a product performs its required functions.