How to Test Early and Often During the Entire Product Development Process

Here's the thing: the way we build products is broken. Let me explain that.

For the past twenty years or so, the startup world has moved from waterfall to agile development processes. Smaller deliverables replaced long projects as product teams transitioned to an incremental approach to delivering software.

With the rise of agile development practices and design thinking, more and more companies started adopting an iterative design and product-building mindset, releasing and learning from users more often.

However, that shift has happened incrementally and we’re still not learning as fast as we could be. We got better and quicker at shipping and learning after launch, but we still hadn't found a viable way to learn from users during the product and design processes.

Only the companies that afford the cost of research and see the value in bringing dedicated researchers onboard benefit from insights and access to users at scale. Even so, the typical ratio of researcher-to-designer-to-developer today is 1:5:50.

Most product teams rely on releasing and learning after development, which ultimately leads to product and design teams doing avoidable rework. It’s been estimated that the cost of fixing errors after launch can be as much as a hundred times higher than fixing them during the development stage.

As the CEO and co-founder of Maze, a rapid testing platform for product teams, I've been at the forefront of this challenge, making it our mission to enable anyone to test and learn rapidly during the design and product build cycle. We've purposely built a platform that allows anyone from designers to marketers to test any experience as part of their workflow to learn what works and what doesn't and make informed decisions faster.

Why continuous testing changes the way you design and build

For product and design teams, testing and learning are key steps to building great user experiences. When you test your work early in the process, you avoid spending huge amounts of time and resources on developing the wrong ideas and redoing your work later.

Moreover, when testing occurs cross-departmentally and everyone from PMs and designers to UXRs test their piece of the puzzle, the result is a validated process from beginning to end. Product managers only add ideas that have been researched and validated to the product roadmap, designers hand off thoroughly tested mockups, and marketers launch validated campaigns.

"The single biggest driver of business impact is the strength of an organization's learning culture." - Josh Bersin, President Bersin by Deloitte

Finally, the more you test and learn from users, the more you and your team acquire knowledge about what works and what doesn't. This institutional know-how about your users and the problem space is a key driver of your organization's success. And when that happens inside each department across the entire organization—that’s one of the biggest competitive advantages your company can have.

How to test early and often with the Rapid Testing Framework

Now that we looked at the importance and benefits of testing let's see how you and your product and design teams can implement it in your organization.

Here at Maze, we developed a Rapid Testing Framework that helps you run testing at every stage of the process from idea to launch. But first, let's define what we mean by rapid testing. Rapid testing is a decision-making framework in which every design and product decision is made with user input.

The Rapid Testing Framework [Image: Maze]

This framework approaches every decision as a testable input—concepts, designs, copy, wireframes—all of these are tested and validated with users during the product build cycle.

IOTA Loops support the Rapid Testing Framework at key stages of the process, from idea and concept validation to prototype and copy testing. The IOTA acronym stands for Input, Objective, Test, and Analysis. Let’s look at these stages in detail.

IOTA Loops [Image: Maze]

Input

In every IOTA loop, the testing process starts by defining an input—a testable hypothesis or question. Examples of inputs:

• Is the product feature we want to develop going to bring value to our users?
• What design concept should we move forward with?
• Is the copy clear and readable?

It’s important to define the right input from the outset, as this will define your decision. Take your time understanding the problem and what it is you need to test with users.

Objective

Next, set your Objective. The objective(s) is the result you’ll measure success by. Each test should have clear objectives, so you can easily determine whether your test has succeeded or not. In the case of the latter, you’ll need to refine and iterate on the solution until the objective is reached.

For example, if you’re testing a high-fidelity prototype at the latest stages of the design process, a good objective can be:

“We want more than 80% of test users to complete the tasks successfully.”

And another one:

“Our goal is for 85% of test participants to associate the new messaging with ___.”

Establishing clear objectives and goals at the start of the process is a core component of testing rapidly—when you know your goal, you know what to look for.

Test

Next, create your Test. This includes defining the right testing methodology from 5-second testing to prototype testing and more—the method you need depends on the input and objective you defined above.

At this stage, you can also create a test plan and define the questions you’ll ask users, the tasks, or any other details that you’ll need to take into account. Running the test becomes easier when you’ve planned, so make sure you prepare for it accordingly.

Analysis

Last but not least, the Analysis part includes assessing your results to determine the next steps. If the test results meet the objective set, then you and your team can make the decision and the next phase of the design process begins. If not, iterate on the solution until the results are met.

Making testing part of your design process

Making rapid testing a part of your design workflow will take some practice, but the most important thing is to get started somewhere. Like anything else, testing and research is a muscle that needs to be trained and built over time. Here are a few tips on testing during the design process:

　1. Embrace vulnerability

Most of the time, designers are reluctant to put their work out there. But, from sharing your work with colleagues to ask for feedback or running user testing—being vulnerable is a part of becoming a great designer. In fact, sharing what you’ve built with users or anyone else is a key step to becoming better at your job. You’ll get feedback on how you can improve your designs from the people who’ll use what you’re building—your users. And sometimes, getting an outside perspective on your work can help you find a new solution or approach the problem from a different angle. Embracing vulnerability is a central part of the design process and can only make you better at your job.

　2. Start testing early

Another mistake most designers make is waiting until they have a high-fidelity prototype to share with users. That’s related to the above point, as sharing work in progress is about being vulnerable.

But, testing at the earliest stages of the design process is critical, especially when you’re working based on assumptions on a new product or feature. Helen Tsvirinkal, Product Designer at Shopify, talks about the importance of early, formative testing:

“If a project is fixing a well-known problem, maybe testing early is less necessary. But if you’re working off assumptions, you need to test low-fidelity designs as soon as possible to find out what a user expects.”

You can start researching and testing before the design process even begins. From problem exploration and generative user interviews to idea and concept validation—start testing your key design decisions early in the process to avoid significant reworks down the line.

　3. Share the findings with your team

Building a testing muscle inside your organization won’t happen overnight. But you can build on it gradually if everyone is on the same page about the benefits of testing.

One way to showcase the value you and your team can get from testing more often is by sharing the findings publicly. Behzod Sirjani, former Head of Research & Analytics Operations at Slack and founder of Yet Another Studio, mentions why this is important:

“I’ve seen the most success with usability testing when designers, engineers, and product managers are involved in understanding how people actually use or don’t use a product. You want people who have the ability to make decisions to be involved, so you can make sure that all of your research is building a case for the important decisions that you want to make, whether that’s as high level as “do we release this product?” or as focused as “Is this a more usable menu?” - Behzod Sirjani

There are multiple ways to get people more involved in the process, including:

• Document your findings in a report and highlight key learnings
• Do a Loom video and run over the main findings
• Invite people to participate in interviews or debriefs
• Do a hands-on product team meeting to discuss results and next steps

For instance, the team at BlaBlaCar has a Slack channel for ongoing user interviews that everyone in the company can join and see feedback in real-time. Making testing part of your process is all about building awareness about the value user feedback provides and that starts by sharing any insights or learnings with your team.

Ultimately, the more testing you do and the more you communicate its importance and value, the easier it will be to get buy-in on testing in the future. 

ProtoPie helps designers bring their ideas to life in the form of high-fidelity interactive prototypes. Turn those design ideas into clarity, insights, and business gains. To learn more about how ProtoPie can help you get from concept to creation, explore the website or try it for free!

If you want to see Maze integrating with ProtoPie, let us know.