Product Validation

How do you validate your problem?

How to learn about your target market

Who would use your software? What are their burning issues? 

Let’s look at different ways to learn about your user and their environment.

Contextual Inquiry

Contextual inquiry is the deepest, most valuable way to learn about your user and their environment. Laura Klein’s book, Build Better Products, says using this methodology will give you the most in-depth information about your user. That’s because in a contextual inquiry, you’re observing your user in context — in other words, in the environment your customer will be in when they’re using your final product. So if a teacher will be using your product in a classroom, you need to sit in that classroom while they’re teaching to better understand the dynamic of their environment. There are other names for contextual inquiry (a day in the life or a fly on the wall), but whatever you call it, the result is the same: Your goal is to understand the customer’s needs and environment. 

Klein notes that you can do this kind of learning and discovery at any point in the development process, but it’s most effective when done at the beginning of the validation process because that’s when you’re still trying to understand your market.

User interviews

With user interviews, the goal is to learn about both your users and the problem you want to solve. What motivates your users? Is the pain point you’re trying to solve a legitimate, real world problem they have? How do they solve this problem right now? Why is it a big deal?

During user interviews, ask participants to describe scenarios where they bumped up against the problem and when they expect to encounter it again. How often does the problem surface? You’ll get a sense of the participant’s feelings and frustrations as well as the size and scope of the problem.

It’s important to remember that user interviews are part of the discovery phase of verification and validation. That means you’re not asking users what they think of your product or what features they want to see in your product. You’re not looking for that kind of direct feedback. Remember to focus on discovery-related questions, such as “Tell me about a time you had to do [task]?” rather than questions that lead users to the conclusion you’re looking for, such as “How difficult is it for you to do [task]?” Sure, they might give you the answer you want, but as Klein says in Build Better Products, it’s easier than ever to build a new product. What you want is to develop a great product that people actually want to use.

Still not sure how to get the most out of your customer interviews? Former Fluxible, uxWaterloo, and ProductTank Waterloo speaker Steve Portigal’s talk on uncovering insights or his book about how to conduct great customer interviews, called appropriately enough, Interviewing Users: How to Uncover Compelling Insights, can help. The book offers a framework for interviewing users and a detailed plan for how to prepare for the interview itself.

Are you building the right product?

Does your product solve your target market's needs? How do you validate your solution?

Having a business idea is one thing. Testing your business idea to make sure it’s offering value before you invest critical time and money is another. Validating your solution through small experiments adds another layer of learning. Do it early enough so that whatever you learn can make it into the next iteration of your product — you don’t want all that learning to go to waste. 

Validating a solution is all about sharing your idea with the world so you don't waste time or money. Release something, run a small experiment, and pay attention to the feedback. “A good rule of thumb is that you release before you feel comfortable and make sure you’re prepared to learn,” writes product management leader John Cutler in a post detailing tips for experiments and MVPs (minimum viable products).

Build something to learn, not to succeed. It may sound strange, but if there isn’t a risk that your MVP will fail, you’re probably not going to learn what you need to from your latest experiment. This is when you want to return to your assumptions map and think about your riskiest ones. Before you launch a product is the time to run small experiments that actually give you insight. That’s how you remove risk before bringing a product to market. That’s how you better understand user needs and product-market fit. You don’t want to walk into a dark room without knowing the layout — that could be painful. Small experiments help you understand where the light switches are in the room, turn them on, and illuminate the space so you can see what you might bump into.

When you’re testing your product, be clear about what you’re trying to achieve. What are you measuring? How will you take that learning and adapt? If you’re not clear about how you plan to harness the power of your feedback loop you’ll have a hard time getting the buy in you need to move forward. 

What is the desired outcome of the feature? What’s the intention? Is it to stop churn? Increase revenue? What are the key metrics and how do you expect them to change? Answering those questions helps you build a discussion around the return on investment adding that new feature provides and helps you judge whether it’s a sound investment. If it will cost you $250,000 to boost revenue by $1 million, then it’s worth it. But if it’s the other way around? Probably not. 

You need to know what game you’re playing and how you’ll keep score so you don’t just engage in the theatre of pushing out a product or feature without thinking through its intention and how it’ll help move you forward. Taking this approach gets everyone aligned on what goals and metrics matter. And the only way to make headway on those goals is to ruthlessly scrutinize the proposed problems to solve and feature requests.

Some call this the North Star metric: a single, specific metric that represents your product strategy. These should be real metrics. And by real we mean measurable and attributed to the features that you’re releasing. An easy trap to fall into here is to forget about timeframes. While revenue, retention and conversion are both admirable and valuable goals, you need to make sure you’re also including some leading indicators that signal you are on the right — or wrong — track as soon as possible.

Adopting a product rigour will put you miles ahead of most product organizations and product managers. Your entire company will be much happier for it and you’ll be happy that everyone can make better decisions based on the product context you’ve clearly established.

The best ways to test your product or feature

When you do test your product, service or business, think beyond landing pages and a/b tests. Sure those will give you helpful feedback, but there are no shortage of experiments you can run that offer insight and help you refine your product. Don’t just rely on surveys, customer interviews or landing pages. Bland and Osterwalder list 44 experiments that you can use as templates — from simple to more complicated depending on where you are in the validation process — in Testing Business Ideas. 

Running experiments gives you insight into what your customers value in an experience. But those experiments don’t have to be complicated. Here are three right from Testing Business Ideas:

Feature Stub:

An inexpensive way to test an upcoming feature, this experiment works best to quickly test a new feature on an existing product because you’re asking customers to click on a button and then tracking the number of clicks to gauge interest. And you can build on that by adding a call to action as well, such as asking customers to apply for a beta program, to better gauge their commitment. Signing up for a beta program signals that customers are more than just curious. This gives you a chance to interview potential users and further test your product because you’ve found someone who is actually looking for a solution. Ask them what their solution looks like or if you can test yours using methods in the Are you building the right thing? section. You may find that it’s not what you expected, but it’s better to know before you build it.

Clickable Prototype:

A clickable prototype simulates how your software product will work, without the cost of a full build. Customers click on screens and interact with the prototype as if it were a fully functioning product. The benefit of the clickable prototype is it helps you quickly test the concept of your product with customers. But be warned: even though you can test a prototype and it provides valuable evidence, prototyping is not a replacement for usability testing.  

Wizard of Oz:

The customer feels like they’re dealing with the real thing but they’re really not because everything is happening manually behind the scenes. Keeping the experiment small and simple is what makes it cheap to run, but asking customers about their satisfaction level afterward can provide strong evidence.

Again, running these experiments is all about paying attention to the feedback and adjusting your product priorities based on what you learn from users. So don’t experiment alone. Different people see things in different ways. You need that diversity to help you illuminate your blindspots. 

In some cases, particularly in heavily roadmap driven organizations, feedback is just ignored. And while that’s an option, it’s obviously not a great one since feedback should help inform your priorities for your product. The challenge is to get everyone on the same page about the importance, effort, and impact of the learning coming to light through your feedback loops. One way to do this is using a technique Zeitspace partner Jeff Fedor has found successful in the past: Use a simple Google Sheet with columns that capture a brief description of the issue, the source of the feedback, and the impact.

It’s important that when you run a few experiments you don’t get stuck in the learning cycle.

You don’t want to make the mistake of just setting aside time for testing. Make space to run experiments but also make space to analyze the results and act. Test, learn, act, then test again. Whatever you learn from one experiment should make it into the next. Learning has an expiration date. 

But let’s be honest: The stumbling block isn’t figuring out which experiment to use to test your product, or how to learn, iterate, and learn some more. It’s the actual testing itself that’s the big hurdle. 

It can be hard and scary to test out your business idea. What if someone doesn’t love your baby? But you know what’s worse? Making a $1-million bet that everyone will love your product without testing your idea and later finding out it’s not solving a critical problem, or your customer segment isn’t large enough to make your business profitable. 

“You can spend a lot of money building something and release it without solving a real need at all and flop,” says Bland. “Or you can just run a series of experiments along the way and understand if you’re on the right track before you place a big bet like that.”

Bland has seen that happen too often, wasting the time and talent of good people. 

So what if you run an experiment and find that it didn’t validate your idea? Well, that’s still valuable learning. It’s time and money saved not focusing on an aspect of your product that isn’t viable anyway.

Validate the market’s willingness to pay

You know your product fulfills a market and customer needs, now will people buy it? What price should you charge?

Now that you’ve validated you have a solution to a market that’s large enough, you need to know if what your market will pay for your product or service is enough to keep you profitable. Just like you would test your product or market, you also need to test your price. There are a few ways to test your pricing strategy and the market’s willingness to pay. 

First, don’t base the market’s willingness to pay by what customers say they’ll pay for your product. What someone says they’ll pay and what they are actually willing to dole out are sometimes not the same number. So base whatever you learn about your market’s willingness to pay from actual paying customers, says consultant Lincoln Murphy, who helps companies grow.

Remember at the start when we talked about getting to know your customer and how big of a problem your solution solves? This is when you’ll want to think about what you learned from customer interviews because understanding your customer and the value of your solution to their problem are what you’ll use to underpin your pricing strategy. 

Where do you start with pricing? Come up with an initial price to start your testing and then incrementally increase it until you’re getting about 20 per cent pushback. That’s one way to know you’re in the pricing sweet spot: Some will think you’re charging too much, but most will see your product or service offers value so they’ll pay it. Then you can go back to look at your business model: Is it enough to keep you profitable? 

Similarly, Ash Maurya, who created the Lean Canvas and wrote Running Lean, says you can’t really set an initial price without first deeply understanding your customer or their problem. So along with all that research you did about your customers, their problems, and the size of your market, Maurya suggests also offering a small group of customers your product for a price they can’t refuse so that you can get to know them even better. That price isn’t low, either. It should be higher than what they thought they’d pay but still competitive. That’s because you don’t want to work with a large group of customers right now. You’re trying to learn about pricing strategy, not scaling your product. Think of it as a velvet rope: You’re offering to work with them closely to use your product and understand whether it's validating your value proposition. If it doesn’t, they get their money back. If it does, you’ve just validated your value proposition, your price, and the market’s willingness to pay.

Validation builds resilience in your business 

Taking the time to validate your business idea means when you go to market with your product, you’ll do it with the confidence that customers will pay for your product or service. Going through each step may seem like a lot of work but it’s how you build resilience in your business. 

You’re not just testing your product, you’re evaluating its viability and how it will scale when you’re ready. Because who wants to launch a product or service only to find out it’s not profitable or that it can’t scale? Don’t waste your time, energy, talent, or money on something you’re not confident will succeed. Run small experiments, learn something from each one, and apply it to make your product its best.