How do I validate whether or not the product or feature I just shipped is working for my customers?

In Part 3 we reviewed how to do Usability Testing with a prototype to validate the user experience before building your product. After building out your product it’s time to release it to real customers and validate whether or not it helps them achieve the outcomes they care about. This means validating that when people use it in the wild, it actually helps them complete and accomplish the tasks they set out to do in a way that leaves them satisfied and wanting to use it again.

Part 3: Validating Outcomes

Validating Outcomes Using A/B Tests

If you’re testing a small change or isolated feature in your product, a great way to validate an outcome is through A/B testing. A/B testing allows you to compare the performance of a new feature to your current experience, which acts as the baseline. In the test you’ll pick a metric to measure, which is often a core KPI of your business like a conversion rate or completion of a funnel. If you use conversion rate as a the metric, the A/B test will measure how customers who have the new experience convert compared to the baseline version of the product. If you see a statistically significant lift in the conversion rate, then you know that your new experience has had a positive impact on your performance.

A/B tests are best used when you have a sizable user base and you can create a test cells where single changes can be isolated and compared to baseline. In our example from Part 3 of a savings and investing app, the team noticed that people were dropping off in your onboarding flow when asked to link their bank account. Through usability testing, they learned that this was because many users were unsure of the app’s privacy and security policy. If they decided to add a tooltip explaining their security policy to the “link bank account flow” they might run an A/B test to observe whether users who saw this information went on to link their bank account and complete onboarding more often than people who didn’t.

Validating Outcomes with User Interviews

A/B tests provide objective measures of how iterations in your product experience affect your metrics. But you can’t always A/B test — maybe because you don’t have enough users or maybe because you have a more extensive “bundle” of features to test and can’t create a clean test that isolates one at a time.

Instead of doing an A/B test, you can do an alpha or beta release by giving a small set of users access to the product to have them use it and give feedback.

A good alpha or beta release:

• Has at minimum 10s and ideally 100s of users.
• Enables them to use the product in the real context they would use it in real life.
• Gives users access to the product for long enough to get value from it so that they can give you useful feedback. In the case of our monthly spending calendar, it’s not enough to give the product to users for a few days because the value comes from tracking expenses throughout the month. It would be much better to do a 2-3 month beta to track continued retention and usage over time.

‍

Within the beta, you can collect feedback from users at different milestones like after their first time using it and then 30 days post sign up. For each milestone, you want to understand if and how the user has been able to get value from the product. If they have, what has been most valuable? If they haven’t, what’s preventing them from using it? And most importantly, you want to dig deep to uncover misalignment between how you thought people would use it before they released it to how they’re actually using it.

You can use both qualitative feedback and quantitative metrics to evaluate your alpha or beta release. In the case of our financial app, the team uses week-over-week retention as a core metric because they find that customers who use the product every week are likely to upgrade and subscribe to the paid version. In addition to qualitative feedback on product satisfaction, they also set a goal to have 30% week over week retention.

Goals

• Validate that your product drives the outcomes that are important to your customers.
• Uncover other valuable feedback about the experience that can be used to improve the experience.

Tactics

• Ideal: A/B Test or Beta Release to at least 100 users + qualitative interviews with 15-20 users.
• Quick & Dirty: Small Beta release with a group of trusted customers (10-20) and qual feedback with 5-7 customers at key milestones in the beta (after signup, after first use, 30d after sign up).

Setup

• Release the feature or product to customer so that they can use it within their normal context or workflow.
• Choose a quantitative metric that demonstrates that users are getting value from your product. Make sure you have adequate tracking in place to measure and report on this metric.
• Run qualitative interviews with a subset of users to understand their experience and collect feedback.

Interview Discussion Guide

1. Product Expectations: Ask them to describe why they decided to use the product and what they were hoping to achieve.
2. Use Case Exploration: Ask for a detailed set of steps for how they used the product within their workflow / life.
3. Challenges & Pain Points: Have them detail the challenges they experienced using the product and how they overcame them.
4. Product “Completion”: Discuss any features they expected to have but were missing, and any additional ones they would add to make it more useful. Ask them if this product can fully replace what they were using before to solve this problem, or whether it needs additional functionality before they can switch.
5. Continued Usage: Talk to them about how and when they plan on using it next. It’s never reliable to ask people if they’ll use it again, but it can be good signal if they’ve already made a concrete plan for when they plan to use it in the future.

What signals should I look for in my interviews?

• You meet or exceed your quantitative goal. Give yourself props — you did that!
• The majority of users say that they can fully switch to using your product now for their needs. If there are gaps, they can clearly tell you what they would need to make the switch.
• They can clearly identify when they’ll use it again. An even better signal is that they are willing to pay for the product beyond the trial period.

Examples of interviews with good signal and bad signal

Good Signal

Using our earlier example of the savings and investing calendar app, an interview with great signal might sound like the following:

You: So can you tell me what made you decide to start using the product?

Customer: I saw someone share it in a Discord server I’m in and I immediately wanted to try it out. The past year has been rough because I’m in charge of my money for the first time and I got into a bit of debt from overspending. So I was very motivated to find something to help me fix that.

You: What were you using before?

Customer: A bunch of things inconsistently. I used a spreadsheet but that was hard to maintain. So then I would just check my account every once in a while but I couldn’t keep track of everything. I also tried out a few apps but nothing has stuck.

You: Walk me through how you’ve used the product in your first month.

Customer: Well as soon as I got access I went through and linked all my accounts and entered in my goals. And it was great — it gave me all the information for what I could invest that week and when. And then it sent me a notification every Sunday to review my plan for the next week, which I find really useful. I plan to keep using it this way.

You: That’s great. How would you describe what value the product is bringing?

Customer: It makes me feel confident about my finances and stay on track so that I can reach my goals.

You: Has the app replaced the other things you were using before?

Customer: Oh yeah. It has everything I need. I’ll never use a spreadsheet again for this.

From this interview we know:

1. The user was motivated to use the app based on a real problem in their life, not just because it seemed interesting or cool.
2. They’ve gotten a lot of value from it in the first month. They have already built a routine around the app.
3. They can confidently say the app is much better than the alternative, and plan to keep using it.

Poor Signal

An interview with poor signal might sound like this:

You: So can you tell me what made you decide to start using the product?

Customer: I saw someone share it in a Discord server I’m in and I thought I would try it out because I’m always interested in trying new apps.

You: What were you using before to manage your finances?

Customer: Different things on and off. Mostly just check my bank account every week.

You: Walk me through how you’ve used the product in your first month.

Customer: I signed up and linked my accounts — that was super easy. And then I saw that it recommended some savings and investing goals but I didn’t feel like I had enough information to follow through with them.

You: Why is that?

Customer: Well I felt like I had just signed up for this new app and it didn’t know much about me, so why would I trust their investment advice? Before I invest in anything I definitely need to talk to a person; I don’t have enough money to just throw it away on something I don’t understand.

You: When do you plan on using this next?

Customer: Ummm…I’ll probably check it out sometime again in the next few weeks now that you reminded me of it!

From this interview we know:

1. This customer used the app because they like trying new products, not because they had a strong need for it. This might mean in the future you might want to do more targeted marketing to bring in eligible users, or look more deeply into whether or not your core audience has a strong need for your product if poor signal persists.
2. This customer’s needs around investing seem to be fundamentally incompatible with your app; they want to talk to a person before investing instead of having an algorithm make recommendations. Is this because you need to create more trust in your recommendations, or because your customers will only make investing decisions through humans? Or something else? Go back and do more interviews about your problem space and the solution you chose to better diagnose what is not resonating.
3. They used the product, but it didn’t fully solve their need. They can’t fully replace what they were using previously.
4. They say they’ll probably use it again but can’t describe a specific time or trigger.

If you aren’t hitting your metrics and you get poor signal from your qualitative interviews, it could be a signal that you need to take a step back to evaluate whether your product is a viable solution.

Next Up: Continue to iterate on your product!

If you’re getting great feedback from your beta release — congratulations! You’re ready to continue to release your product and improve it using both qualitative feedback and quantitative metrics.

As you build new features into your product, you will often find yourself revisiting different phases of the validation lifecycle:

1. Validating Needs
2. Validating Concepts
3. Validating User Experiences
4. Validating Outcomes (this guide)
5. ‍
6. Continue to revisit these methods to ensure that you’re using customer feedback to help you make the best decisions possible about your product.