Written by Kene Anoliefo
|
May 5, 2024
Learn how to use A/B testing and Beta Releases to validate whether a new product or feature is helping your customers achieve their goals.
In Part 3 we reviewed how to do Usability Testing with a prototype to validate the user experience before building your product. After building out your product it’s time to release it to real customers and validate whether or not it helps them achieve the outcomes they care about. This means validating that when people use it in the wild, it actually helps them complete and accomplish the tasks they set out to do in a way that leaves them satisfied and wanting to use it again.
If you’re testing a small change or isolated feature in your product, a great way to validate an outcome is through A/B testing. A/B testing allows you to compare the performance of a new feature to your current experience, which acts as the baseline. In the test you’ll pick a metric to measure, which is often a core KPI of your business like a conversion rate or completion of a funnel. If you use conversion rate as a the metric, the A/B test will measure how customers who have the new experience convert compared to the baseline version of the product. If you see a statistically significant lift in the conversion rate, then you know that your new experience has had a positive impact on your performance.
A/B tests are best used when you have a sizable user base and you can create a test cells where single changes can be isolated and compared to baseline. In our example from Part 3 of a savings and investing app, the team noticed that people were dropping off in your onboarding flow when asked to link their bank account. Through usability testing, they learned that this was because many users were unsure of the app’s privacy and security policy. If they decided to add a tooltip explaining their security policy to the “link bank account flow” they might run an A/B test to observe whether users who saw this information went on to link their bank account and complete onboarding more often than people who didn’t.
A/B tests provide objective measures of how iterations in your product experience affect your metrics. But you can’t always A/B test — maybe because you don’t have enough users or maybe because you have a more extensive “bundle” of features to test and can’t create a clean test that isolates one at a time.
Instead of doing an A/B test, you can do an alpha or beta release by giving a small set of users access to the product to have them use it and give feedback.
A good alpha or beta release:
Within the beta, you can collect feedback from users at different milestones like after their first time using it and then 30 days post sign up. For each milestone, you want to understand if and how the user has been able to get value from the product. If they have, what has been most valuable? If they haven’t, what’s preventing them from using it? And most importantly, you want to dig deep to uncover misalignment between how you thought people would use it before they released it to how they’re actually using it.
You can use both qualitative feedback and quantitative metrics to evaluate your alpha or beta release. In the case of our financial app, the team uses week-over-week retention as a core metric because they find that customers who use the product every week are likely to upgrade and subscribe to the paid version. In addition to qualitative feedback on product satisfaction, they also set a goal to have 30% week over week retention.
Using our earlier example of the savings and investing calendar app, an interview with great signal might sound like the following:
You: So can you tell me what made you decide to start using the product?
Customer: I saw someone share it in a Discord server I’m in and I immediately wanted to try it out. The past year has been rough because I’m in charge of my money for the first time and I got into a bit of debt from overspending. So I was very motivated to find something to help me fix that.
You: What were you using before?
Customer: A bunch of things inconsistently. I used a spreadsheet but that was hard to maintain. So then I would just check my account every once in a while but I couldn’t keep track of everything. I also tried out a few apps but nothing has stuck.
You: Walk me through how you’ve used the product in your first month.
Customer: Well as soon as I got access I went through and linked all my accounts and entered in my goals. And it was great — it gave me all the information for what I could invest that week and when. And then it sent me a notification every Sunday to review my plan for the next week, which I find really useful. I plan to keep using it this way.
You: That’s great. How would you describe what value the product is bringing?
Customer: It makes me feel confident about my finances and stay on track so that I can reach my goals.
You: Has the app replaced the other things you were using before?
Customer: Oh yeah. It has everything I need. I’ll never use a spreadsheet again for this.
From this interview we know:
An interview with poor signal might sound like this:
You: So can you tell me what made you decide to start using the product?
Customer: I saw someone share it in a Discord server I’m in and I thought I would try it out because I’m always interested in trying new apps.
You: What were you using before to manage your finances?
Customer: Different things on and off. Mostly just check my bank account every week.
You: Walk me through how you’ve used the product in your first month.
Customer: I signed up and linked my accounts — that was super easy. And then I saw that it recommended some savings and investing goals but I didn’t feel like I had enough information to follow through with them.
You: Why is that?
Customer: Well I felt like I had just signed up for this new app and it didn’t know much about me, so why would I trust their investment advice? Before I invest in anything I definitely need to talk to a person; I don’t have enough money to just throw it away on something I don’t understand.
You: When do you plan on using this next?
Customer: Ummm…I’ll probably check it out sometime again in the next few weeks now that you reminded me of it!
From this interview we know:
If you aren’t hitting your metrics and you get poor signal from your qualitative interviews, it could be a signal that you need to take a step back to evaluate whether your product is a viable solution.
If you’re getting great feedback from your beta release — congratulations! You’re ready to continue to release your product and improve it using both qualitative feedback and quantitative metrics.
As you build new features into your product, you will often find yourself revisiting different phases of the validation lifecycle:
Get the best content on user research, and product building delivered to your inbox.
Get notified when we publish the next installment of Off The Record