Valueships

When it comes to validating product ideas in the Software as a Service industry, A/B testing is the most common go-to method. It has become so embedded into startup DNA that any product optimizations, tests or experiments are directly associated with A/B testing.

Mainly because it's straightforward to do, and it doesn't require much effort or analysis. All you need to do is come up with two different price versions and split your visitors' flow into them equally. Whichever gets better conversions is the winner, and as such, should remain on your website, pulling in the best customers in the tested segment.

Not.

A/B testing, as much as it works great for comparing landing pages or CTA (Call to Action) buttons, is not suitable for testing the pricing. And by "not good", I mean terrible to your business, which I will break down in detail below.

In this article, you will learn:

Why you shouldn't A/B test your price? Period.

‍

It might be illegal

According to the EU's law, price discrimination is illegal when you charge customers differently just because they are of another nationality. The product's cost might be different, for example, due to other delivery costs or TAX rates, but you're not allowed to differ the price based on one's country of residence.

For example: if you're A/B testing the pricing of your project management product, and someone from France purchases it at a higher cost than a customer from Belgium, you might be fined if the one that paid more made a complaint to the European Consumer Centre.

Yes, you're conducting a random test, and such occurrences are not of your will, and yes, the chances that someone will spot the issue and then make an official complaint are low. But they're still here, which leaves the A/B testing in a grey area of legality.

‍

Your traffic is not enough, most likely

Statistical Significance is a mathematical term that describes the reliability of a statistic. If the results of A/B tests are statistically significant, they represent an actual correlation between tested variations and their conversion rates. In other words: they're highly likely to be true.

However, for your split testing to be statistically significant, you need to conduct it on a particular group of people. The bigger the sample size you can get, the higher the significance.

Imagine you have monthly web traffic of 1000 and your conversion rate for purchasing a subscription is 1%. Your sample size of 10 users won't give you any conclusive information at all. In this article by Michal Fiech you will find instructions on calculating a minimum size sample to make your test results trustworthy.

And that's not even it. Most probably, you want to perform test across different buyer personas. Not only you need to identify them, but divide them into other test groups too. This leads to an even bigger required sample size, and most of the companies I speak with simply don't have it.

A/B price testing is suitable for websites with massive traffic and decent conversion rates. If you're not in this group, it will give you false information.

Do you need more practical insights?

More about pricing

Do you need more practical insights?

Learn more about pricing

You can turn off your potential and existing customers

Due to the randomness of your traffic, the A/B test can lead to problematic customer service scenarios. If a sales specialist who got caught by your 10€ pricing is encouraging his or her boss to purchase software, that to this person shows for 20€, you will most likely anchor them in the lower pricing point. It will be a nightmare converting them to higher pricing if it turns out to be the optimal one.

Now imagine another company where neither of these 2 was lucky enough to land on the cheaper plan. They became regular users, and through word of mouth, they figured out that they could've paid two times less. I'm sure you wouldn't like to be in their situation, and most likely, you'd raise questions. The more significant the price difference between plans, the harsher the questions will be.

To avoid any bad customer experiences, you need to be sure that they use only one device to browse your website, don't revisit it, be the only employees visiting the pricing plan, and never discuss software prices with their colleagues or anyone else. If you can't be sure about even one of the above, then A/B tests will likely expose you to frictions in the sales process.

‍

You're overcomplicating things

If your pricing strategy consists of 3 different plans, then the A/B test will double the number of buying possibilities and create a total of 6 segments. On the marketing level, you're making a different analytical thing to cover. From a sales perspective, you're splitting your pricing policy, which is yet another thing to take care of by the sales team (just like discount policy).

Cohort analysis is already complicated, and by multiplying it, you make it even harder. With A/B testing, you need to deal with two pricing plans that will affect your day to day sales and marketing efforts.

‍

You're limiting your business potential

The ultimate goal of A/B testing is to discover which of the two variables performs better. You're revealing if the first one is more successful than the other in A and B's duel.

This, by any means, doesn't indicate that the winner is an optimal setup for your business. You can understand if A is better than B, but you don't know if A is good at all. You're limiting your global vision to nothing but a price point, which is just a part of complex pricing strategies.

The perfect pricing point is found through surveying your customers and gathering their feedback. You need to understand what features do they value most and which they want to get rid of. And ultimately, what is their Willingness to Pay for the product.

If you limit your pricing strategy to nothing but A/B testing, you're overlooking multiple factors that are significant in setting up an optimal pricing strategy. Or, as Vilgefortz, one of the sorcerers from The Witcher's world, said: you mistake stars reflected in a pond for the night sky.

‍

What we have learnt

A/B testing is an excellent tool for comparing conversion rates of website visuals, such as CTA buttons or different web designs. It's suitable for landing pages where significant amounts of people convert in short periods.

If your business is not in this stage yet, the A/B test won't provide you with valuable conversion rate optimization insights. On the contrary, it will expose you to sales, marketing, customer service and PR challenges.

To avoid those risks and to start working towards a suitable pricing strategy, I encourage you to read this article first. And in case of any questions, don't hesitate to reach out to us via the contact form below.

‍