Valueships

Mainly because it's straightforward to do and doesn't require much effort or analysis. All you need to do is come up with two different price versions and split your visitors' flow into them equally. Whichever gets better conversions is the winner and as such, should remain on your website, pulling in the best customers in the tested segment.

Not.

A/B testing, as much as it works great for comparing landing pages or CTA (Call to Action) buttons, is not suitable for testing the pricing. And by "not good," I mean terrible to your business, which I will break down in detail below

It might be illegal

According to the EU's law, price discrimination is illegal when you charge customers differently just because they are of another nationality. The product's cost might be different, for example, due to other delivery costs or TAX rates, but you're not allowed to differ the price based on one's country of residence.

For example: if you're A/B testing the pricing of your project management product, and someone from France purchases it at a higher cost than a customer from Belgium, you might be fined if the one that paid more made a complaint to the European Consumer Centre.

Yes, you're conducting a random test, and such occurrences are not of your will, and yes, the chances that someone will spot the issue and then make an official complaint are low. But they're still here, which leaves the A/B testing in a grey area of legality.

Your traffic is not enough, most likely

Statistical Significance is a mathematical term that describes the reliability of a statistic. If the results of A/B tests are statistically significant, they represent an actual correlation between tested variations and their conversion rates.

In other words: they're highly likely to be true.

However, for your split testing to be statistically significant, you need to conduct it on a particular group of people. The bigger the sample size you can get, the higher the significance.

Imagine you have monthly web traffic of 1000, and your conversion rate for purchasing a subscription is 1%. Your sample size of 10 users won't give you any conclusive information at all. In this article by Michal Fiech, you will find instructions on calculating a minimum-size sample to make your test results trustworthy.

And that's not even it.

Most probably, you want to perform tests across different buyer personas. Not only do you need to identify them, but divide them into other test groups too. This leads to an even bigger required sample size, and most of the SaaS companies I speak with simply don't have it.

A/B price test is suitable for websites with massive traffic and decent conversion rates. If you're not in this group, it will give you false information.

Do you need more practical insights?

More about pricing

Do you need more practical insights?

Learn more about pricing

You can put off your potential and existing customers

Due to the randomness of your traffic, the A/B test can lead to problematic customer service scenarios. If a sales specialist who got caught by your 10€ pricing is encouraging his or her boss to purchase software that to this person shows for 20€, you will most likely anchor them in the lower pricing point. It will be a nightmare converting them to higher pricing if it turns out to be the optimal one.

Now imagine another company where neither of these 2 was lucky enough to land on the cheaper plan.

They became regular users, and through word of mouth, they figured out that they could've paid two times less. I'm sure you wouldn't like to be in their situation, and most likely, you'd raise questions. The more significant the price difference between plans, the harsher the questions will be.

To avoid any bad customer experiences, you need to be sure that they:

use only one device to browse your website,
don't revisit it,
are the only employees visiting the pricing plan,
and never discuss software prices with their colleagues or anyone else.

If you can't be sure about even one of the above, then A/B tests will likely expose you to friction in the sales process.

You're overcomplicating things

If your pricing testing methods consist of 3 different plans, then the A/B test will double the number of buying possibilities and create a total of 6 segments.

On the marketing level, you're making a different analytical thing to cover. From a sales perspective, you're splitting your pricing policy, which is yet another thing to take care of by the sales team (just like the discount policy).

Cohort analysis is already complicated, and by multiplying it, you make it even harder. With A/B testing, you need to deal with two pricing plans that will affect your day-to-day sales and marketing efforts.

You're limiting your business potential

The ultimate goal of A/B testing is to discover which of the two variables performs better. You're revealing if the first one is more successful than the other in A and B's duel.

This, by any means, doesn't indicate that the winner is an optimal setup for your business. You can understand if A is better than B, but you don't know if A is good at all. You're limiting your global vision to nothing but multiple price points, which are just a part of complex pricing plans.

The perfect pricing point is found through surveying your existing and new customers and gathering their feedback. You need to understand what features they perceived value most and which they want to get rid of. And ultimately, what is their Willingness to Pay for the product?

If you limit your pricing strategy to nothing but A/B testing, you're overlooking multiple factors that are significant in setting up an optimal pricing strategy.

Or, as Vilgefortz, one of the sorcerers from The Witcher's world, said: you mistake stars reflected in a pond for the night sky.⭐

What we have learned

A/B testing is an excellent tool for comparing conversion rates of website visuals, such as CTA buttons or different web designs. It's suitable for landing pages where significant amounts of people convert in short periods.

If your business is not in this stage yet, the A/B test won't provide you with valuable conversion rate optimization insights. On the contrary, it will expose you to sales, marketing, customer service, and PR challenges.

To avoid those risks and to start working towards a suitable pricing strategy, I encourage you to read this article first. And in case of any questions, don't hesitate to reach out to us via the contact form below.

FAQ

#1 Is A/B price testing illegal?

A/B price testing methods can be illegal if it involves price discrimination based on a customer's nationality. Charging different prices solely because of someone's country of residence can be against the law, such as in the EU where price discrimination is prohibited.

#2 Why does A/B testing require a significant amount of traffic?

A/B testing requires a large enough sample size to ensure statistical significance. To obtain reliable results, you need a sufficient number of visitors to your website or users of your product. Small sample sizes may not provide conclusive information.

#3 Can A/B price testing turn off potential customers?

Yes, A/B price testing can lead to customer dissatisfaction if they discover they could have paid a different, lower price. Inconsistent pricing experiences can create confusion and undermine customer trust in your brand.

#4 Does A/B testing overcomplicate pricing strategies?

A/B testing can complicate pricing strategies by increasing the number of price segments and creating additional complexity for marketing and sales teams. Managing multiple pricing plans can be challenging and may lead to confusion.

#5 Does A/B price testing limit business potential?

A/B price testing focuses only on comparing two variables and doesn't consider other important factors in pricing plan, such as customer feedback, willingness to pay, and feature preferences. It can restrict your understanding of optimal pricing.

#6 Is A/B testing suitable for all businesses?

A/B testing is not suitable for all businesses, especially those with low traffic or limited conversion rates. It is more effective for websites with significant traffic and higher conversion rates where meaningful insights can be obtained.

#7 Can A/B testing create customer service challenges?

A/B testing can lead to customer service challenges if customers discover different pricing options and feel they were not given the best price. Inconsistent pricing can result in negative experiences and difficulties in upselling.

#8 What are the risks of relying solely on A/B testing for pricing decisions?

Relying solely on A/B testing for pricing decisions overlooks various factors crucial for setting an optimal pricing plan. Important considerations such as market demand, cost-plus pricing, and consumer expectations may be disregarded.

#9 How does A/B testing impact marketing efforts?

A/B testing affects marketing efforts by creating additional analytical and strategic considerations. It can complicate cohort analysis, split pricing policies, and require separate sales and marketing approaches for different price segments.

#10 What alternatives should be considered instead of A/B price testing?

Instead of A/B price testing, businesses should consider surveying customers to gather feedback, understanding their preferences, and determining their willingness to pay. This comprehensive approach provides a more holistic view of the pricing plan.