The Case for Testing in Accepting Payments

There's a laundry list of payment methods out there, and PMs have to ruthlessly prioritize which payment methods to integrate first. Yet, testing is rare in payments. So why is that?

Here's what we cover

Introduction (or why testing doesn't happen)

Testing is notoriously difficult in payments because of the high cost of integrating new payment methods. Adding new payment methods can take months of planning and engineering work. When the cost is so high, integrating a new payment method to test it doesn't make sense.

It's not like there's reliable third party data that product managers (PMs) can rely on to make data-driven decision. Most of it is either biased (towards the company that produced) it or incomplete.

While difficult, testing is still what PMs should aim to do. Here are a few strategies that PMs can use, depending on their payment stack.

Painted door tests

The painted door test is great for businesses that manage payment integrations in-house. Learn more about painted door tests.

You can use this to showcase a new payment method on your checkout page that isn't fully functional yet. Users who click on it are directed to a page or message saying that the payment method will be available soon.

Yes, it may not be a great experience for the customer. But it gives you important baseline data on how popular a payment method might be with very little development time.

This is particularly helpful to guiding an expansion strategy, and prioritising a long list of local payment methods. By creating different painted doors for each market (for example, Alipay for China) you can get an estimate of the impact to conversion rate and revenue.

Closed door test

In some cases it may be worth testing if your checkout page performs better with a payment method removed, i.e. a closed door test.

Close a few doors and see how this impacts volume and conversion rate:

• Remove the payment method with the lowest volume
• Remove payment methods with a low conversion rate

User surveys

User surveys are great when technical resources are tight, and you need more data before making a decision. This could be instances where you're:

• Expanding into a new market and trying to understand consumer payment preferences
• Building your payments roadmap and rank-ordering which payment methods to prioritize

Pro-tip: add a survey to the redirect page of your painted door test

Running A/B tests

You should A/B test wherever you can.

Some ideas of where to run them:

• When running a painted door test (or closed door test), split your audience across the two versions of the checkout, one with the painted door and one without
• When choosing between two, similar payment methods (Klarna vs. Affirm or Apple Pay vs. Google Pay) run two versions of a painted door test, one with each payment method

What's the experimentation tool for payment product managers?

Marketers and PMs use tools like Optimizely or Google Optimize as their testing infrastructure to efficiently run experiments and A/B test new features or tool. So what should payment product managers use?

Ideally, they want a tool that can help them spin up and turn off payment methods easily and with inbuilt reporting connected to it. You could build this in-house, but my recommendation would be to either use a payment service provider (PSP) or payment orchestration layer.

If you're going via the PSP route, look PSPs like Stripe and Adyen offer A/B testing functionality out of the box, and also make integrating new payment methods easier. See Stripe's resources on A/B testing.

Or you can consider orchestration platforms like Spreedly and Primer (full disclosure, I worked at Primer.io) which have an added benefit of allowing you to easily spin up new PSPs in addition to alternative payment methods. So you can A/B test not just Klarna vs. Affirm, but also Stripe vs. Adyen. This gives you serious firepower in line with what you might see in Amazon, Uber, or Netflix's payment teams.

Conclusion

Testing can be done in payment, no matter your payment stack. When you can spend months integrating a new payment method that drives no impact, it's better to run a test first.

Feel free to check out the other growth / payments related content on my blog 👇🏽

The Growth Story Behind Buy Now Pay Later

The Growth Marketing Approach to Scaling Partner Programs

Do you have any other ideas on how to test in payments? Feel free to hit me up on LinkedIn or Twitter. 👋

You can also follow me or hit subscribe to keep in touch. I write regularly about growth, marketing, and partnerships in tech and fintech.