Test Your Way Out of the Pandemic Fog

Making good business decisions is a difficult process even in normal times. But these are not normal times. The world we live in is undergoing an unprecedented change. Making business decisions in this environment can be extremely difficult. Tools we relied upon to guide us in the past are giving us bad guidance.

To succeed, businesses will have to find a way through the shroud of this pandemic fog. Fortunately, Test & Learn processes can provide that visibility, enabling better decisions and outcomes.

Why We Need a New Approach

Many organizations have adopted predictive analytics tools and techniques such as AI or statistical analyses to help guide decision making. Central to these approaches is the notion that past is prologue. If we are able to detect a pattern of consumer behavior in the past, we can be confident that pattern will continue in the future – under the same circumstances. In times like these however, the past is no longer prologue.

Renaissance Technologies (RenTech) is a quantitative investment company that relies on the output of statistical models to make investment decisions. Until recently RenTech’s analytical approach had produced spectacular results, far outpacing traditional funds.  In fact, their flagship fund had averaged a 39% return after investor fees since 1988! Even in the rarefied air of quantitative investment firms, RenTech is considered the best.

However, in 2020 things changed.  The models that had been so accurate in the past stopped working.  RenTech’s international equities fund lost 19% in 2020, its institutional diversified alpha fund tumbled 32% over the same period, and its institutional diversified global equities fund slumped 31%.  More importantly, their clients have demanded the return o more than $5 billion of invested capital.

For B2C decision makers, how are you supposed to anticipate consumer behavior when buyer incentives, motivations, and resources look nothing like they did just 12 months ago? Previous consumer profiles and personas are no longer valid, as both sides of the retail market have drastically changed. Continuing to rely on past analytics may end up causing greater harm than good when trying to forecast market developments. It’s like using your navigation during a bad rainstorm. It may predict the fastest route over a bridge. But if that bridge is washed out, following that route can lead to disaster.

One approach to mitigate the risk of inaccurate predictions is to utilize Test & Learn.

What is Test & Learn?

Contrary to other predictive approaches, Test & Learn makes no assumption that we are able to predict future outcomes from past behavior. So instead of predicting results, we develop and test ideas. It’s the scientific method: study the facts; develop a hypothesis; test the hypothesis; learn and adapt.

When to use Test & Learn

There are myriad ways in which Test & Learn can help improve your business, from innovation to process optimization. Typically, the best candidates have the following characteristics:

  • Frequent – There needs to be a high process frequency to allow for a good sample size for testing.  Something that’s done once a month won’t work.
  • Measurable – There are clearly understood and measurable success metrics for the desired outcome(s) – if you’re not closely measuring your customer retention process for instance, you won’t know whether new tactics are working or not.
  • Testable – There is an ability to conduct tests without adversely affecting the business – some processes are too critical, such as monthly close; others may be regulated like securities offerings which would make it difficult or impossible to experiment.

Here are just a few of the ways that Test & Learn can be applied to your organization.

Demand Elasticity

One example of Test & Learn in business is examining demand elasticity, i.e. the impact that changes in price have on purchase behavior or demand. Understanding demand elasticity is critical in setting the price which maximizes total revenue. Too high and you’re driving buyers away.  Too low and you’re leaving money on the table.

Understanding demand elasticity is often attempted by analyzing historical data. But that technique is notoriously plagued by noisy data. Market conditions, advertising, competitive activity, economic cycles can all skew the analysis, making the approach untrustworthy. With Test & Learn, you can establish dependent and independent variables that you measure against a control.

Customer Lifetime Value

Another effective usage of the Test & Learn methodology is in maximizing customer lifetime value (CLV) in subscription type business models such as credit cards, health clubs, insurance, etc. There are three main levers which can be pulled to improve CLV: customer acquisition, spend and attrition. Maximizing the rate of customer acquisition, increasing spend during the relationship and minimizing customer attrition. Test & Learn is extremely effective for optimizing each of those levers to increase CLV.

Website Optimization

A third and very common use of Test & Learn is the optimization of ecommerce site design to maximize intended outcomes, such as purchases – a process known as conversion rate optimization. Using testing software, it is possible to design and execute tests of changes in the website – such as text or navigation – to determine the effect on purchase behavior Changes that increase purchases are implemented and those that do not are cancelled. This is a very effective way to maximize the revenue of the site through a series of incremental changes, rather than a large scale overhaul of an uncertain design.

Here’s an example of an e-commerce business using the Test & Learn approach to improve their website conversions:

One of our clients manages a large event space in a major metropolitan city. Most of their revenue comes from ticket sales via their website. The site averaged approximately 1 million visitors per month, producing average monthly revenue of approximately $2 million. However, their conversion rate was only 0.75%, which was substantially below the industry average.

By looking at the user flow metrics, we found that buyers for the event space were abandoning the purchase page at an unusually high rate. Our initial hypothesis was simple: The CTA was below the fold and the text of ‘Visit’ is too vague. To test this hypothesis, we developed a test which moved the CTA above the fold and examined the effect of the following variants for the CTA text:

  • Get tickets
  • Buy now
  • Buy
  • BUY

In this instance, the winner was BUY NOW. Its implementation resulted in the conversion rate increasing by 7%, which translates into almost $1.5 million in additional revenue – just for modifying one CTA!

One final note: remember, testing is only half of the process. For every test, make sure you track the results and ‘learn’ from your mistakes. You want to record what worked and what didn’t, so you don’t have to repeat these tests in the future. This is especially important on bigger sites where teams of folks are responsible for online management. Remember, even unsuccessful tests can increase your knowledge.


During events like a pandemic, predictive analytics are going to struggle due to the reliance on historical data from environments which look nothing like today. Relying on the past to predict the future under these circumstances can lead to disappointing results. But diminishing your reliance on predictive analytics doesn’t mean you should throw out analysis altogether. By utilizing a Test & Learn strategy, you can implement smaller, faster, incremental changes that are supported by in-depth data analysis.

To learn more about Test & Learn or how Enabled Concept can help you implement it, contact us via our website.

Posted in

Ron Bisaccia


  1. […] rather than together. As enterprises start to appreciate the value of omnichannel marketing and Test & Learn strategies, more and more firms are beginning to adopt a holistic approach to customer engagement. […]

  2. […] rather than together. As enterprises start to appreciate the value of omnichannel marketing and Test & Learn strategies, more and more firms are beginning to adopt a holistic approach to customer engagement. […]

  3. […] than together. As enterprises start to appreciate the value of omnichannel marketing and Test & Learn strategies, more and more firms are beginning to adopt a holistic approach to customer […]

  4. […] rather than together. As enterprises start to appreciate the value of omnichannel marketing and Test & Learn strategies, more and more firms are beginning to adopt a holistic approach to customer engagement. […]

Leave a Comment

You must be logged in to post a comment.