‘Tis the season for the holidays, consumer spending, and major ecommerce traffic increases. Over the next several weeks, the National Retail Federation estimates consumers will spend a staggering $655.8 billion, or just over $935 per person on gifts, services, and other holiday expenses. The weeks leading up to Thanksgiving, Christmas, and other winter holidays will see a definite spike online: the NRF estimates online shopping will rise between 7% and 10% this year, reaching $117 billion.
What does this spike in activity mean for busy marketers who are focusing on conversion optimization? Can you conduct CRO testing during the holiday season without losing sales?
The Case Against Holiday CRO
Any discussion of holiday CRO testing needs to start with a closer look at why some companies run minimal tests during this period – or stop CRO testing altogether. The classic reasons include:
- Being too busy with sales and the website-related challenges that can come with holiday site traffic. Marketers and IT staff may be busy keeping promotions going and ensuring your website has the bandwidth to keep up with increased demand.
- Sites are often under code freezes during this time of year, as developers work on bugs and updates that will launch after the New Year.
- An increase in holiday traffic may also mean that more site visitors are unqualified or acting differently than they normally do during the rest of the year, meaning that data is less reliable.
- Finally, business owners are hesitant to conduct testing at this time of year because any traffic that is shown a losing variation may be less likely to convert.
From sample pollution to the risk of losing sales, running the wrong kind of testing during the holiday season – especially without a clear strategy driving your actions – is risky. However, with the right forethought, the holiday period can be a smart way to generate insights that fuel further testing and better conversion rates all year long.
What Are the Advantages of CRO Testing During the Holidays?
With all the potential risks, why do people conduct CRO testing during the holiday season? There are several reasons why having a solid holiday CRO strategy makes sense:
- During the holidays, websites often see more traffic. More traffic makes it easier to collect more data and insights – often on an accelerated schedule.
- The spikes in traffic make it easier to find the answers to simple CRO questions more quickly.
- Tests conducted in the early holiday period – or those that use an adaptive model like Bandit testing, which is described below – can help you gather insights to optimize your site performance for the rest of the holiday period.
So How Can You Test – Without Losing Sales?
During the holidays, there are two ways to maximize your insights and minimize your risks. One is to use a specific framework to structure the tests you run, and the second is to narrow down the items you test. Let’s take a closer look at each in turn and how they can help you reach your goals.
Using Bandit Testing Instead of Straight A/B Testing
One potential solution, as many experts have expressed, to testing during high volume and high value time periods is using Bandit testing instead of A/B testing. During A/B testing, you highlight a variable, and then send half of your traffic to each option. After you reach a statistically significant point in the process, you’re able to declare a winner and adjust your site accordingly.
With Bandit testing, you take an adaptive model to data collection – which learns over time. As one author noted, “This is what I really like about the Bandit way of looking at the problem, it highlights that collecting data has a real cost, in terms of opportunities lost.” Essentially, as the testing model reveals which variations are winning, traffic is gradually shifted that way in order to take advantage of the higher conversion rates – while a smaller percentage of traffic continues to go to the losing variation. By dynamically driving traffic toward the winning variation, you’re able to begin immediately capturing the benefits while minimizing the opportunity costs of data collection and CRO testing.
Another related strategy is to throttle a lower percentage of your traffic flow toward tests – say, 10% rather than a standard 50/50 split. While it will take longer to get your answer, and it may introduce other variables into the mix that need to be considered during analysis, some CRO strategies use this approach to continue gathering data while also mitigating the risk of lost sales.
Focus Your Tests Strategically
If you’re not mitigating the risks of testing through structure, you may be able to focus on specific types of testing during the holiday period. The greatest risks come when you focus on large site-level structural changes or running tests that can slow down your website. Holiday shoppers are busy, often buying in high volume, and generally want to make purchases after quick price comparison. However, there are certain kinds of tests that are lower-risk during the holiday season. Consider focusing on options like:
- Experimenting with discounts: If a customer has been lingering on your site but failed to make a purchase, could a pop up discount with an urgent deadline help them convert? In many cases, the holidays are a useful time to experiment with discounts and find out what, where, and when they are likely to spur your visitors to take action.
- Building tests around retargeting: Cart abandonments may rise during the holidays, as shoppers price compare, get frustrated with the checkout process, or simply become distracted. If you’re not using a retargeting approach, the busy holiday season may be a good time to see if a follow up email or a discount can woo back shoppers who abandoned their carts.
- Experimenting with better support options: The busy holiday season may be a reasonable time to introduce new customer support options, such as giving site visitors the chance to interact with a live chat agent. Providing expanded options for customer service – as long as you’re confident the backend team can deliver on par with other channels – can be a smart move during the holidays.
- Trust-building and risk-reducing features: In general, the tests which are most likely to help – and least likely to harm – are those that help build trust with consumers and reduce their risk of doing business with you. Examples might include adding trust labels, offering free shipping, or adopting a more generous returns policy, and then monitoring how those features impact sales.
- Promotion and channel optimization testing: Businesses typically invest in promotions and advertising on different channels, yet the landscape changes from year to year. Optimizing your social media, email marketing, and other key techniques during this period can provide progressive intelligence for the rest of the holiday season and give you insights to implement over the next year.
- Urgency: Create a sense of urgency, to see if that moves your audience to action. Holidays provide a natural and non-intrusive deadline to hook your marketing and testing efforts on.
- Audience Data Collection: The holiday period can be a useful time to collect data on your audience, from what devices they’re using to what behavioral patterns they exhibit when on the site. With the use of the right tools, this can often be done passively without influencing the customer experience.
- Promotions on Confirmation Pages: There are a number of confirmation pages customers see during the holiday season: when they sign up for your mailing list, after submitting an order, or when receiving a confirmation email. This is a natural time to test whether a discount or promotion – or a related product recommendation – can spur additional action.
Final Thoughts on Holiday CRO Testing
The key to holiday CRO testing is being smart about it. Where possible, sites should be thinking ahead to the holidays and implementing best practices from the past year in order to optimize their performance during this critical retail period. At the same time, the holidays provide a unique period to conduct testing.
As long as you remember that your audience may act differently during the holidays than during the rest of the year, it’s possible to put that behavior in context. Many companies that run CRO testing during the holidays will bucket those insights for future holiday periods, and retest during more normal periods to see if the results stay steady. Finally, by planning ahead, it’s possible to strategically limit your risk by dynamically shifting your traffic toward winning variations and focusing on holiday-friendly areas with your testing.
What’s your approach to holiday testing? Let us know in the comments below.
Image credit: purchased/licensed Adobe Stock
