Segment is a very popular CDP (customer data platform) that specializes in unifying data sources in real-time across your digital touchpoints and with your ever growing marketing technology stack.

Segment is an especially popular choice with product and engineering teams because it can serve as the organization’s centralized hub for data collection, integration, and syndication. And what really makes Segment so powerful is its 300+ API integrations with top marketing technology platforms.

This has resulted in Segment becoming the foundational data platform for thousands of organizations that now use this unified data to power a variety of business use cases throughout their entire marketing stack and across their digital touchpoints.

Today we are going to do a deep dive on delivering site personalization using Segment data. This can be accomplished through the FunnelEnvy’s Segment integration which makes it easy for Segment customers to test, target, and personalize user journeys using all of that rich user and event data in real-time.

Activating Segment Data with FunnelEnvy

Segment is one of the out of the box integrations that come with the FunnelEnvy Platform.

The first step to activate the integration is to select Segment as a Data Source within the FunnelEnvy Platform.

Next, copy the API Key displayed in the integration details, check the “Active” checkbox and save the integration.

The Segment integration will be listed as active and FunnelEnvy will now start tracking identify calls and associated traits from Segment.

The last step is to configure FunnelEnvy as a new Destination within the Segment Platform. In Segment search for and add FunnelEnvy as a destination.

In the settings you’ll need to add the API key that you copied from the FunnelEnvy platform when you activated Segment integration.

Save the destination and you’re done. Identify calls and track events and associated traits/properties will start flowing from Segment to FunnelEnvy.

That’s it! The FunnelEnvy and Segment integration is now ready to go. If you want to also set up FunnelEnvy as a data source for Segment (Segment as a FunnelEnvy destination) you can follow the steps outlined at the bottom of this help guide. This step is useful for syncing FunnelEnvy audiences with Segment and tracking which FunnelEnvy campaigns and experiences a user is associated with.

Now that we are up and running and sending our Segment data to FunnelEnvy, let’s explore a few popular personalization use cases.

Popular Personalization Use Cases With Segment Data

The most exciting use cases with using Segment data for personalization is the ability to deliver targeted 1:1 experiences based on the user’s relationship with you and serving the most relevant offers for them.

The exciting promise of CDP’s like Segment is they contain your best data view of your users and customers. Now by integrating Segment with FunnelEnvy we can target your users in a more personalized way and present them the best offers and experiences to drive better business outcomes.

Let’s look at how to set up a few of these personalization use cases in FunnelEnvy.

Personalizing for Existing Customers

The majority of websites are static and prioritize the site experience for that new visitor or prospect. This is the right strategy for growth but it results in a suboptimal user experience for our most valuable audience, existing users and customers.

Common sense tells you that if you can, you should personalize the site experience for your existing customers that may be coming back to your site frequently to login and use your products and services.

In this client example, we have a SaaS business where users must login to the website to access their services. Because this client tracks active logins in Segment we can easily personalize the site for this user audience.

Once the Segment integration is activated you can create new conditions, audiences, or goals based on any combination of Segment data. To begin, just create a new condition and select Segment as your data source.

In the below screenshot we will create a new condition in FunnelEnvy based on the user being an active user based on Segment data. In the drop down I can pick from any and all Segment data being passed to the FunnelEnvy Platform. In this case, I select the “Is Active User?” attribute and specify it must be true.

Once created, this new condition can be used to build targeted audiences that I can then use to personalize content to existing customers. My newly created Active User audience can now be easily applied to any of our A/B, predictive, and rules based campaigns, like in this example below:

I can now target active users with personalized customer content the next time they return to my website to login.

We can create simple audiences like in the example above or we can create more advanced audiences that combine multiple attributes from Segment and other data sources. In this example below, I am creating an audience based on the user being an active user AND has viewed our resource content section based on their onsite browsing behavior.

And the best part is that I can set the audience to automatically sync with Segment as one of FunnelEnvy’s data source destinations.

Use Next Best Offers to Convert More Prospects into Customers

Another great use case with personalization is the idea of serving your prospects the next best offer based on where they are in their buyer journey and based on what offers they have already activated.

While this is a very straightforward campaign in your email nurture campaigns, it’s often hard to execute on your website in real-time. Because of this challenge, most sites continue to show the same offer over and over again to prospects even if they already signed up for that offer. The end result is another missed opportunity to provide a more engaging experience to another high value audience, active prospects.

Luckily, this challenge is easily solved, and is just another straightforward site personalization use case when you pair Segment data with the FunnelEnvy Platform.

For this client, a product demo is the most popular entry offer for new prospects, but once they watch a demo there are a number of other offers that can help move the deal forward. They have webinars, calculators, and case studies that all work well as offers in the middle and bottom of the buyer journey.

In this example below we want to target active prospects who have completed a demo. Our client tracks in Segment which users complete a demo based on a Registered for Demo attribute. That allows us to easily create conditions and audiences based on the user being in the demo stage, like in the condition screen below.

With a few clicks we have created an audience of all site visitors that have completed a demo. We can now target our homepage banner to run a personalization campaign that will predict the next best offer for that active prospect from a collection of their top offers.

This is another example where instead of presenting an irrelevant offer, we can provide a more relevant experience and personalize the next best offer for our valuable active prospect audience.

Make Better Real-time Personalization and Attribution Decisions

The first two personalization use cases I shared focused on creating very specific audiences based on user stage data coming from Segment. Our last use case is focused on leveraging all of that rich Segment data to make better personalization decisions and to measure the attribution impact on revenue.

If your Segment implementation is typical of most of our clients, you likely have dozens if not hundreds of data attributes about your users coming from your customer databases, transaction systems, and various other marketing platforms. Typically, Segment has the richest and most accurate real-time data profile for your users, especially when it comes to revenue data.

For example, did that trial user successfully convert into a paid user 30 days later and generate revenue? How much revenue has that user generated over their customer lifetime?

Going back to our previous SaaS client example, they have a service that you typically start as a trial, and they can generate revenue from that user from both a monthly subscription fee and from specific actions they take in their account. In this case, the client tracks all revenue activities for a user with a Segment event named Generated Revenue.

Because all Generated Revenue for a user is tracked within Segment we can easily build a goal in the FunnelEnvy platform that will update whenever that Segment event fires and populate with the actual revenue generated from that event.

Since Segment track events become FunnelEnvy events, they can be used for goal tracking like any other event in our platform.

In the example below, we are setting up a new goal in FunnelEnvy that fires each time the Segment track event named Generated Revenue fires.

In real-time, every time that Segment track event fires we are now tracking that revenue and associating it back to the user and personalized experience they saw.

With this data flowing back to use in real-time we can now unlock two powerful capabilities.

Predicting Better Experiences to Maximize Revenue

Because we can track the total revenue a user has generated we can now start using that data to predict which personalized experience generates the most revenue for the business, not just the most trials or initial orders.

The FunnelEnvy personalization platform uses machine learning to predict which experience will perform best for an individual based on that user’s profile and on the history of how similar users converted on those same experiences.

This allows us to take advantage of all the data we have as the FunnelEnvy platform will predict and promote those experiences that maximize generated revenue for each user.

So in this case Segment data helps us convert more visitors to higher revenue generating experiences, all without the manual hassle of pre-defining segments or personalization rules. We let machine learning crunch all the data and predict the best experience for each user, all in real-time.

Better Attribute Incremental Revenue to Personalization

While generating more revenue is always priority one, as marketers we also need to demonstrate that our programs and campaigns are working and providing strong ROI.

In our example with the Generated Revenue event, in addition to making better predictions, we can also clearly demonstrate the impact on the revenue we are creating through our personalization initiatives.

With our integration with Segment, we are able to credit and report on how much revenue our personalization activities are generating and which personalization experiences covert best. We can then show incremental revenue by running a controlled A/B test that serves half the audience the control experience and the other half the predicted experiences. We then measure the difference between the two groups and report on uplift in conversion and value per visitor and calculate statistical significance.

With the Segment integration, it is now much easier to measure and demonstrate the incremental value of personalization. Not just on the initial order, lead, or trial but on customer lifetime value (CLTV).

This can all be done in real-time right out of the FunnelEnvy Reporting UI without any ongoing support from your IT or data team and without having to wait for delayed manual reporting.

As a marketer, you can finally focus on optimizing your personalization strategy without all the data and operational headaches. Wouldn’t that be nice.

Getting Started

As you can see, integrating Segment data into your personalization program is very straightforward and can unlock some very valuable capabilities and use cases.

If you’re not yet using FunnelEnvy but are interested in personalizing your website using Segment data we’d love to hear from you! You can contact us here: https://www.funnelenvy.com/contact/

When it comes to personalization there is a growing trend with using machine learning to predict and optimize for the user experience at a 1:1 level.

We call the collective use of these machine learning techniques as predictive personalization campaigns.

Predictive personalization refers to a type of campaign where a machine learning model is used to predict what is the best experience to serve a visitor based on current/historical performance and the user’s individual data profile (contextual bandit). Decisions are made in real-time and at a 1:1 level and the model makes use of all the data available about that user and also takes in the context about the location, content, and other factors that go into that experience.

The predictive campaign will send the majority of traffic to the experiences that the model predicts will perform best, exploiting those insights in real-time, but continue to hold out some traffic to continue learning and exploring performance trends for the other experiences.

Just like with an A/B test, you can test predicted campaigns vs. a control to determine if a statistically significant uplift is achieved. But unlike an A/B test, the expectation with predictive campaigns is that it is “always on” and that it’s constantly adjusting traffic to the right experience at a 1:1 level.

The adoption of predictive personalization campaigns is still in its early days for our industry. For many programs, experience and maturity with these techniques are still low but there is a growing interest, especially as the solutions on the market continue to grow.

In this article we discuss best practices we have acquired from years of running predictive campaigns across a range of mid-market and enterprise clients using the FunnelEnvy platform.

What Makes For More Effective Predictive Personalization Campaigns

Below is a list of our observations after running hundreds of personalization campaigns over the years on the optimal conditions that contribute to successful predictive personalization campaigns.

1. A variety of user intent. With a predictive campaign the more varied in intent and behavior that exists with users the more valuable predictive models will be in detecting and predicting better user experiences. The more varied the signal the more value predictive decisions can play a role in deciding and predicting the best outcome for a range of user intentions.

A good example of this is with predictive campaigns on the homepage, where user intent typically varies across the board.

2. A variety of goals to predict for. A variety of possible user outcomes to predict for is another valuable ingredient for a successful predictive campaign. Like with user intent, we want to capitalize on the strengths of predictive models, and the more varied outcomes to predict for usually results in better business outcomes.

Good example of this is with a campaign that rotates a variety of goals or user journeys as the primary offer and call to action (CTA). Like in B2B, where you may have 4-5 different offers like free trial, request demo, request pricing, book a Drift meeting, download whitepaper, or register for a webinar. Having multiple offers to present or predict usually results in better performance of the predictive models.

3. A variety of goal values to predict for. Multiple goals are a good thing, and having multiple goals with different values is even more helpful to your predictive campaigns. Once again we are seeking out scenarios where we have multiple goal options, all with a range of perceived value. This becomes important because we want to give the model more detailed feedback on what is working and what is more valuable to the business. If like in the previous example, you had 5-6 B2B goals but all of them were valued the same, (say $100 per goal completion), all goals would be perceived as equal, providing less signal back to the model. However, if your goals vary widely in range from $50 to $1000 per lead, then the predictive model has far more data and data points to work with.

An example of goal variety we typically see with SaaS clients is that goal values will fall into 2 tiers, lower value content engagement goals (content download, webinars, video views), and higher value sales intent goals (request demo, request pricing, contact sales, free trial).

4. Goals are aligned to business revenue. While goals can vary widely, it is important that your goals are either revenue goals or events that correlate closely with revenue. Ultimately, predictive campaigns do best when they predict which experience is the most valuable.Therefore it’s important that you set up your campaigns to predict high value outcomes like purchases, trial to paid conversions, MQL/SQL, and closed/won deals. Where possible avoid micro-conversion and vanity metrics as your primary KPIs whenever possible. Examples of vanity metrics would be clicks, page views, video plays, etc. Whenever your KPIs are not tied to revenue/business value the harder it will be to have predictive campaigns be effective for you.
5. There is some version of a revenue journey that you can track and optimize for. The more diverse and complex your buyer journey is the more you will benefit from using a predictive campaign that can take in all the data and outcomes and predict for better outcomes for your users.

If anyone has the same exact journey, there is less to predict for. However, if your a typical SaaS business and you have a variety of SMB to enterprise offerings, and you offer self service and enterprise sales scenarios, or you have a longer sales funnel that includes MQLs/SQLs/Opportunity Stages, then identifying the valuable trends across multiple goals in that revenue journey is a specific challenge that machine learning models do better at predicting for.

6. Testing of high impact placements on the page. This recommendation is not unique to predictive campaigns and holds true for all types of campaigns, including A/B tests. If you are going to run a predictive campaign, you need to run it in a high impact location for it to be most effective.So examples of good placements would be primary offer/CTA on page sections above the fold. We have seen predictive campaigns run on small content strips or on content sections below the fold, and while they may contribute value, it’s just not going to generate a significant impact compared to continually optimizing for the high impact locations.
7. Higher volume of traffic and conversions. Another no brainer but worth calling out. You want to run campaigns on highly trafficked pages that correlate well to conversions and revenue (think product and pricing pages vs. community or blog pages).

You should be prioritizing opportunities where you can move the needle and truly impact revenue for your site.

8. More available data attributes for predictions. Predictive models need data to provide relevant signals. In many of the above recommendations I call out that the more variety of intent and goal outcomes, the more data points the model has to work with for its predictions. The objective here is to generate more contextually relevant data attributes that can feed into the model. The strategy here is to instrument more meaningful data events and attributes to feed the predictive model, often referred to as feature engineering.

Examples here would be setting up more targeted audience segments (flagging prospects and customers for example), or integrating a new data source like a firmographics API from Demandbase or Clearbit, or integrating with your CDP, CRM or Marketing automation platform, or defining content and product affinities based on user behavior and pages visited. In each of these examples we are feeding additional relevant data around user intent, behavior, or data we have about our users. All of which can help a model identify additional experience/user matches that correlate better to revenue.

9. Stakeholders understand and appreciate the differences between A/B testing and predictive campaigns. This is an organizational consideration, but when you run predictive campaigns, there are certain differences that you and the larger organization needs to be aware of and comfortable with. Unlike an A/B test, your experiences may not be sticky to a user over the life of the campaign (optional). Secondly, predictive campaigns are designed to maximize revenue, you may not have a clear winner or a straightforward outcome or winner after running the campaign. In predictive campaigns there is often no one winner. You find that different experiences perform better for different user segments. Some organizations are very comfortable with this reality, others struggle and prefer the simpler situations and outcomes that come with traditional A/B tests where you have a clear learning and you can full scale a single winner for the entire population.This is where you need to educate your organization on the pros and cons of running predictive campaigns.
10. Stakeholders value and are responsible for revenue. While everyone in the company will say they value revenue, not everyone is responsible for generating revenue on behalf of your business. And that is ok. But, when it comes to predictive campaigns the best outcomes are usually achieved when the primary stakeholder is aligned to revenue targets and takes the responsibility to grow those revenues as best they can with proactive initiatives like site personalization. This is where we often see the most success with our clients. Being motivated by revenue vs. just learnings, will result in a more focused approach to revenue optimization, and in many cases a predictive campaign is more effective than an A/B test in maximizing revenue for your site and user experience.

Please keep in mind that while these are 10 preferred conditions we believe improve the likelihood of your predictive campaigns being successful, this full list should not serve a prerequisite or requirements before you launch your first or next predictive campaign. Typically, if you can align on 2-3 of these conditions you are often in good shape to see success.

The goal of this list was to share with you lessons learned and what to look out for as you continue to grow and expand your personalization programs and are ideally incorporating predictive campaigns more often into your portfolio of marketing and personalization initiatives

Getting Started

Now that you know what to look out for when designing and prioritizing your predictive campaigns, let us know how we can help you maximize your personalization initiatives.

We’d love to hear from you! You can contact us here: https://www.funnelenvy.com/contact/

For the majority of marketers, when you talk about CRO and optimization programs, people immediately jump to A/B testing and assume that is the primary tactic and is the only tactic to demonstrate success for their programs and initiatives.

While A/B testing is a critical tool in your optimization program it shouldn’t be the only option on the table. While A/B testing is a valuable solution it is not a one size fits all solution for all your business challenges.

With advances in new data capabilities, marketing technologies, and real-time computing power in recent years, there are now more ways to solve for the same optimization problem.

For this discussion, we want to focus on two different techniques.

A/B test – A controlled experiment where traffic is split across 2 or more experiences and visitors are randomly assigned an experience and they stay in that experience for the duration of the test. At the end of the test, you determine a single winner based on which experience generated the best outcome for your primary KPI based on a predefined sample size/test duration and ideally reaching a specific statistically significant threshold. You then stop the test and full-scale the single winner for all traffic either directly in the testing platform or hard-coded into your CMS/platform.

Predictive Bandit – An experiment where traffic splits are not equal and where visitors are not randomly assigned. In these campaigns, a machine learning model predicts what is the best experience to serve a user either based on current + historical performance (multi-arm bandit) or based on current/historical performance + the user’s profile (contextual bandit). Like an A/B test, you can run the predicted experience vs. control to determine if a statistically significant uplift is achieved. But unlike an A/B test, the expectation with bandits is that it can run ongoing.

Now that we better understand the two techniques let’s explore reasons why both have a place in your optimization strategy.

Pros and Cons of A/B Tests

A/B testing is embraced in the analytics community as the smart and scientific way to measure the impact of change. Controlled experiments are used in testing the effectiveness of new medicines, academic/scientific studies, and of course in marketing. A/B testing is not new to marketing. It’s been done for decades in the direct mail/direct response world. And has become the recommended way to test changes on your website and mobile app assuming you can convince your stakeholders that testing is needed within your organization (it’s 2020 but still some organizations resist).

PROs:

1. Quantify the impact of changes on your site. Don’t leave change to chance, measure and quantify the positive and negative impact of changes in experience with confidence.
2. A/B tests are well understood in our industry. And for the most part well understood across an organization. Statistics may not be, but in general most folks understand the approach. In the end, if everything goes according to plan you have a clear outcome; either control won, or one of the challengers is declared the winner.
3. Clear learnings. Related to the 2nd benefit of being well understood, benefits of an A/B testing is not just impact on business results, it’s shared learnings of what may or may not work for your site/business. With A/B tests you ideally gain a better understanding of your visitors and customers.
4. At scale, you can create an organizational culture of experimentation. This change in culture leads to more creativity, risk taking, and better data driven decision making. Typically organizations that test more, make better and more informed decisions for their organization.

CONs:

1. Success rates will vary, but are often low. Industry averages place the average success rate of A/B tests at 30-40%. So you have to expect that the majority of the time you are not going to end up with a new winner. Even within the winners, even fewer are high impact wins. The one hidden benefit to this, is that it makes the argument of why A/B testing is so important. If we didn’t test, the majority of the time those great ideas we think will perform better, actually perform worse or have no impact.
2. One size fits all and the winner takes all. When you analyze your typical A/B test you will often see at a segment level, different groups convert very differently across the variants. At the end, you pick the variant that was best on average across all your traffic and full scale that one. But in doing so you do leave some money on the table, as the winner will not be the winner of all segments.
3. Results can and do change over time. If you look at performance trends over the life of the test, it’s often the case where test results trend up and down over time. Guess what, that data reality continues to occur even after you hard code and full scale the winner. You can lock in the experience but you can’t lock in the results going forward. Results will continue to change, and a decent percentage of the time, as tests continue over time you experience a regression to the mean, where results start to flatten out. After all, if you run the same test twice, you will seldom get the same results. This is why, for many programs when you full scale that winner you usually don’t experience that lift ongoing. It still is a better way to make decisions, but as we know audiences and behaviors change. Seasonality can also be a factor.
4. You sacrifice business value for concrete learnings. By design, an A/B test is designed first and foremost to generate a solid learning. Usually you are willing to sacrifice short term uplift if a variant does better than the rest, and willing to suffer through some short term downside if a variant clearly underperforms. You are willing to accept a sub optimal impact on revenue during the duration of the test because you are prioritizing a clear test result for short term business benefits.
5. You need traffic. Not every organization can run A/B tests. Sufficient traffic and conversions are required to reach a statistically significant outcome. Not all sites have that, especially in B2B for example.
6. Operational costs can run high. From selecting a testing platform, bringing on web developers, additional creative, and data analysis, the tool and people costs can be meaningful. Plus, there are the operational costs of introducing more time and resources to launch something, and the occasional negative costs from broken experiences or flawed tests. All marketing teams and programs incur people, tools, and operational costs, and testing is no different and often carries more cost overhead.

While I go into more detail on the cons with A/B tests, I do some because some of the cons are less understood. Still, when done right, A/B test is worth the effort and the benefits will far outweigh the cons.

But as I hopefully outlined above, A/B tests do have different strengths and weaknesses.

Pros and Cons of Predictive Bandits

Bandits have gained more popularity in marketing in recent years as computing power has advanced to the point where real-time machine learning predictions can be applied in more and more marketing technologies and for more use cases.

We have seen the trend emerge in digital advertising where bid and creative recommendations are often driven by machine learning decisions.

However, when it comes to site optimization the adoption and acceptance of bandits as a proven technique is still in the early days. For many programs, experience and maturity with these techniques are still low. Even though some of the most popular testing platforms have included those capabilities for a number of years. Let’s discuss why.

PROs:

1. Bandits are by design biased toward business outcomes. Unlike A/B tests which are designed to maximize time to a clear learning. Bandits are typically designed to maximize the business outcomes at the expense of clear and precise learnings. The algorithms typically send more traffic to experiences that perform best, and route traffic away from experiences that underperform as a whole or for a specific audience segment.
2. Bandits use all the data to your business advantage. In A/B tests you may use data to inform and drive your test hypothesis, but when it comes time to setup your test, you typically set it a controlled randomized experiment where traffic is split evenly and your visitors are randomly assigned to one of the variants during the duration of the test. Again, this is absolutely the right way to run a controlled experiment to generate your best chance at a concrete learning. However, this also means you are ignoring the data and trends that live within a test and the micro trends of which traits and segments perform best for each variant. With Bandits, machine learning is consuming and using all available data on the variant performance and the user to determine the best possible user experience for that user to drive the optimal business outcome. It’s far from perfect, but you are not leaving the outcome to chance. You are using all the data you know about the user and situation to make the experience decision to maximize your business outcomes.
3. Bandits adjust to changing trends and behaviors. With Bandits, the intention is for it to be always on and constantly adjusting to the latest performance trends of the campaign. Unlike an A/B test where you pick one winner and lock it in, a Bandit can adjust as results change over time, and minimize any loss in performance and capitalize on any shifts in winning experiences.
4. Bandits can work with less traffic. Because you are optimizing for revenue instead of learnings, you can still benefit from bandits even though you have less than optimal traffic to run a clean test.
5. Bandits work well in time sensitive situations. Popular content and Black Friday sales are good examples. By the time an A/B test gives you the right answer the opportunity may have passed you by. With Bandits, it reacts in real-time to the trends and that allows you to take advantage of short term and seasonable situations.

CONs:

1. Bandits are hard to interpret, understand and communicate. While A/B tests are well understood, bandits are not. The fact that decisions are controlled in real-time by a machine learning model vs. set A/B splits, means users do not have certainty about why an experience was shown or control over the traffic rotation.
2. Bandits offer limited learnings. A/B tests by their design as controlled experiments are designed to produce learnings. Bandits will typically sacrifice clarity of learnings when it comes to which experience works best. You can often be informed about what segment or what feature influenced the bandit model decision, but it’s not as clear cut on what is the final absolute winner. Often you are A/B testing the technique, do bandits outperform control or an A/B test for this page/site. But with bandits it’s harder to generate clear learnings on winning experiences. In the end you are optimizing for revenue and other business outcomes at the expense of a clear isolated learning. As an organization you need to be aware and accept this reality and your stakeholders need to accept this reality too.
3. Bandits can provide an inconsistent user experience. As we called out earlier, to maximize business outcomes, bandits by design do not make experiences sticky to the user, and will often serve a different experience to the same user if the data suggests it will result in a better outcome. While this maximizes revenue, it can lead to experiences being more dynamic and changing for a given user. While dynamic websites are generally considered a positive, because it is not very common in 2020 (surprisingly), this dynamic approach to site experience can be a drawback for some users.
4. Bandits require more thoughtfulness on experience design. While you can A/B test most things from layout, to copy, to color, to offers, the same isn’t always true for predictive bandits. Bandits work best where you have varied user intent and ideally varied offers and outcomes. Bandits are more effective if they can predict for more outcomes for a larger range of user intent. If you are running CTA color and size changes you are better off running a traditional A/B test. There is likely less signal in the data in terms of user preference of a button color/design and the results will likely hold true on average for most users. That is why bandits work better when intent varies across the visitors and potential offers displayed can also vary.

When to Use A/B Tests and When to Use Predictive Bandits

Now that you know the PROs and Cons for A/B tests and predictive bandits let’s talk about some practical applications of each and when you should one over the other.

When A/B Tests are Recommended

1. When you need a clear learning and more certainty on the final decision. Examples here would be a pricing or homepage test.
2. When you want to lock in a specific design or experience for all. Examples here are things like a new homepage design, a new form layout, or say our site wide CTA treatments. Here there is more operational value in locking in that specific winner and then optimizing further on that.
3. When intent and offer options are narrow in scope. When intent for all users is similar the offer is the same for all, then A/B testing usually works better than predictive bandits. An example would be optimizing for the final cart checkout page. Everyone there is ordering (or not) and the offer is a checkout (or not).

When Predictive Bandits are Recommended

1. When you want to maximize revenue. If you have aggressive revenue goals then bandits are a better tactic to get you there as you are using all the data available to make the optimal revenue decision. A good example here is presenting the right offer on a homepage hero or promoting a specific price/package on the pricing page. In those situations, the general site is the same but you are predicting the best offer and experience to spotlight to maximize revenue.
2. When intent and offer options vary broadly. Predictions work better when there’s a wider range of users and intent and a wider range of offers and outcomes to present. This is where the value of machine learning and crunching dozens and hundreds of data attributes in real-time is helpful. Good examples here can be homepage, brand landing pages, and pricing/plan pages. In these scenarios intent and options/offers presented can vary.

Now that you know the strengths and benefits of both tactics I think you will agree that both tactics should be part of your optimization and personalization toolkit.

Running A/B Testing and Predictive Campaigns in FunnelEnvy

With the FunnelEnvy platform we give you the ability to run both A/B and predictive campaigns and apply the right tool for the job. Unlike other platforms, both tactics are available as part of our standard license.

To get started you just create a new campaign and then select the preferred template option between Predictive and A/B testing.

If you selected an A/B testing then the “Campaign Decision” section will default the settings typical of an A/B test.

As you can see A/B testing is the decision type defaulted, and the “Persistence Variations Decisions” feature is checked so the same experience is served to the user across sessions. Lastly, you can adjust the traffic allocation to determine what percentage of traffic enters into the test. By default and typically it’s set to 100%.

If you were to select a Predictive campaign template, the decision type would instead default to Predictive as seen in the screen below.

In addition, the “Persist Variation Decisions” feature is not selected by default. As mentioned earlier, we typically see improved revenue performance when the campaign can rotate different offers over time to the same user. That makes sense, as a user may not respond to an initial offer, but maybe convert when presented another. But we do recognize their legitimate reasons to also make predictive decisions sticky, especially when running in locations like your pricing and plans page. And like with the A/B template, you also can set traffic allocation from 0-100%.

What is specific to Predictive campaigns is the predictive experiment options. Here you have two choices to make. The first choice is what percentage of traffic should be included in the predicted group vs holdback. The experiment section is where you can test the incremental value of a predictive campaign by holding back a portion of your traffic as the control.

In the screen below we have set the holdback to 50%. This will result in 50% of the traffic being assigned to the predictive experience and the remaining 50% gets served a control. This allows you to test the incremental value of running a predictive campaign on your site.

Typically, we recommend the client start a new campaign as a 50/50 campaign and as they see the predictive campaign outperform the control we then recommend full scaling the predictive campaign to 100%.

The second option you have with a predictive campaign is to determine the composition of your holdback group. You can choose between assigning the variation a specific variation, like baseline control, or you can set holdback to random in which case the holdback will run as an A/B test for that traffic across all the available variants. The “Random” option is useful if you want to determine the incremental uplift in running a predictive campaign vs. an A/B test.

As you can see, our predictive campaign setup still allows you to isolate and measure the incremental impact of a predictive bandit approach by running it as a controlled experiment.

Our typical client will often run both types of campaigns simultaneously on different areas of their site and targeted for different audiences. Rather than having to choose one tactic or another, FunnelEnvy clients have the best of both worlds.

Getting Started

As you can see, A/B testing and predictive bandits are both valuable tactics in your optimization/personalization programs. Like any tactic you need to pick the right tool for the job.

If you’re not yet using FunnelEnvy but are interested in personalizing your website using a combination of A/B tests and predictive campaigns we’d love to hear from you! You can contact us here: https://www.funnelenvy.com/contact/