It’s somewhere on your ever growing to-do list – “figure out A/B testing”. You may have even read how the Obama campaign increased sign ups by 161% and Twitter increased clickthroughs by 173% through testing and measurement.

But who has time for that? You have a product to build, investors to pitch to, and customers to find. Testing and analytics seem too complex, only for the high traffic sites and a big distraction. Oh and did we mention – we don’t need numbers, we have a huge disruptive vision!

It could be a colossal mistake and a significant missed opportunity.


Instinct is good but data is great. Wherever you are on the startup journey, there can be significant advantages to A/B testing immediately. It’s one of the most effective tools for a digital startup trying to reduce wasted effort, make better decisions and engage customers. The good news is that the tools and techniques have improved to the point that it should only take you 15 minutes to run your first test – and integrating testing into your process will yield dividends throughout the life of your startup.

To maximize the effectiveness of your testing efforts it’s important to understand the Why, When and How of A/B Testing based on where your business is.

1. Why You’re Testing & What to Test

It’s crucial to have your (and your team’s) expectations with respect to testing objectives and outcomes.

  • For the very early stage startup you’re main objective is to identify & learn about a specific market by testing messages and potential product offerings that resonate.
  • Later on (closer to product / market fit), you have a better understanding of your market but are working on testing and gathering evidence to support major aspects of the business or product, such as pricing, the onboarding experience, or your referral program.
  • In the growth stage you’re optimizing – testing specific aspects of the online user experience to quantitatively improve engagement, plug leaks in the funnel and increase conversions.

As a general rule the less mature your startup the more you should focus on coarse-grained (high level) tests, and worry less about the small stuff. Initially it’s about high level value propositions and market segments, later on you’ll include tweaking copy and minor design elements. Having a good understanding of where you are and what you should be testing will help avoid the problem of wasting time & effort testing at too small a granularity to achieve meaningful results.

2. When to Start Testing

The goal of each A/B Test is to derive an actionable conclusion from an observation that is unlikely to have occurred by chance. As a result you’ll need to have a reasonable number of visits (traffic) to the test to actually produce statistically significant results. What ‘reasonable’ means in the previous sentence can vary, but the the less traffic that your test gets the longer that you’ll have to run it. If you have Google traffic you can get meaningful results in minutes, but for us mere mortals I often try to have tests complete by 4 or 5 weeks at the high end. There’s a great online experiment calculator by Dan McKinley that can help you estimate how long your tests will take to run.

So if you estimate that you’ll need 400 visits to a test to arrive at conclusion that you’re confident in, where do you get it? If you are far enough along that you have enough traffic to your site then you’re all set, you’re all set and can skip ahead. If not, you’ll probably need to figure out ways to drive that traffic to the test (which is out of scope for this post but could include paid ads or social campaigns). The good news is that this will let you test various traffic sources and market segments, and figure out which ones you’re able to engage best with your product and message.

3. How to Test

If you’ve figured out what stage you’re at, what you’re going to be testing and how to get traffic to it then this is the easy part. The available tools have advanced significantly in recent years both in terms of sophistication and ease of use. Again it helps to think about where you are as a company and what you’ll be testing:

  • For early stage companies Unbounce integrates landing page creation with testing, making it a good option to get testable pages up with minimal techical knowledge. If you already have landing page variations Google Content Experiments (formerly Google Website Optimizer) is integrated into Google Analytics.
  • Offering much more capability (at a correspondingly higher price) are Optimizely and Visual Website Optimizer (VWO). Both are similar, providing you with a WYSIWYG editor and render variations on the client with Javascript. You also have more sophisticated options such as multivariate and multipage tests. I’ve used both and prefer Optimizely but many other A/B testing consultants use VWO.
  • Heatmaps, clickmaps and similar visualizations can often yield insight into user behavior that numbers alone don’t. VWO has integrated heatmaps into their tests, while Optimizely takes the integration approach with CrazyEgg and ClickTale.
  • A/B testing tools provide data on discrete tests, but determining where to start testing and how your conversion rates are improving over time you need a complete analytics solution. I’m sure you already have Google Analytics installed (right?) but for more complete, user centric data you may be inclined to look at KISSMetrics or Mixpanel.

You may have seen some pretty incredible examples of what single A/B tests have done for landing pages and conversion rates, but the real power of being disciplined with testing & measurement is the cumulative value that each incremental improvement produces over time. That’s often a much less risky and more cost effective approach than any number of radical site redesigns.

Happy Testing.

Arun