The Straight North Blog

Back to Blog

10 Steps to Create an Effective A/B Testing Campaign

Posted by:

To create an effective A/B testing campaign, you don’t need to buy fancy software with complex features. You don’t need to deploy advanced algorithms either. What you do need is to follow a few basic rules and to use common sense. Here are 10 steps to achieve your objective:

Figure Out What You Want to Test

You can test any number of things – your landing page, advertising copy, sales copy, sales emails, and much more. Plus, you can test both on-site and off-site elements. The point of any test is to determine which features and elements convert best to justify spending money on those aspects. Work with a base list of things you want to test, and then test them one at a time.

List All the Variables You Want to Test

After you’ve identified the item you want to test, make a list of all its associated variables. For example, if you’re testing your landing page, list variables such as length of sales copy, headlines, keywords, CTA (call to action), and so on. Do this for all the things you want to test, from newsletters to emails and so on. Type an extensive list of variables for each test item into your test document for reference.

Start Your Test with a Hypothesis

Summarize what you’re going to prove, along with the variable you’re going to test and the winning success metric. This helps you know for what baseline result you’re striving. Then, when you test two options against each other, you can tell which variable performs better than what you have currently. Before starting a new test, create a document with your test hypothesis.

Define a Single, Clear Success Metric

You need to define a single, acceptable success metric before you test to measure the success of your test. Align the variable with the success metric you want to achieve. For example: you’re testing landing page conversion based on the effect of multiple offers to a single strong offer. In this case, your success metric is the conversion rate, and the variable you’re testing is the number of offers. Don’t look at multiple success metrics and pick the winning metric when the test is done.

Change One Variable at a Time

Keep everything constant on your test version except the one variable you are changing. For example, if you’re testing only your sales copy headline, keep all other landing page elements constant. If you change more than one variable, your results are inconclusive. If this happens, you won’t be able to determine the reason for the success of one version over another.

Test Only Those Elements That Need Testing

There’s no point in testing everything even though everything is testable. Apply best practices, existing data analyses, and common sense to evaluate what’s already working. Focus on testing those variables about which you are unsure. Also, focus on testing those variables that have a chance of increasing the performance of whatever you’re testing.

Establish Sufficient Test Volumes for Statistical Significance

For the data to be statistically significant, you need to achieve sufficient volume in results, differences, and test groups. For example, you’re sending an email offer to 10,000 people. You want to see how many people click your offer, which is your success metric. Therefore, you need to achieve a good volume in number of clicks to properly evaluate the statistical significance of the email offer.

Apply Volume to Each Test Group

Apply volume to each test group, especially when you plan to do split testing between the control and test groups. You can do a 50/50 split or even do an uneven split (until 95/5). If you have a winning variable, use a smaller uneven split such as 90/10. If you’re doing the test without a champion variable, go with an even split.

Run Tests Simultaneously

If you’re testing a landing page, test all variables one after the other to avoid too much variation. Some variables change over time, and you don’t want to risk inconsistent results due to a variation in timing. So run tests simultaneously for variables within a version. You can also run tests in conjunction, such as landing page A with newsletter B and so on. Always allot sufficient time for testing to be sure that you’re getting the correct data.

Diligently Document Tests and Results

As you formulate a new hypothesis, open a new document and write it down. Be very specific about documenting every aspect of your test so that you can always refer to previous tests and data in your progression. This way you won’t repeat tests unnecessarily, while simultaneously educating others.

Stephen Cambra works for Invesp, a leading conversion optimization company. He writes articles on this subject for various blogs and publications.