Everything you need to know about a/b testing email campaigns

A/B testing emails is a process in which two different versions of the same email are sent to subscribers. The goal is for one version, A or B, to be more successful than the other at increasing subscriber activity and revenue. Basically, you want your recipients not only to open an email but also click-through on links from it that take them through conversion processes like purchasing products or signing up for newsletters with their contact information.

Table of Content

One of the easiest ways to make more money from your eCommerce website is by driving traffic. But how? Start with A/B Testing Email Campaigns, a great way to get those additional conversions and reducing churn rate effectively. It's an effective method to figure out what changes you need to make to achieve increased open rates and click-through rates in no time!

A/B testing is a powerful tool in today's email marketing kit. In one experiment, SitePoint tested out different layouts on their newsletters for an entire week and saw that they got much better responses from the people who received emails with new designs than those without them—32% more conversions just by changing their layout!

A/B testing is a component of Conversion Rate Optimization (CRO), where you use qualitative and quantitative user insights to understand your potential customers. You can gather these data points through A/B testing and then optimize your conversion funnel based on the information gathered from this technique.

This blog will show you what A/B testing really is and how it can improve your open rates, as well as give you some interesting ideas for A/B testing your emails.

What is A/B Testing?

In the context of email marketing, A/B testing is used to determine which campaign variation garners better results by sending one version of an email out to a subset group and another version to another. It can be as simple as trying different subject lines or complex enough for entirely new templates, but in every case, it's about discovering what works best so you know how best to deliver your message.

In simple words, A/B testing is a process of comparing two versions of the same thing and then choosing one. Typically, whichever email version gives higher conversion rates becomes the winner, and that winning variant can help you optimize your site for better results!

Think about the last time you purchased a product online. Chances are, it was because of an A/B test! Many companies run these tests to figure out what advertising is most efficient and gets people clicking more often on their site. It's also used for other things like deciding where new products should be featured in stores or figuring out which model car color will sell best next year (maybe green?).

If you are a Shopify store owner and using LatticeAI for A/B testing your emails, then it becomes really easy. This email service lets you test and sees what works for your customers. You can easily use its features to create two versions of each campaign, send it out automatically to a different subset of people, then analyze which version is more effective based on open and click-through rates.

How to perform A/B Testing on Email Campaigns?

LatticeAI's drag-and-drop email builder makes it quick and easy to run the A/B test on your emails, so you don't have to code multiple versions of the email, just change what you want and click "send". However, before you begin, here are some important steps you must follow to launch a successful A/B test:

Step 1: Do Your Research

Researching your email marketing campaigns and how they're currently performing ensures that you know to build an A/B testing plan. You'll need data on things like who is opening emails, what percentage of those people are clicking through the Calls To Action(CTA) button, where conversion goals vary from campaign to campaign for different competitors etc.

Step 2: Have a Hypothesis

The basic hypothesis is the most important part of A/B testing. If you don't have a good one, then your campaign will not perform as well. Here are some examples to get your creative juices flowing:

  • Personalizing the subject line with your first name is crucial for your email campaigns. It will stand out in the inbox and increase the chances of it getting opened!
  • Add a button instead of just a text link to grab people's attention and get them to click-through.

Step 3: Prioritize your A/B Test

You will likely have a lot of different ideas for these areas that could provide great results. However, when prioritizing which tests to run though, it is best not to overcomplicate things by introducing too many variables into one single A/B Test campaign.

Note: Never make two drastically different sets of emails or SMS messages when running an A/B test. The distinction should always be based on a single parameter, eg: a unique subject line or different background color, etc.

Step 4: Analyze your results and Deploy

Not every A/B test you run will increase conversion, but that's not the point. The key to this is learning from each variation and using what we've learned for future campaigns so they're better than ever before.

When testing out an A/B campaign be sure to spend some extra time analyzing the result - including looking at things such as increased traffic or how much people are engaged with specific content. If everything goes smoothly then congratulate yourself but also make note of why certain parts were beneficial through "insights".

Step 5: Think like a Statistician (Optional)

In addition to the above steps, you can also include two different statistical approaches to testing: Frequentist and Bayesian.

The frequency-based approach of probability defines the likelihood that an event will happen if it occurs a certain number of times(frequency) in each trial. This is not always accurate when applied to A/B testing, because some people using this method may need more data than others for their conclusions. For example, someone with only 50 visitors tested would have less information on which to base their decisions compared to another person who has 1 million trials and can come up with much better results quicker. In this technique, you only use data from your current experiment.

Bayesian statistics is a theory that analyzes beliefs in an event and how it changes based on new information. In this case, the more you know about an event, the better outcomes you can predict. Instead of being fixed, the probability can change as more knowledge is gathered about an event. The Bayesian approach uses existing data from previous email marketing for concluding.

What should you test?

LatticeAI lets you test any aspect of your email marketing campaign for better results. Here are a few ideas to get you started:

Subject Lines

Your email subject line can make or break your email campaign. They are like the gatekeepers of your marketing experiment. According to ConvinceandConvert,69% of email recipients report email as spam based on email subject lines alone.

Subject Lines

It is the first thing people see when they open their inbox. The importance placed on subject lines for email campaigns cannot be stressed enough – it's a vital component in determining the open rates and should absolutely be incorporated into your A/B tests to drive increased opens! Here are some ideas you can use:

  • Length: The ideal length for an email subject line is generally 61-70 characters and email marketing professionals are divided on this question. However, a recent survey from Return Path has identified 71 as the optimal number of letters.
  • Word Order: If you want to make your email subject lines more engaging and attractive, it's important to think about where the words go. Place key phrases up front for people who are scanning their inboxes quickly or at a glance will be able to see them first. For example: "Get 25% off on your next purchase using this coupon code" might attract more attention than "Use this coupon to get 25% off on your next purchase" if someone is looking closely through an unorganized list of emails that they have received recently because it starts with something intriguing!
Subject Line length | Source: ReturnPath


You may include a tag referencing the customer's first name with an interesting subject line for them when sending out these types of messages. This can be done by using a subject line such as "Your items are waiting, Lee!" instead of "Items are waiting."

You can personalize your email campaign even more by including products that you think they'll be interested in. For first-time customers, include trending or best-selling product recommendations. For existing customers, add products of more variations based on their past purchase, e.g. same item with more colors/fragrance.

With Personalization Tags, you can make your messages stand out and drive people to take action that benefits both of you!

Personalized content is no longer a nice-to-have: it's now necessary for reaching those elusive inboxes with ease, which could lead to increased open, click-through rates--and most importantly sales.


Do you think visuals in your email campaigns are a powerful way to get your message across? Research shows that the human brain processes graphics 60,000 times faster than text. This means using images in an email campaign can help explain complex messages more quickly and easily for subscribers.

You could A/B test different kinds of visual layouts within emails to see which one has the most impact!

A/B Test - Image or no Image

In a study of 10,000 emails each with an image and without images, the conclusion was that using them in some campaigns works better than others. If you want to see if they have a positive effect on your business, then try A/B testing by including images in some emails and excluding them from others!

The most important finding from this A/B test is that when you use these pictures it's all about timing. The winners were those who included one or two at the beginning to get attention before they move on to writing their message.


Visuals can make or break an email campaign. The way you style, for example- your design and layout - is important in determining the success of A/B Testing. Different visuals have different effects on your recipients, and the way they perceive what you're saying can change according to their tastes. That's why LatticeAI offers pre-designed templates that are customizable so every campaign looks like a custom design!

A/B Testing - Styles

Almost everyone uses email today - from millennials who prefer quick glimpses at information while multitasking (think: scrolling through Facebook) all day long, to people who are retired and have more time available.

No matter which demographic group your customers belong to, one thing stays true across the board - they prefer something eye-catching and quick to demonstrate the content, hence, an attractive template with the right color combination and minimal design will always work wonders.


We're living in an age where great writing is more important than ever before: if you can't quickly and easily explain your product or offer, you'll struggle to get enough clicks on your campaigns. This trend towards shorter copy design for emails comes as no surprise with how texting has replaced traditional emailing amongst many younger people (and we all know what happens when someone’s text message reaches its limit).

However, if you are sending an informational email related to a detailed summary of products/services, you may opt for longer emails. This will provide readers with deeper knowledge about what you offer and increase conversion rates.


The tone you use in your email marketing campaigns can have a big influence on click-through rate and purchase. Studies show that when you incorporate positivity into the body of an email, readers are more easily able to understand what is being communicated and will be much more motivated to buy than if they read something neutral. Test different tones for yourself using A/B testing so make sure every email marketing has its own unique personality!

Every business owner knows that marketing is crucial to the success of their company. Some have found, however, that a positive tone in email copy can be more effective than one with negative undertones when it comes to driving clicks and purchases.

Call To Action

CTAs are one of the most important aspects of your email marketing campaigns. They help increase click-through rates and keep readers on board with your company. In the example below, you can see that Sephora has included a prominent call to action that ensures readers know exactly what they need to do next.

There are usually two options for inserting call-to-action in your email campaign: Button vs Text and Single vs Multiple Call to Action buttons.

Button Vs Text

In a study conducted by Adespresso, it is found that using buttons is a better technique, and brands can acquire a 27% increase in their click-through rate by using a button instead of a text link.

However, this may not work for all types of e-commerce businesses. To decipher which ones work the best for you, use LatticeAI to test your email campaigns with both text links and CTA buttons for your unique audience.

A/B Testing - Call to Action

Single Vs Multiple Button

A study found that clear, specific copy like “Book now” has positive effects on email campaigns than more generic phrases such as "Read more." The click-through rate increases by 10% when the action is clear and urges you to take an immediate next step.

Emoji or No Emoji

There’s a whole debate of whether emojis are a hit or miss in 2021. The pro-emoji argument says that they have enabled email marketers to convey feelings, emotions, and desires without the use of words which was previously limited by character constraints. Regardless if you're for them or against it, there is no denying their ability to stand out among other emails in your customers' inboxes while also conveying complex messages with just one emoji!

Top popular emojis | Source: MailChimp

To find out what your customers think about this new trend - put your email campaign up and A/B test: send two versions—one containing emojis and another not—to different segments via email blasts; then measure open and click-through rates.

Urgency or No urgency

One of the cleverest ways to improve your emails’ open rates is by using people’s FOMO. According to a study done by Mailchimp, words like urgent, breaking, important, and alert tend to be more actionable for subscribers than others in terms of improving email open rates dramatically. Similarly phrases such as ‘# days left’ and ‘last chance!” are also found useful when it comes down to playing on FOMO (fear of missing out).

Right Timing

Thinking about when to send out your emails? Mailchimp conducted a study that found Tuesdays, Wednesdays, and Thursdays are the best days.

The results of an email campaign can be ruined if you don't know what time is optimal to send them at. While many people assume Mondays would be the day for sending mailers due to it being common knowledge in most workplaces on how much busyness happens through weekends; but according to Mailchimp's research, Tuesday-Thursday appear as more ideal times while Saturdays have high traffic rates which could divert attention away from your emails inside their inboxes - not good!

How many days to wait?

Before we jump right into the technicalities, you need to understand what is "sample size" in probability and how it is applied in A/B Testing.

Sample size means the number of subjects you include in your study, i.e. number of customers you perform your A/B Testing on.

The formula for determining how long you should test is: Expected experiment duration = sample size/number of people who opened your email.

So, in our example, if the tested email had an open rate of 2000 per day, then let’s plug in the numbers:

If we consider a minimum improvement of 30%, the minimum sample size is 6,756.

Expected duration = 6,756/2,000 = 3.3 days

So, one must run the A/B Test for three days, right?

The answer is NO!

Why not, if our calculations demonstrate that all we need is three days to get our A/B Testing results?

It has got a lot to do with statistics. The fundamental aspect of our calculation is the assumption that we are selecting a random sample size for the email we are testing. Without that assumption, the laws of probability will not be applicable.

In A/B testing, the sample size is more important for the accuracy of calculations than the test itself, for deciphering the duration of the A/B test, and for determining a winner. However, this could be a tough job to do, unless you are a qualified statistician.

Therefore, to help you get accurate A/B Test results, experts recommend that you run your test for a minimum of one to two weeks.

What are the Mistakes to Avoid While A/B Testing?

Mistake #1: Testing too many elements together

Industry experts warn against running A/B tests on multiple elements simultaneously as this may pose a hindrance in analyzing which of the changes were beneficial and which of them weren't. For example: if an email has several elements that are put to test, like subject lines, style, images, etc. you cannot really tell which element influenced your customers to book your service or purchase your product. Therefore, prioritization of tests is statistically significant to successful email marketing A/B tests.

Mistake #2: Having an unbalanced audience

Testing unbalanced users is a common mistake in business. However, this can be avoided by following the appropriate testing protocol with your target customers. Using different customer demographics for tests will increase or decrease campaign success rates and generate inconclusive results when they should not-- which could lead to the failure of an otherwise successful email campaign strategy!

Mistake #3: Not Following an Iterative Process

A/B testing is an iterative process, i.e. every A/B Test leads to another consecutively. Businesses often give up on A/B testing if their first A/B Test fails but to improve your chances for success in the future, you must gather data from past experiments and deploy them in your next experiment.

This will increase statistically significant data points and lead to optimized versions of elements for successful email A/B Testing!

Mistake #4: Incorrect duration

Based on your customer and goals, it is important to run A/B tests for a certain period until achieving statistical significance. If you do not give the test enough time or have too many variables in place at once, then this can result in an inconclusive verdict.

The duration of the campaign varies depending on factors such as existing website traffic and company size; however, generally speaking, let campaigns last no longer than three weeks before declaring one variation victorious over another.

Mistake #5: Sticking to generic A/B Testing methods

You don't want to commit mistakes early on in your experimentation journey. While most marketers recommend that you start off by running small A/B tests, this can cause issues later down the line when testing multiple permutations of CTA buttons and other variables such as color, text, or banners for pages. In such cases, you may use the Bayesian or Frequentist approach of email A/B testing for better results.

Mistake #6: Not Analyzing A/B Test Results

What businesses often fail to do or find difficult is interpreting their A/B test results. Interpreting test results after they conclude is extremely important because it helps you understand why that email A/B test succeeded, so your next one can be just as successful!

A poor email A/B Testing is pivotal for the success of your next marketing campaign. With every failed subject line or unattractive visual, you gain more knowledge and data about what does or doesn't work in terms of marketing strategy - this will help guide future tests so that they're successful!

The important parts to remember from a negative result is not just whether it was unsuccessful but also how close it came to being successful; if there were any factors outside our control (such as competitor's strategies) which impeded on its results; and finally, where did we go wrong?


If you want to improve the results of your email marketing, consider A/B testing. This is a great way to gauge which emails are most effective in terms of open rates and conversion rates. And when done correctly, it can be an extremely cost-effective solution that pays dividends over time as you see success across all metrics. Contact us today if you're ready to perform the A/B Testing approach for your targeted customers and take your email marketing game up a notch!

How can LatticeAI help?

LatticeAI can help you make your emails more effective with its powerful A/B testing feature. It lets you create an unlimited number of campaigns and test them against each other in real-time, so your team can see which emails work best right away. You don't have to wait until the end of a campaign cycle or even next week to find out if something worked or didn't work - you'll know in a couple of minutes!

Email Marketing Campaigns require five key metrics to blow up your/B Testing process:

Test Status: If the test has been completed or is still in progress.

Hypothesis: Which variation will work the best for your customers and match their expectations, eg. different subject lines, from names, body copy, images, etc.

Test Metric: Impact of your A/B Test and areas of improvement.

Test Result: Find your results, all in one place without digging old emails.

Follow-up: Newer A/B Testing ideas based on your previous successful test.

Our platform automates all the above processes so that you don't have to do a single thing manually. With a single click, set your A/B Testing to spree.

LatticeAI makes running experiments so easy, you'll be done in a matter of minutes. Start by picking the campaign that needs A/B Testing and then choose whether to design your experiment or let our expert system do it for you!


What components of an email can be tested in the A/B test?

An A/B test is a great way to figure out what works for your email. You can choose which components of an email you want to change, such as the subject line, copywriting style, layout, images, CTA, and see how they're received by your recipients before deploying them in full scale.

Are subject lines an important aspect of A/B Testing?

Subject lines can make or break your email marketing campaigns and they should not be taken lightly. Your subject line should be somewhere between 61-70 characters, it should be short, crisp, and expressive.

Why Should You A/B Test Emails?

You want to make sure that your emails are effective, right? However, you're not 100% certain of what the most successful emails look like or how they work. That's why it can be helpful for us to A/B test them: we compare different versions and see which one people prefer more so then know which is better at getting their attention!

Which aspect of the email body can be evaluated with A/B email testing?

The content of your email, including the subject line and body text - its tone, length, and visuals can be evaluated with A/B testing.

The most important aspect to evaluate in an email is what you write about it - that's right, not how fancy or funny you make it seem but if people actually read all the way through!

curved line

Latest Articles

View All

Growing SMS subscribers via LatticeAI

SMS is one of the most reliable channels for marketing with over 90% open rates. LatticeAI provides SMS capabilities and makes it easy to grow your subscribers.

September 21, 2021

How ios 15 update will impact email marketing campaigns?

The updated Ios 15 had a major impact on email open rates which has hindered the ability to analyze email marketing.

July 20, 2021

Shopify abandoned cart recovery: How to boost sales and increase conversion rates by 40%?

How do you get your abandoned cart emails to actually convert? You need to make sure they are not just generic "here's what we have" messages. Make them personal and include the products they left in their carts! This blog post will teach you how to boost sales via abandonment cart emails.

June 17, 2021