Achieving success in digital marketing can sometimes feel like “a game of inches.” Whenever clients ask me for advice on what design changes to their website would work best for driving conversions, I often assess how we can strengthen our recommendations with A/B tests.
Experimentation is the perfect proving ground for bolstering a client’s bottom line. Regarding conversion rate optimization, our work as strategists ranges from improving user journeys toward a purchase to increasing competitive edge in paid and organic search engine results pages.
Conversion rate optimization tests can take on many forms. One version of A/B testing that doesn’t get talked about enough is multivariate testing. Whether this is the first time you’re learning about multivariate testing or trying to figure out when to use this experimentation method, this blog post is for you.
We’ll take a comprehensive look at what multivariate tests are, when to use them, multivariate test examples, and some ways you can start experiments of your own.
Multivariate Testing (MVT): A Definition
Multivariate testing (MVT) is a form of A/B testing in which numerous combinations of variants from sections of a website or app are used to determine which changes on a page will optimize conversions. Strategists build combinations by selecting variations to the page or interface to test.
Think of it as trying to find the right brand and amount of chocolate chips your cookies will need that earn the most praise among your friends and family. Maybe it’s one of three brands of chocolate chips that work best? Maybe it’s a cup of chocolate chips, or perhaps a cup and a half? Between three brands and two measurements of chocolate chips, there are six possible combinations of cookie recipes to test.
“How does this concept work in a website?” you might ask. For example, take a scenario in which we are running an MVT test on the call-to-action(CTA) banner from our Portent CRO page:
In the example above, I have split the CTA banner into two sections: Section A has an original and two versions of the copy for a total of three variants, and section B has an original and one version of the button’s copy for a total of two variants.
In a MVT of this CTA banner, our experiment would use every possible pairing of sections A and B for a total of six combinations to test.
A/B Testing vs. Multivariate Testing
Of the many ways A/B testing can be used, MVT takes your analysis a step further into determining what precise combination of changes provided the best lift in conversion rate. A traditional A/B test can involve a variation with one minor change or a completely different layout. Should that variation provide the largest conversion rate lift in that test, the results may not be able to indicate if the layout’s image, content, or combination was a winning factor.
This is where a MVT can identify whether or not precise changes or a combination of them was your best recipe for optimization success. The example above represents how four combinations of the chart and copy designs can be evaluated and help declare a clear test winner.
Why Is Multivariate Testing Important?
MVT is an essential tool to learn for CRO strategists and digital marketers. Consider these two reasons when gauging how impactful MVT can be in your optimization strategy.
Range
MVT is used in numerous marketing and design practices. From social media marketing to UX research, an MVT is a versatile way to gain insights into what parts of your website and customer experience are helping your business’s bottom line.
Results
MVT provides a way for CRO strategists and digital marketers to identify what design changes to a website or interface affect conversion rate performance the most. Unlike traditional A/B testing results, MVT has the potential to glean more comprehensive insights into the effectiveness of specific content and design combinations.
Benefits of Multivariate Testing
Conversion rate optimization can benefit greatly from a sound MVT strategy. Two benefits stand out as recurring themes from our experience and research in the field.
Interaction
Combinations of variants you introduce in a MVT will ultimately provide the best insight into what elements, visuals, content, etc., work best or interact with one another and influence the best conversion rate possible.
Efficiency
Running sequences of A/B tests to determine whether or not your variations would perform better than preceding iterations can prove inefficient. An MVT can run several variants as combinations concurrently in the same amount of time a single A/B test may have been planned.
Challenges of Multivariate Testing
MVT is not without some downsides worth knowing in CRO strategy. We recommend keeping two things in mind when approaching MVTs.
Sample Size
Adequate site traffic or visitors to your pages is a significant factor that could ultimately rule MVT out of your immediate CRO strategy. For example, in an A/B test, you would split 1,000 site visitors into groups of 500. A MVT would split site visitors further into smaller samples depending on how many combinations your variates create. Smaller traffic samples to your experiments would likely mean it would need a longer test time to reach statistical significance.
Sequence
One challenge a MVT can present that goes overlooked is how combinations of changes can work in sequential order. For example, suppose a test’s combination of a CTA banner’s copy and button text does not make sense or reads awkwardly for users. In that case, it’s probable that it would not convert as well as other combinations and thus be a wasted opportunity in the overall experiment.
MVTs take a lot of planning and consideration to avoid challenges. You can anticipate some meaningful MVT results if your CRO strategy is complete with a sufficient sample size and adequate variations that can impact your conversion rates.
When to Use Multivariate Testing
While not every opportunity will call for it, multivariate testing for CRO strategists will have a lasting impact on marketing strategies when planning perfect moments to run experiments.
Common Uses for Multivariate Testing
Most often, MVT on a website or landing page will occur when a design needs some refined changes. If your e-commerce site is looking for a combination of the best product detail layout and call to action, consider a MVT to find its most optimized arrangement for conversion rates.
The Nielsen Norman Group example provides a perfect case for when an A/B test should be prioritized over an MVT for larger-scale redesign experiments. MVTs will work best when they can determine the best combination of variables on a webpage or interface design.
When Not to Use Multivariate Testing
If time is of the essence and your sample size of sessions or users to evaluate is limited, it’s best to explore options or testing strategies that can meet the needs of the digital marketing strategy timeline. One alternative a team may explore is how to run A/B Tests on low-traffic sites.
Strategists should also avoid MVT for experiments that involve variations with numerous large-scale design changes to an experience. For example, CRO strategists do not use MVT to test three completely different page layouts with minimal changes. I would recommend reassessing the testing strategy using our guide to the risks and rewards of page layout experiments.
To reiterate from an earlier point, MVTs are best used when they use all combinations of variables in a design or experience.
Methods of Multivariate Testing
Conversion rate optimization uses MVT in several ways to glean the best insights in almost any scenario. While the ongoing debate over which method would be most effective, it’s important to identify which types of MVTs are commonly referenced in the statistics community.
Full Factorial
Full factorial MVTs are the most commonly executed and understood methods. Every variation’s combination is equally treated with the same amount of traffic and provides an insightful understanding of changes made in each. Because the number of variations and combinations can rapidly increase, full factorials can take a long time to run or require a larger sample size of traffic to have a chance at statistical significance.
Fractional Factorial
MVTs with a fractional factorial method are very challenging to conduct and most necessary for tests with dozens of possible combinations. Dozens of variations create a complex web of permutations that will require additional resources to analyze its results.
Fractional Factorial experiments take a lot less time to run than a full factorial. However, this is because it only runs a calculated sample of the possible combinations used in a test to determine the conversion rate of each one. Fractional factorial MVT can rely on more robust experimental software to conduct and calculate tests.
How to Run a Multivariate Test
Running a MVT will require careful planning to execute effectively and efficiently. To get you started, here is a brief checklist our CRO team uses for their experiments:
1. Assess your multivariate test approach.
This stage of planning an MVT requires considerable thought to accomplish; a who, what, where, when, and why. Identify what audiences you’re going to gather as test traffic, your conversion objectives, where on your website or flow to test, when and how long the experiment will take place, and why this test outcome will be important.
2. Outline all variations needed for the test.
Consider how many combinations your variants will produce based on the experience you’re testing. Depending on your page’s daily traffic, less than six combinations would be ideal, but no more than 10. Choose only the best design or content-based insights from analytics or user research.
3. Set up your Multivariate Test in your testing environment.
Our CRO team uses Google Optimize for a MVT environment. I’ve included links to a few other popular MVT tools and platforms below. Referencing our earlier example of the CRO Services page CTA Banner, you can see how we’ve staged our combination of variants in this image.
4. Run your test and analyze the results.
Results of your MVT can be found on your testing environment’s reporting platform, complete with calculated probabilities for a variant to be best and the modeled conversion rate for each combination.
Results like the one seen above in Google Optimize show that the combination of A1-B1 (Section A’s Variant 1 and Section B’s Variant 1) had a 34.10% conversion rate; an improvement of 8.48% from the original (A0-B0) baseline’s rate of 31.43%. A1-B1’s probability to be best of 51% indicates that it would be most likely to earn the highest conversion rate and might be the best combination to implement on the page.
Multivariate Testing Best Practices
Like all A/B testing, MVT is built in various ways but shares similar best practices that are paramount to follow when conducting conversion rate optimization.
Some Multivariate Testing Dos
- Determine your test’s one to two KPI goals or objectives to gauge success.
- Set a reasonable span of time for your test to run to collect enough traffic for every combination of variants.
- Monitor your MVT from time to time to ensure your data is reliable.
Some Multivariate Testing Don’ts
- Do not create too many combinations that split your traffic in several directions if your typical session count is limited.
- Do not test areas of your site that get very few interactions or engagement.
- Do not make any changes to your test variations during the test.
Multivariate Testing Tools & Resources
To learn more about MVT, we highly recommend getting familiar with some tools and resources before building your first MVT.
Tools and Platforms
The CRO strategist community relies on test calculators from sites like CXL, AB Testguide, and Dynamic Yield to plan their experiment durations using expected traffic and baseline conversion rate metrics. These same tools supplement results calculations from testing platforms like Google Optimize, Unbounce, and Optimizely.
Helpful Resources
The conversations on in-depth MVT strategy will be ongoing as digital marketing evolves. One of the best resources for understanding MVT comes from Google Analytics Academy’s YouTube playlist of Google Optimize features. Because Google Optimize’s testing platform provides a seamless connection to Google Analytics, their Resource Hub is a great place to supplement an understanding of MVT.
Wrap Up
Of the different kinds of CRO tests strategists use in their arsenal, MVT is one of the least understood but potentially important methods to refine designs in a user’s journey across a website or app. Using a combination of variants in an MVT method allows for precise insights to be gleaned across numerous experiences and interfaces websites rely on to improve their business’s bottom line.
We hope you found this guide a comprehensive and excellent starting point for your journey into MVT. For more information on how Portent’s CRO team can help your optimization efforts, check out our Services page or hit us up on our Contact page!
