Customer Experience Digital Optimization

What and Why of A/B and Multivariate (MVT) Tests | Key Differences

After 4 months of consistent efforts invested in UX designs, prototypes, visual design, CMS, various integrations etc., you finally have re-launched the seemingly great website. As a marketing head, you have guided your team that had invested hours in analyzing the usability, deep diving the web analytics data, compliance with designing standards, SEO and UX conventions.

2 weeks down the line the numbers do not improve. In fact, on deeper analysis, you discover that some sections that went through an aggressive redesign process were generating far more leads before the redesign.

You are perplexed!

Optimization of digital experience is one of the key areas that a lot of companies either miss out on or end up under doing it. In both the scenarios, the end product fails to match up.

A/B and MVT (Multivariate) is one of the key aspects of digital experience optimization.

A/B testing is a statistical hypothesis testing which includes 2 or more variants of a web page. These variants are referred to as experiments.

You set up these experiments and then split that traffic to these pages. In most cases, these split is in equal proportions. So it’s either 50:50 (Two variants), 33:33:33 (Three variants), or 25:25:25:25 (Four variants).

The default version is usually referred as the Control and the others (Challengers) are called as Treatment. The tests are run for a specified period of time after which the winning experience is retained.

Few Use Cases (A/B Tests):

1)      Changing the button color in the Hero area. (Red vs Green)

2)      Changing the form. (Single page v/s Multipage forms)

3)      Changing the CTA placements (Top Navigation vs Hero area)

4)      Changing media types (Static vs Video)


Multivariate tests focuses on testing combinations of various elements within a web page. So while A/B tests focus on the variation of a page and find a winning page among the lot, a MVT (Multivariate) test focuses on the combinations and tries to answer the following question.

–       How the different combination of elements impact the conversions.

–       How each element impact the conversion.

  Example of a Multivariate Test (MVT) : 

–       Variation 1 : Red CTA + Image 1

–       Variation 2 : Green CTA + Image 2

–       Variation 3 : Red CTA + Image 2

–       Variation 4 : Green CTA + Image 1


 Key Differences between A/B and Multivariate (MVT) tests.

–       Traffic: MVT tests require a much higher traffic than A/B tests.

–       A/B tests evaluates 2 or more versions of a web page. MVT tests evaluate combinations of various elements within a web page.

–       Subsequently, A/B tests are easier the set up while MVT tests are a bit more complex.

–       MVT tests have a longer time frame than A/B tests.

Leave a Reply

Your email address will not be published. Required fields are marked *