A/B Testing

Introduction:

A/B testing is a research method used to compare two versions of a web page or mobile app to determine which variation of the UI results in better performance. It is a popular method that allows individuals and companies to optimize user experience and make improvements to their application.

Steps/How To:

Identify Changes: Identify any potential problem areas within the UI and create a hypothesis of how to fix it

Create Variation: Create two versions of the applications UI to run the experiment on, create the new design based on the hypothesis, whether it be changing the font, colors, or re-designing the whole page.
Note: These changes can be determined through website analytics tools to identify areas with the most engagement vs the least, as well as heatmap tools to identify any areas in need of improvement. Create a hypothesis or an idea that you think will help improve the UI, this will be your second version.

• The original page is known as the control.
• The altered page is known as the variation.

Running The Experiment: Randomly filter the traffic of the application where one group (Group A) sees the original, or controlled version, and the other group (Group B) sees the modified version.

Collect Data: Record and measure different metrics such as user session duration or other performance metrics related to the application.

Analyze Results: Based on the results, determine which version improves performance and deploy the winning variation. The experiment can be repeated to make further improvements.

• Determining which version results in an improvement involves a combination of statistical analysis and pre-defined success metrics defined by the company.
  • The defined success metrics are goals that align with the purpose of your website (i.e. user engagement, session duration, etc.)
  • The statistical analysis is used to determine if the data observed is statistically significant. This involves calculating p-values, Bayesian statistics, or using other statistical models to determine if the results are reliable.

Types of A/B Testing:

Split URL Testing:

• Split URL testing is a variation of A/B testing where two different web page urls are used, rather than running the experiment on the same url. It is typically used for testing completely new page designs.
  
• Cons: Managing and maintaining different URLs and variations can be resource-intensive, especially for websites with extensive content.

Multivariate Testing:

• Multivariate testing tests various changes to different page elements simultaneously to analyze which combinations perform the best. Can be used to save time rather than performing multiple A/B experiments.
  
  • Cons: Testing multiple combinations requires larger sample sizes to achieve statistical significance, which might be challenging for smaller websites with limited traffic.

Multi-Page Testing:

• Multi-Page testing tests changes to an element on the web page over multiple pages within the website. Allows for a consistent set of pages when testing variation with control.
  
  • Cons: Complex implementation that may make it difficult to set up and manage multi-page tests.

Benefits of A/B Testing:

A/B Testing is useful in helping identify any potential issues with the webpage design before releasing it to the public which can mitigate risks in practice. It also,

• Optimizes user experience which allows you to create a more effective application.
• Can provide insights into user behavior which can help with product development.
• Is a quick and efficient way to test UI changes.

Alternatives to A/B Testing:

There are some situations in which A/B testing is not very useful. For example, when you don’t have enough users on your website, the sample size of the experiment may be too small to see significant or reliable results. Or another case is if the changes to the UI are minor, it may not be worth the resources to run the whole experiment. Instead A/B testing should be used on changing features that are a vital part of your web page or features that are a main part of the flow of your website.

Some alternatives to A/B testing:

Private Beta Testing: Private betas for users offer a pre-released version to a selected group of people for testing and feedback purposes. Instead of making changes to the website solely based on data collected from an experiment, Beta testing uses feedback directly from users. Feedback can include feature requests, suggestions on improvements, or overall user experience. After improvements are made the changes are launched to the public.

User Surveys: User surveys are a refined set of question used to gather feedback and insights from users regarding user experience. These surveys offer a direct way to gain insights on user preferences and allows you to find any problem areas. This offers a user centric approach when making changes to UI.

Additional Resources:

More Information: https://vwo.com/ab-testing/
Examples: https://vwo.com/blog/ab-testing-examples/