How do you run an A/B test?

There are many ways for a product manager to conduct an A/B test. One useful example is the approach offered by Product School, in which A/B testing follows a five-stage process:

Stage 1: Determine the data you’ll be able to capture.

First, determine what types of information you’ll be able to collect and analyze, before building your experiment and running the tests. If you skip this step, you might waste time and resources developing an experiment where you can’t accurately measure the results.

Stage 2: Develop your hypothesis.

Based on the data you know your team will have available, you’ll now want to identify the opportunities for your experiment and formulate a theory about how users will react to a specific element of your product.

For example, you might assume that users will want the steps required to complete a task using your new feature to be ordered in a particular sequence. That’s your hypothesis.

Stage 3: Build your experiment.

Now you’ll want to develop the details of your test. This will have your team create a variant of your planned feature — using, for example, the same functionality but with the steps sequenced differently.

During this stage, you’ll also need to generate the different segments of your user base that will receive the variants of your new feature. You’ll also want to define the metrics you’re going to measure. Will you gauge user preference of one variant based on surveys after your users have had a chance to engage with the product? Or will you base it instead on monitoring usage data and, if so, how will you determine users’ preferred method?

Stage 4: Run your test.

Now it is time to send the different versions of your new feature out to your various user segments and wait to see how the groups respond to each version.

Your team will need to determine for yourselves how long to run your A/B test, how much data to collect, etc. — because this will vary for each company and because you want to gather and analyze enough data that you know you’re working with a statistically significant representative sample of your user base.

Stage 5: Measure your results.

Finally, you will review the data you’ve collected from your A/B test and make a determination about which of the two features (or layouts, or color-schemes, or whatever you are testing) earned the most positive response or the greatest degree of engagement from your users.

Here are some resources to learn more about A/B testing: