Month: June 2018

Curious about the ever-evolving world of A/B testing and conversion rate optimization? Lead Developer, Jeremy Sell, is here to answer some frequently asked questions on the subject. Have even more questions? Drop us a line!

What is A/B testing, and what can it help me achieve?

In development, people often refer to A/B testing synonymously with the whole process of testing. But the definition is much more narrow. A/B testing is defined as comparing two versions of a web page or app against one another to determine which one moves the needle on a predetermined metric which is usually a conversion rate. Whether you are a lead marketer or a CTO, you are always looking to increase your ROI. Testing helps you get there by helping you improve your key metrics and create a better user experience across your site.

How is multivariate testing different than A/B testing?

As discussed above, A/B testing is comparing two components against one another. In multivariate testing, we might be comparing multiple pieces of a web page against each other. Making multiple changes requires that we isolate the changes so that one winning change does not lift up another change that is either neutral or negative on conversion. Therefore, when running a multivariate test we might be testing two new headlines, adding similar products to a page, and creating a space for reviews to be more visible. In this case, running A/B tests could take months to collect enough data to pass the first round. Then we would have to test how they all play together. Multivariate testing instead puts all of the changes together in all possible variations and releases those simultaneously. The math in this case would be  [# of changes for component 1] x [# of changes for component 2] x [# of changes for component 3] .

• Two new headlines + original headline = 3
• Adding similar products + original without = 2
• More visible reviews + original = 2
• 3 x 2 x 2 = 12 variations

When is A/B testing a good idea? When is it a bad idea?

Good Idea: When you have a clear hypothesis with with measurable metrics and traffic high enough to reach statistical significance. Testing can almost always make your user paths cleaner and more clear.

Bad Idea: There are a lot of times testing can be not only bad, but actually harmful to your site. This can be the case when declaring winners before proving statistical significance. Another trap is choosing to increase traffic during a test to prove statistical significance quicker. This leads to unusual traffic and can produce false positives. To me, the worst cases involve not being able to implement wins after tests produce positive results. 

What testing platform should I use?

This depends wildly on budget and the other technologies you rely on. I also wrote a whole blog post on this!

How many variables can there be in one test?

How much time do you have? If you have multiple variables then you will have to run multiple variations. This number increases quickly if each of those variables have multiple variations. See the answer on multivariate testing for more on that math.

How many tests can I run at a time?

This is a tricky question. Let us say the testing is on an e-commerce site with 3 tests on different sections of the site, and the main metric for all tests is revenue increase. In this case each test is affecting the hypothesis for the other tests, which is increased revenue. In this case you may want to run mutually exclusive tests to avoid data pollution. Then again, running mutually exclusive tests means you will wait longer to predict a winner. Sometimes it is more important to prove out winners and move the needle on conversion rates than it is in being 100% accurate on all the tests individually. This varies by situation wildly and how quickly results are needed.

How do I set up tracking to make sure I’m collecting valid and meaningful data?

For basic usage, the built in tracking is fine, but should be tested to make sure the analytics are flowing in correctly. For more robust analytics I would recommend using Google or Adobe. All testing platforms choose winners based upon Bayesian Theory or something close, and having an outside source where you can track user behavior can be invaluable. What happens when your test fails? Can you dig in and look at the users in variation 3 to see what other paths they might be taking in your experience? Google and Adobe offer this and the ability to break down your data to really understand your experiment, whether winning or losing.

How long should I run tests, and when can a winner be declared?

You need to reach statistical significance. There are calculators for this here, here, and here!

What is ‘statistical significance’?

This is the likelihood that the difference is significant enough to overcome any randomness between the baseline and variations. For any test to be declared a winner, it needs to overcome a percentage that could be attributed to random differences. This number gets smaller based upon the number of users within an experiment.

What if a test performs poorly?

Dig into the analytics and learn! Losing tests can teach you about what users are looking for. We had a test where we built out a signup form for a college within a larger page that had links to informational pages. The form lost in conversion to the original that was on a stand alone page. What we found was users who completed the form were more likely to have looked through the informational pages and became more qualified leads for the sales team.

What should I do with a winning test?

Implement it! I wrote a whole blog post on this one, as well.

How do I encourage my team to get onboard with A/B testing?

Each department can be encouraged in different ways. Mainly, testing gives you an opportunity to better understand your users needs on a regular basis and offer a product that aligns more closely with their interests in what you are offering.

Written by Jeremy Sell, Senior Experience Engineer

Here at roboboogie we have been developing tests using Optimizely for over four years. When we started, Optimizely only had one version of their service, now known as Optimizely Classic. When developing on Classic, we wanted to integrate a tool to improve our development process instead of using the limited and somewhat cramped Optimizely UI (not knocking Optimizely, it’s a problem with trying to develop within the confines of a browser window). Once we found Funnel Envy’s OptCLI, an open source command line interface(CLI) tool for Optimizely, we knew this would completely change the way we developed tests. OptCLI sped up our development process by over 40% and as much as 60% for templatized experiences. It allowed easy local hosting with the ability to push experiments and variations to Optimizely without the need to be logged in to the service. When we considered moving to Optimizely X, even with the additional features, such as expanded custom attributes and ‘always on’ visitor tracking, we hesitated because we had to give up our development flow. OptCLI had become a fundamental part of our process.

How we accidentally built a fully functional CLI:

We made the jump several months after Optimizely X launched because we needed more freedom with snippets for overlapping tests, which Optimizely X provided. After a thorough search, we discovered a few CLI tools that were in the works to integrate with Optimizely X, but nothing that was fully functional. Those that existed lacked the features we desired for our development process. So, in the early days of a slow, cold January 2018, the roboboogie development team made some API calls from our terminals to test data complexity. We dug into the data the API returned to determine how difficult it would be to build out a minimum viable product (MVP) with the ability to push code. We wanted a tool we could use internally to keep us in our development environments, save us time copying/pasting to Optimizely X over and over again and keep more consistent version control by allowing us to push and pull experiments.

We set out to build a CLI that would push our code to Optimizely X. By pushing code rather than copying/pasting, our team is much more likely to make all our changes in our code editors, where we have version control and linting programs, that will error-check our code. These are very important tools, as they help keep all tests up to date and errors to a minimum. Our original plan was so simple we weren’t even sure we were going to release it. But throughout our coding cycle, as we discovered new opportunities and unlocked new efficiencies in the CLI, we knew we were constructing a tool that could help the development community at large build faster, more precise tests.

When we started wrapping up the build process, we had exceeded our own expectations. Not only did we have the ability to push and pull experiments, but almost all of the features we hoped would come much later. We couldn’t help ourselves once we dove in, so we started adding commands like pull all pages info and the ability to sync metrics to any experiment.

X-Optimizely-CLI is designed to uplevel any test development process. You can now push and pull experiments from Optimizely X to keep environments in sync and so much more. Using the hosting option you can easily switch between variations for testing, without the need to keep hitting the preview button or hunting down new query parameters to host different variations. Also, you can organize and push metrics in common groups, so you don’t have to add common metrics manually to every test, like a group of funnel metrics.

X-Optimizely-CLI can optimize your development process:

Below is a highlighted list of available commands available through the tool.

Set token for API calls: Creates a hidden directory labeled .optclix in your current working directory and stores your API token for use with all future API calls. Attempting to run any other command will prompt you to create a token as well. You can find all that info here on Optimizely.com.

Initialize a new Optimizely X project locally: Initializes a new Optimizely X project in the directory you are currently in, creating a project.json file with the project_id. This will hold your experiments.

Create a local experiment: Creates a local experiment with an “Original” variation that has a starting weight or traffic allocation of 100%. Optimizely uses a scale of 10000, so that’s what you will see in original/variation.json. Page ids are attached to all variations using experiment.json.

Create a local variation: Creates a local variation and builds the necessary source files. Setting traffic allocation here will necessitate an update to the other variations so that total traffic allocation adds up to 10000 before pushing to Optimizely (we thought adding this manually would be easier than having a prompt asking to add things up to 10000 every time a new variation was created).

Push experiment to Optimizely X: Pushes and/or updates an experiment and all variation to your Optimizely project. This will overwrite what is on Optimizely for that experiment ID. The -m flag can be used to send metrics with your experiment!

Pull experiment: Pulls a specific experiment from Optimizely to a specific directory of your choosing, allowing you to sync your experiment from Optimizely X with your local project folder. Like push-experiment the -m flag can be used to send metrics with your experiment.

Pull all pages for a project: Pulls all pages for a project into a single JSON(JavaScript Object Notation) file. This is mostly a reference file for page and metric IDs. This makes it much easier to add pages to new experiments and push metrics in groups.

Host variation locally: Injects the CSS and JS of the variation you choose using your preferred extension. This gives you the power to work locally without needing to wait for Optimizely to update any code.

Ready to start using X-Optimizely-CLI?

Below are the details on where you can find it and how to set it up for your organization:

Installation options:

Our recommended option for installation is through NPM. It is the quickest and easiest way to to get up and running. Never used NPM before? Check out these links for more information: what is NPM and how to install NPM.

If you already have Node installed, you can simply run the following command to install X-Optimizely-CLI directly: npm i -g @teamroboboogie/x-optimizely-cli. If you see an error, you might have to use sudo npm i -g @teamroboboogie/x-optimizely-cli. This means there is a permissions issue.

Note: If you are not sure if you have node installed, you can check by typing node -v into your terminal. If it comes back with a version then you can use npm. If not you can use brew, ”the missing package manager for macOS,” and install node with the brew install node command or head over to nodejs.org and download a version of node.

If you prefer to install it manually the tool is also available on GitHub. GitHub also provides an opportunity to preview the code, ask questions, make suggestions or submit pull requests to help us continually evolve the tool.

Troubleshooting issues related to local hosting vs. Optimizely hosting:

Using a script injector like Tampermonkey vs loading through Optimizely will load the script after the page has loaded. This can cause two issues.

1. You might run into severe flashing during local development. This is due to using a script injector like Tampermonkey that has to load the script after the page has loaded. Optimizely X should eliminate the flashing as the Optimizely snippet is loaded at the top of the head. Therefore, do not worry and continue to build believing that Optimizely will take care of that issue.
2. If you are developing on a Single Page Application such as Angular, you will probably notice that everything looks fine locally. But upon testing through Optimizely, everything looks half done, meaning some parts of your code my have been overwritten and the page is only showing some of your changes. This is due to the state changes forcing the reloading of elements and is common for SPAs. See this article by roboboogie developer, Darius Capizzi, to help overcome the pain points of A/B testing on SPAs.

Life is too short to copy/paste

At roboboogie, development is an ever evolving process. We see X-Optimizely-CLI as following suit and we are excited for it to grow with us and the testing community’s needs. We encourage conversation and feedback to help improve the tool via GitHub, or drop us a line at development@teamroboogie.com, and of course feel free share this with anyone in your network who would find it useful. We hope X-Optimizely-CLI improves your test development workflow, and unlocks new opportunities for your organization.

Happy testing!

-The roboboogie Development Team