Speed up Your Optimizely Testing with X-Optimizely-CLI

Go Back

Here at roboboogie we have been developing tests using Optimizely for over four years. When we started, Optimizely only had one version of their service, now known as Optimizely Classic. When developing on Classic, we wanted to integrate a tool to improve our development process instead of using the limited and somewhat cramped Optimizely UI (not knocking Optimizely, it’s a problem with trying to develop within the confines of a browser window). Once we found Funnel Envy’s OptCLI, an open source command line interface(CLI) tool for Optimizely, we knew this would completely change the way we developed tests. OptCLI sped up our development process by over 40% and as much as 60% for templatized experiences. It allowed easy local hosting with the ability to push experiments and variations to Optimizely without the need to be logged in to the service. When we considered moving to Optimizely X, even with the additional features, such as expanded custom attributes and ‘always on’ visitor tracking, we hesitated because we had to give up our development flow. OptCLI had become a fundamental part of our process.

How we accidentally built a fully functional CLI:

We made the jump several months after Optimizely X launched because we needed more freedom with snippets for overlapping tests, which Optimizely X provided. After a thorough search, we discovered a few CLI tools that were in the works to integrate with Optimizely X, but nothing that was fully functional. Those that existed lacked the features we desired for our development process. So, in the early days of a slow, cold January 2018, the roboboogie development team made some API calls from our terminals to test data complexity. We dug into the data the API returned to determine how difficult it would be to build out a minimum viable product (MVP) with the ability to push code. We wanted a tool we could use internally to keep us in our development environments, save us time copying/pasting to Optimizely X over and over again and keep more consistent version control by allowing us to push and pull experiments.

We set out to build a CLI that would push our code to Optimizely X. By pushing code rather than copying/pasting, our team is much more likely to make all our changes in our code editors, where we have version control and linting programs, that will error-check our code. These are very important tools, as they help keep all tests up to date and errors to a minimum. Our original plan was so simple we weren’t even sure we were going to release it. But throughout our coding cycle, as we discovered new opportunities and unlocked new efficiencies in the CLI, we knew we were constructing a tool that could help the development community at large build faster, more precise tests.

When we started wrapping up the build process, we had exceeded our own expectations. Not only did we have the ability to push and pull experiments, but almost all of the features we hoped would come much later. We couldn’t help ourselves once we dove in, so we started adding commands like pull all pages info and the ability to sync metrics to any experiment.

X-Optimizely-CLI is designed to uplevel any test development process. You can now push and pull experiments from Optimizely X to keep environments in sync and so much more. Using the hosting option you can easily switch between variations for testing, without the need to keep hitting the preview button or hunting down new query parameters to host different variations. Also, you can organize and push metrics in common groups, so you don’t have to add common metrics manually to every test, like a group of funnel metrics.

X-Optimizely-CLI can optimize your development process:

Below is a highlighted list of available commands available through the tool.

Set token for API calls: Creates a hidden directory labeled .optclix in your current working directory and stores your API token for use with all future API calls. Attempting to run any other command will prompt you to create a token as well. You can find all that info here on Optimizely.com.

Initialize a new Optimizely X project locally: Initializes a new Optimizely X project in the directory you are currently in, creating a project.json file with the project_id. This will hold your experiments.

Create a local experiment: Creates a local experiment with an “Original” variation that has a starting weight or traffic allocation of 100%. Optimizely uses a scale of 10000, so that’s what you will see in original/variation.json. Page ids are attached to all variations using experiment.json.

Create a local variation: Creates a local variation and builds the necessary source files. Setting traffic allocation here will necessitate an update to the other variations so that total traffic allocation adds up to 10000 before pushing to Optimizely (we thought adding this manually would be easier than having a prompt asking to add things up to 10000 every time a new variation was created).

Push experiment to Optimizely X: Pushes and/or updates an experiment and all variation to your Optimizely project. This will overwrite what is on Optimizely for that experiment ID. The -m flag can be used to send metrics with your experiment!

Pull experiment: Pulls a specific experiment from Optimizely to a specific directory of your choosing, allowing you to sync your experiment from Optimizely X with your local project folder. Like push-experiment the -m flag can be used to send metrics with your experiment.

Pull all pages for a project: Pulls all pages for a project into a single JSON(JavaScript Object Notation) file. This is mostly a reference file for page and metric IDs. This makes it much easier to add pages to new experiments and push metrics in groups.

Host variation locally: Injects the CSS and JS of the variation you choose using your preferred extension. This gives you the power to work locally without needing to wait for Optimizely to update any code.

Ready to start using X-Optimizely-CLI?

Below are the details on where you can find it and how to set it up for your organization:

Installation options:

Our recommended option for installation is through NPM. It is the quickest and easiest way to to get up and running. Never used NPM before? Check out these links for more information: what is NPM and how to install NPM.

If you already have Node installed, you can simply run the following command to install X-Optimizely-CLI directly: npm i -g @teamroboboogie/x-optimizely-cli. If you see an error, you might have to use sudo npm i -g @teamroboboogie/x-optimizely-cli. This means there is a permissions issue.

Note: If you are not sure if you have node installed, you can check by typing node -v into your terminal. If it comes back with a version then you can use npm. If not you can use brew, ”the missing package manager for macOS,” and install node with the brew install node command or head over to nodejs.org and download a version of node.

If you prefer to install it manually the tool is also available on GitHub. GitHub also provides an opportunity to preview the code, ask questions, make suggestions or submit pull requests to help us continually evolve the tool.

Troubleshooting issues related to local hosting vs. Optimizely hosting:

Using a script injector like Tampermonkey vs loading through Optimizely will load the script after the page has loaded. This can cause two issues.

1. You might run into severe flashing during local development. This is due to using a script injector like Tampermonkey that has to load the script after the page has loaded. Optimizely X should eliminate the flashing as the Optimizely snippet is loaded at the top of the head. Therefore, do not worry and continue to build believing that Optimizely will take care of that issue.
2. If you are developing on a Single Page Application such as Angular, you will probably notice that everything looks fine locally. But upon testing through Optimizely, everything looks half done, meaning some parts of your code my have been overwritten and the page is only showing some of your changes. This is due to the state changes forcing the reloading of elements and is common for SPAs. See this article by roboboogie developer, Darius Capizzi, to help overcome the pain points of A/B testing on SPAs.

Life is too short to copy/paste

At roboboogie, development is an ever evolving process. We see X-Optimizely-CLI as following suit and we are excited for it to grow with us and the testing community’s needs. We encourage conversation and feedback to help improve the tool via GitHub, or drop us a line at development@teamroboogie.com, and of course feel free share this with anyone in your network who would find it useful. We hope X-Optimizely-CLI improves your test development workflow, and unlocks new opportunities for your organization.

Happy testing!

-The roboboogie Development Team