Go Back
There are a lot of multivariate and A/B testing tools in the marketplace. Which one fits your company’s needs best depends on a myriad of factors, including, but not limited to: budget, number of visitors to your site, how you compile your analytics, and where you need to test. As an agency with diverse clients, who all have different customers and integrate their analytics in different ways, it’s our practice to stay platform-agnostic as much as possible. We do this by continually reviewing the top tools in the marketplace internally before recommending anything to our clients.
Our most recent review uncovered some exciting trends in the field. The floor of testing platforms has been elevated substantially, while the flagship enterprises seem to be making opinionated moves to distance themselves in very substantial ways. (Note: In the mirror weeks since conducting our audit, we are seeing some of these platforms respond to those opinionated moves by building out similar tools and closing those gaps once again.)
Experimentation tools
In our context at roboboogie, saying we are testing platform agnostic is saying we want to use whatever tool helps our customers test and improve their impact on their industries the most. Remember, it’s not about a tool, but about the impact we can have on our client’s customer base. Because, happy customers convert.
This list is not a list of which tool is the best. What this list hopes to be is a short, but detailed, description of some of the most powerful experimentation tools in the space with a touch of the best ways to use them for your testing purposes. We will be taking a look at Optimizely, Visual Website Optimizer (VWO), Convert, Adobe Target, and Google Optimize & Optimize 360.
To start, I will say they all claim to have ~99.99% uptime, some type of flicker-free testing, and their version of a “What You See Is What You Get” (WYSIWYG) drag and drop editor.
Here’s some information on the most common statistical models used for testing analytics, and the two we see from these five testing platforms. The Bayesian Statistical Model is the most common, but Convert and Optimziely believe the Frequentist Inference Model has an important place in moving along experiments quickly. Here’s an article on Bayesian vs. Frequentist statistics from Optimizely and this xkcd comic is just too funny.
Optimizely
Optimizely is the optimization platform where you can test “across every device, every channel, every customer touchpoint.” You can test across front-end, back-end, mobile apps, Xbox, and even a tvOS. They are the leading A/B testing tool in the world – 26 of the Fortune 100 test with Optimizely, and they have achieved this status by creating server-side and client-side testing unified on a single experimentation platform.
In Classic, draft and paused experiments were included in the snippet, creating extra load, but in X, draft and paused experiments are not included in the snippet making the load considerably smaller. By offering custom snippet creation, it allows teams to build snippet payloads that include only experiences that are relevant to the page a visitor is on. This requires adding more snippets but keeps experiments running leaner and faster. X now includes individual CSS and JS files (Yay!). Through the new pages process, X is able to offer always-on visitor tracking, so no more wishing you had added a pageview or click goal. If you have it set up, you can add it to an experiment at any time and the data will backfill. Through pages there is also page-level conditional activation for Single Page Applications (SPAs). Optimizely uses a sequential testing statistical model (using Bayesian theory).
Program Management, Optimizely’s newest product, is designed to help scale your experimentation cycle across your enterprise. The Program Management tool is exactly what it sounds like: it’s a place where you can link hypothesis, experiments, new ideas that have a rating system, development cycles, stats (with annotation options), and conversations all in one place. This keeps your team from duplicating information across platforms, documents, and teams.
Visual Website Optimizer (VWO)
VWO turned from an A/B testing platform to a CRO Platform, but they still currently offer both options. Their optimization platform, now called “Connected CRO Platform,” makes it possible to utilize several CRO tools from the same platform – including ideation, design, tracking, analysis and insights, heat maps, recordings, documentation, and A/B testing. They are the ideal platform for bringing together everything under your conversion optimization program, from visitor research to A/B testing, all in one place. They claim to be the “World’s Easiest A/B and Split Testing Software,” and I believe them. Everything about their platform is intuitive and easy to use. They include individual JS & CSS files that are easily reached and they have the smoothest UX design of the tools we tested. Paras Chopra from VWO said, “Our focus with Visual Website Optimizer is to make A/B and multivariate testing easier by taking technical hassles out of the picture. In fact, ideally, we want our users never to touch any code (be it HTML, JavaScript, CSS or PHP).” (https://conversionxl.com/blog/server-side-vs-client-side-ab-testing-tools-whats-the-difference/)
The main difference between VWO and Optimizely isVWO is looking to optimize your optimization, and they do this in a few different ways. First, their preview is cookie-based rather than URL-based, making cross-site testing much easier. Testers previously had to append a query parameter to every page or set the experiment live to a cookied audience. They were one of the first to move forward with this method, but now it’s the industry standard. Their platform automatically takes screenshots of each of your variations on save, so you can use it as a quick preview to see how your changes are coming along. VWO uses the Bayesian statistical model and will load your snippet asynchronously so your site isn’t slowed. They achieve flicker-free testing by making your page hidden until it is loaded.
Convert
Convert’s About Us page says, “we’re always optimizing for a better product, a better planet, and a better way to work.” They rebranded in a radical way in the past couple of months and are taking a positive, caring, culturally-aware approach to show that they want what’s the best for your business. They are working toward gender equality in the workplace and being carbon neutral 100X over. This is a huge shift from the way they used to describe themselves, as “affordable enterprise testing that includes personalization.” For what it’s worth, I believe both descriptions still hold true. They are probably the least expensive of the 5 when you compare the number of visitors and domains allowed on one account (caveat: after you surpass about 300k visitors a month).
Convert uses a JavaScript snippet to load their tests. It loads all active, paused, and completed tests in the project folder. Archiving tests not running will slim the snippet. Their editor tool works similarly to Optimizely Classic, but has a bunch of added benefits, like separate CSS files for each variation and personalization elements built-in. Their audience targeting system natively includes a slew of options, including GEO-IP targeting and customer list, which are usually paid features. Convert uses the frequentist statistical model with 2-tailed z-testing analysis. They are also actively building their product and making meaningful changes to make things better, like adding 18 more integrations planned to release before the end of Q2.
Convert’s customer service is consistently available through email, chat, and phone. They are quick to respond, and actually seem happy to help out! They have a small company feel, but still put out an enterprise-level product. They include all of their features in all plans except their most affordable, which only loses a few small features. You really need to be testing at least 300K visitors a month, across all clients and platforms, for their pricing structure to make sense.
Adobe Target
Adobe Target is a pioneer in the space and they are all about channels, channels, and channels! We interact with brands, not just on one website, but through all of our channels. By having all your channels testable across all devices, Adobe enables you to target a user through an ad and then personalize everything else they see across your media. The name “Target” really speaks to the goal of their product. Adobe wants you to create and test personalized experiences served to highly targeted users or groups across all mediums. They want the whole conversion funnel to be a place you can create personalized experiences for your user.
The tool feels like it was designed more for marketers than developers, but the beauty of that is anyone can use it in a sophisticated manner. With this approach, there is no need for a marketer to create a ticket for the developers to build a test. Adobe is moving past analytics to what they call “digital intelligence” which is, “in today’s mobile, multi-device, and multi-channel world … the ability to transform digital data into real-time, actionable, customer-centric insights.” Another cool feature Adobe Target uses is real-time statistics so you can tweak your experiences across all channels whenever you see a change in data.
Google Optimize & Google Optimize 360
Google Optimize is a free A/B testing platform that is the basis for Google’s premium service. Optimize does not include the G-Suite integration that 360 does, but is a great option for smaller companies wanting to get started with testing. In Optimize, you can use their WYSIWYG editor, add CSS and JavaScript, as well as target your audiences by region, city, query parameter, and a bunch more options, all for free. Optimize loads through an extension, and as such, it feels a little clunky to me. One example is because it’s loaded with a plugin, every time you need to switch between variations it will reload the whole page and then apply the extension. The JavaScript editor is also hard to find; I was reviewing their product after not using it for a month and had already forgotten how to find it. Luckily, Optimize uses separate CSS and JS files to code in, so you don’t have to add any tricks to separate variations. This version of their tool uses a JavaScript snippet to load your experiments on the front-end.
Google Optimize 360 offers enterprise-level website testing and personalization using your analytics audiences. No need to recreate those audiences within another platform, as Optimize 360 offers native integration with Google Analytics 360 (GA). With Optimize 360, you can also leverage the segments/audiences you have already built in GA to create personalized experiences across your site. This allows you to test into segments natively that would otherwise be 1) not be possible, or 2) would take a lot of setup time to initiate. Also, the data that you get back can be easily parsed in GA, offering you the power to dig into your data in a much deeper way than most other testing platforms. Google uses session-based targeting, meaning a user who ends a session may see a different experience the next time they arrive on your site. This being said, Optimize 360 combines a solid, basic, A/B-testing platform with seamless analysis within Google Analytics 360.
Choose the best tool for your needs
Optimizely has my vote for the best overall A/B platform as it integrates front-end, back-end, program management, analytics, and a bunch of other useful tools. However, one thing to take into account is their pricing structure. It creates a steep yearly buy-in wherein each tool adds another line to your bill, but if you need the power Optimizely offers, they are a great choice.
If back-end testing is not necessary, but you want a tool that includes a place to manage your experimentation, then VWO is the CRO platform for you. They have plans starting at 10K visitors per month, so VWO can help small businesses at a fair price, but they also offer plans that can support the largest enterprise customers. They make the job of testing easy and keep you organized.
Convert is still the underdog here, including all their tools in each plan without a pricey upgrade to their services. If you have multiple clients or your company has more than 300k visitors a month, then Convert creates a flexible product with a ton of tools available at the lowest cost per visitor at this threshold. Plus, they are trying to create a positive impact on the world around us which always gains respect from our team.
If you’re invested in Adobe Analytics Cloud, then having the ability to test across all your channels in one place makes a lot of sense. Adobe Target will make that process seamless and leverage the data in the most powerful way.
If you use the Google Suite of products like Audience, Attribution, and Analytics, then utilizing a simpler tool like Google Optimize to gain the functionality of Google Audiences will open testing avenues that a standalone tool cannot give you out of the box. Also, if you want to dig into your testing data in a native and straightforward fashion, Google Optimize 360 is one of the strongest on the market.
In the end, the marketplace for A/B testing is huge and one tool cannot meet every customer’s needs. Keep this in mind, and choose the platform that best suits your needs for each account. Good luck and happy testing!
Written by Jeremy Sell, Senior Experience Engineer
