Don’t let the sun set on your Digital Optimization Program

Google is dropping its A/B testing product, Optimize, in September, and if you don’t have a plan to replace it yet, you might be left in the dark.

  1. The stakes: Budgets for innovation are lean, this is a hard time to get buy-in for a big tech investment with a recurring cost. You could lose your whole program budget if you can’t find a replacement, but the hard part is selecting a tool that ensures the future of the program and getting the budget for it.

  1. The pitch: You have a plan that will align your optimization strategy with revenue and cost savings and show positive ROI on the program quickly. You just need to get the budget approved to get this tool.

  1. The choice: Now you have to make sure you choose the right tool that will future-proof your program and activate your plan. You need to get it up and running quickly so you can start gaining momentum and getting the return.

  1. The promise: You need to get test velocity and wins, so that you are making good on the promise you have in this investment.

  1. The proof: You need to show the ROI in a short period of time and get the program in the black, this requires a measurement model and reporting that you can share with leadership that makes certain your program is seen as profit-center.

In today’s digital landscape, brands rely heavily on data-driven decision making and a carefully selected suite of technology to manage and optimize their websites, enhance user experience, and drive revenue growth. A critical tool in AB testing programs has been the free version of Google Optimize, which allows businesses to experiment with different variations of their website and measure the impact on user behavior and conversions. In September 2023, Google is sunsetting this product, and referring customers to adopt other products, most of which are not free.

What’s at Stake

With the discontinuation of Google Optimize, many brands are being forced to rethink their suite of technology and make new plans for the future of optimization. This shift will require more investment into infrastructure, training, and staff. In the current economic climate, where organizations are actively seeking cost-cutting measures amid declining innovation investments, it becomes difficult to get a budget for recurring costs in software licenses, and training staff to use them. In order for brands to keep their testing programs, they are going to have to find the budget to replace Google Optimize, but there are ways to recoup these costs and add momentum to the program by strategically selecting the right A/B testing platform.

The Pitch

If you are about to embark on finding a Google Optimize replacement and the budget to pay for it, first make sure you have a good read on where your optimization program is today, where it could be with the additional power of a new platform, and also what it would take to grow your practice to where you want it to be. Consider this migration from Google Optimize as a critical piece of your future growth strategy and this may be your big chance to ask for the budget you really need to secure the future of the program.

The Choice

Selecting a reliable A/B testing platform is crucial as it directly impacts the accuracy, scalability, and flexibility of experiments. By choosing the right platform, brands can gather actionable insights to inform their optimization efforts and achieve long-term success. A/B testing platforms all have different strengths and weaknesses, and a wide variety of costs.

When selecting an A/B testing platform to replace Google Optimize, brands should evaluate several factors. These include ease of use, scalability, integration capabilities, statistical rigor, support for personalization, and advanced targeting options. By carefully assessing the features and functionalities offered by different platforms, brands can make an informed decision that aligns with their optimization goals and long-term vision.

Key features that really make the difference

When considering each of these features as they relate to choosing the best A/B testing tool for your company, here’s what you should keep in mind:

  • WYSIWYG (What You See Is What You Get) refers to an interface that allows non-technical users to make changes visually without writing code, while the code-based approach requires coding knowledge to make modifications. Consider the technical expertise of your team. If you have non-technical members who need to make changes, a WYSIWYG editor would be more user-friendly. If your team is comfortable with coding, a code-based tool may provide more flexibility and customization options.

  • Statistics Engine: The statistics engine of an A/B testing tool calculates the statistical significance and confidence levels of the experiment results. Look for a tool that uses robust statistical methods to ensure accurate and reliable results. It should provide features like p-values, confidence intervals, and sample size calculations to help you interpret the data effectively.

  • Targeting and Personalization Rules: Targeting and personalization rules allow you to define specific audience segments for your experiments and deliver tailored experiences. Consider the sophistication of targeting options. Look for a tool that offers flexible targeting criteria based on user attributes, behavior, demographics, or any other relevant data. Advanced personalization capabilities can help you create more targeted and impactful experiments.

  • Martech and Analytics Integration: Martech (Marketing Technology) and analytics integration enable seamless data exchange between your A/B testing tool and other marketing or analytics platforms. Check if the A/B testing tool integrates with your existing marketing technology stack, such as customer relationship management (CRM) systems, analytics tools, email marketing platforms, etc. Integration allows you to leverage existing data and streamline your workflows.

  • Experiment Management: Experiment management features help you organize and track your A/B tests efficiently. Look for a tool that offers a user-friendly interface to create, schedule, and monitor experiments. It should provide options for segmenting experiments, setting experiment goals, tracking progress, and generating reports. Collaboration features like role-based access control and annotations can be beneficial for team collaboration. Remember that the specific requirements of your company might vary based on your team’s skill set, goals, and budget. It’s important to assess these features in the context of your unique needs and select a tool that aligns with your business objectives.
The Promise

Developing a strategic optimization plan is key to maximizing the benefits of A/B testing, and a vital step to get a high return on testing. This roadmap should highlight clear optimization objectives around unlocking trapped revenue. This is revenue that is being lost by abandonment, inefficient ad spending, or areas to reduce cost by increasing self-service or other online behavior. By aligning tests with revenue and cost goals, brands can prioritize high-impact experiments, streamline the testing process, and ensure a return on their software investment. 

An efficient A/B testing platform enables e-commerce brands to achieve high test velocity, allowing for rapid iteration and optimization. Test velocity refers to the speed at which experiments can be conceived, executed, and analyzed. With faster experiments, brands can uncover winning variations sooner, optimize conversion funnels, and continuously improve their website to drive revenue growth.

By identifying and implementing winning variations quickly, brands can maximize the time they are seeing an increase in conversion rates, average order values, and customer lifetime value, resulting in a net revenue gain for the organization. The effect of a winning test doesn’t last forever, come back to your winners regularly and see if they need a fresh experiment.

The Proof

Investing in an A/B testing platform is a strategic decision that requires careful consideration of cost-effectiveness and return on investment (ROI). While there may be upfront costs associated with the software, a well-executed optimization plan can help recoup the investment within a short period. Be upfront with budget owners about the cost of ownership of an A/B testing tool, but also show them the ROI model you use to make your business case for testing.

Consider that testing not only helps you incrementally increase the revenue rate of your site, it also helps stop you from making costly unvalidated changes. There are both cost savings, and revenue generation values in testing. CRO is one specific strategy that works well for same-visit purchases, but also think about the value you get from a “do no harm” test, and calculate that into your program ROI.


In conclusion, having a clear vision for how A/B testing and optimization will impact your bottom line, and a strategy for turning your product investments into actualized business value is the key to moving forward with a Google Optimization replacement that will set you up for the future.

With these tips, we hope you are able to create that vision, put together a solid case, and find the A/B testing tool that works best for your business. If you are looking for a partner to help you build your case, create your strategy, or implement your new A/B testing platform, Roboboogie is here to be your guide.

10 Years of A/B Testing: Embracing Failure for Better Tests 


Next month marks the 10-year anniversary of my first A/B test. How things have changed since those early days!

The A/B testing hustle was real – partnering with Optimizely and the incredible founding team there, under the innovative leadership of Dan Siroker and Pete Koomen.

We were all just figuring things out.

We knew that A/B testing had the power to do something incredible in the market (in fact, it helped support Obama’s re-election success in 2012).

At Roboboogie, we were excited that we could add a new layer to our UX strategy and design, with measurable results. Real-time feedback from actual users based on how they performed? Incredible!

Focus groups and user testing had been helpful tools for us, but they only provided academic feedback. A/B Testing, on the other hand, unleashed a new approach – an interactive, scientific methodology that could guarantee positive outcomes.

Growing up with a biology professor and a chemistry teacher running our household, I viewed the world through a scientific lens. Developing hypotheses was how I figured out the world, and my place in it.

Combining science with marketing in my career was a no-brainer. Bring on A/B testing!

Fast forward to June 2013.

Roboboogie engaged our first A/B testing client. A highly innovative, fast-paced e-commerce client willing to take (smart) risks, with the ‘go-fast-and-break-things’ mentality. “Let’s embrace the experimentation. Test fast, and iterate.”

We were scrappy back then. I remember singlehandedly doing test ideation, sketching UX test variation concepts, hopping on a call with clients for alignment, then jumping into the WYSIWYG editor to build it, set up analytics, QA it, and launch it – sometimes all in the same day.

At our peak, we were launching 12 tests per month (all possible due to their high site traffic and quick purchase cycles).

And we saw big success. Experiments were increasing revenue, unlocking new customer segments, and helping inform product development and positioning.

But the go-go-go approach wasn’t without missteps.

I still cringe to this day about one test in particular.

We tested dynamic pricing for a certain set of luggage products. Depending on the variation, we were showing pricing differently, presenting the retail price, promotional price, and an additional discount layer. Regardless of the variation, the product was priced at $74.99.

The test strategy and architecture were sound. The test was built to spec and seamlessly passed QA.

But when we launched the test, the results were staggering. There were massive lifts in product engagement, product add-to-carts, initiated check-outs, user time on site, total pages views per session, and… site-wide add to carts?

That’s when I got the phone call from our client partner.

“Umm, we have a shopper on the line talking to customer service who is pretty upset that they can’t buy our featured trip to Costa Rica for $74.99. What is going on?! We need to halt all testing immediately.”

Oops.

Instead of my testing parameters applying only to the backpack products we were piloting our experiment on, they had been applied site-wide. To clothing, snowboards, bikes, kayaks, and even trips. The test was running perfectly for our intended products – but also everywhere else.

The go-fast-and-break-things approach had … broken things. While we had massive wins, we also now had a sobering misstep. The experiment was only live for about 25 minutes, but it had caused some need for damage control with several customers. Luckily, customers were understanding, and with some exchanged store credit, everything was smoothed over fairly quickly.

That mistake, however, significantly shaped my professional approach and how we approach our testing methodology at Roboboogie. The experience has proved invaluable time and time again. For it is out of failure, that our best growth and maturity comes.

My attention to detail has never been the same since – an approach we now incorporate into our testing process. We have a fully immersive, multi-disciplinary approach to each step of the testing process – involving thoughtful strategy, smart UX, tight UI, pixel-perfect development, methodical data engineering, and double-and-triple-check QA before launch. Each team member has an eye on the bigger picture goal – launching smart tests, free from errors, with the right balance of speed, strategy, and attention to detail.

For us, our only “failed” tests are the ones that are launched broken.

We embrace the mantra of “go-fast-and-BUILD-things” now. We believe our clients and end-users deserve better than broken tests.

Not every first test we launch results in an immediate net-positive CRO impact (it’s experimentation, after all). But we work to ensure that every test we launch is a winner – either driving revenue/leads, elevating the brand experience, unearthing user insights, or unlocking new user segments.

And we’re incredibly proud of our ability to do so.

Would we be where we are today without that mistake? I’m not sure. But I do believe it catapulted us forward. Looking back at that mistake 10 years ago – almost accidentally selling a $3500 trip to South America for $75 – I dare say the mistake may have been worth it.

That failure resulted in a decade of better tests – for dozens of clients and millions of tested visitors since.

  • Jedidiah Fugle, COO @ Roboboogie

The Mission for Fully-Perfect Customer Connections

Each step forward, no matter how winding the path, is a step closer to digital transformation.

Over a decade ago, Roboboogie humbly began as a specialized design boutique focused on solving complex UX challenges for our clients. As time went on, we rounded out our services to cover the full web experience with expertise across digital marketing, creative, and web development. Like our peer agencies, we had to rely heavily on institutional knowledge, best practice, industry insights, and a healthy dose of intuition.

In 2015, we layered on A/B testing thanks to a fresh partnership with our friends at Optimizely, which unlocked a new ability to run experiments in rapid fire. Shout out to our first A/B testing client, The Clymb! A/B testing opened a new and powerful way to help our clients through iterative site improvements. We could now validate our UX strategy and make swift improvements to bring about big, measurable change.

Experimentation made customizable data collection simple, creating the opportunity to let data drive design decisions. The data quality, paired with our ability to translate numbers into insights, provided a deeper path into client partnership. As we have gained experience in the field, the suite of analytics technologies we utilize has grown. We’ve harnessed an array of qualitative and quantitative analytics tools to drive new insights including Adobe Target, A/B Tasty, Google Optimize, Convert, UserTesting.com, FullStory, Crazy Egg, Hot Jar, GlassBox,  among many others. We even developed a healthy obsession with Google Analytics that led us to building a custom A/B testing algorithm within GTM.

However, our service offerings between data, design, and technology remained siloed and clients often connected with us for a single solution. We knew the value of integrating this work, which led to our transition from an agency to a consultancy. As our position within partner organizations continues to elevate, our role has increasingly been about charting the best path forward with our clients. Our teams have become intimately entwined, building off of each other’s work on the mission of digital transformation.

Unified in mission.

Optimization is no longer something we do at Roboboogie, it is how we live and breathe. It is not a product offering or deliverable. It does not live with one role, individual or department.

Everyone on our team is an optimization consultant, and every individual shares that same top-level goal. Our team is routinely asking themselves, “Does the work I’m doing help move my client toward ‘perfected brand-customer connection’? What is the most valuable thing I can do right now to advance that mission? “ 

The secret sauce of optimization is in process. It is the methodology in which you identify needs, prioritize opportunities, develop solutions, and execute plans. It’s in the deliberate weighing decisions against the mission: a lifetime customer bond. It’s a recipe we’ve been developing for over a decade.

Our optimization recipe.

  • Partner with innovative clients who are passionate about holistically driving transformation of their digital experience.
  • Develop audience personas and map customer journeys – informed by data –to understand the deep needs and desires of customers. 
  • Harness wide data sets and powerful technology to unearth gaps in the current digital experience that can be filled with design solutions. 
  • Integrate teams of experts across data, design, and technology.
  • Customize the optimization strategy for each client.
  • Let data guide the journey, but never let numbers stifle the creative process.
  • Stay nimble and responsive. Each step forward, no matter how winding the path, is a step closer to digital experience transformation.

True, authentic digital transformation takes time. It takes a continued commitment to reorient to ever-changing customers and the all-out pursuit of taking each experience one step closer to perfection. Each step provides reward with more customers, happy customers, and improved performance.

Looking ahead.

Internally, we use the playful mantra “Happy Customers Convert.” Since our beginning, we have always held the customer at the heart of everything we do because taking care of people gets the best results. Over the years as we have layered on measurement tools, we have been able to prove that to be true. When customers’ needs are met, business performance follows. It turns out, happy customers do convert.

As we officially transition Roboboogie’s focus from Website Performance Optimization to Digital Experience Transformation, we are eager for the next chapter ahead. You’ll still be able to expect that same great work and services we offer, but with a higher unified vision. There’s so much untapped potential across the digital landscape for organizations to connect their data, design, and technology efforts to create fully-perfect customer connections online. Investing in Customer Lifetime Value pays. We have witnessed it first-hand (with big business impacts) for some of our long-time client partnerships – NordicTrack, Adobe, and Wacom to name a few. 

If you are interested in partnering together to create digital environments that form undeniable connections and life-long bonds between your brand and customers, hit me up (Jed@teamroboboogie.com). I’d love to share more about how we can help you better meet your customer needs in the digital space. I can confidently say you’ll be thrilled with the business outcomes. 

Until then, just remember: Happy Customers Convert.

Jedidiah Fugle, Roboboogie Chief Operating Officer