10 Years of A/B Testing: Embracing Failure for Better Tests 

Next month marks the 10-year anniversary of my first A/B test. How things have changed since those early days!

The A/B testing hustle was real – partnering with Optimizely and the incredible founding team there, under the innovative leadership of Dan Siroker and Pete Koomen.

We were all just figuring things out.

We knew that A/B testing had the power to do something incredible in the market (in fact, it helped support Obama’s re-election success in 2012).

At Roboboogie, we were excited that we could add a new layer to our UX strategy and design, with measurable results. Real-time feedback from actual users based on how they performed? Incredible!

Focus groups and user testing had been helpful tools for us, but they only provided academic feedback. A/B Testing, on the other hand, unleashed a new approach – an interactive, scientific methodology that could guarantee positive outcomes.

Growing up with a biology professor and a chemistry teacher running our household, I viewed the world through a scientific lens. Developing hypotheses was how I figured out the world, and my place in it.

Combining science with marketing in my career was a no-brainer. Bring on A/B testing!

Fast forward to June 2013.

Roboboogie engaged our first A/B testing client. A highly innovative, fast-paced e-commerce client willing to take (smart) risks, with the ‘go-fast-and-break-things’ mentality. “Let’s embrace the experimentation. Test fast, and iterate.”

We were scrappy back then. I remember singlehandedly doing test ideation, sketching UX test variation concepts, hopping on a call with clients for alignment, then jumping into the WYSIWYG editor to build it, set up analytics, QA it, and launch it – sometimes all in the same day.

At our peak, we were launching 12 tests per month (all possible due to their high site traffic and quick purchase cycles).

And we saw big success. Experiments were increasing revenue, unlocking new customer segments, and helping inform product development and positioning.

But the go-go-go approach wasn’t without missteps.

I still cringe to this day about one test in particular.

We tested dynamic pricing for a certain set of luggage products. Depending on the variation, we were showing pricing differently, presenting the retail price, promotional price, and an additional discount layer. Regardless of the variation, the product was priced at $74.99.

The test strategy and architecture were sound. The test was built to spec and seamlessly passed QA.

But when we launched the test, the results were staggering. There were massive lifts in product engagement, product add-to-carts, initiated check-outs, user time on site, total pages views per session, and… site-wide add to carts?

That’s when I got the phone call from our client partner.

“Umm, we have a shopper on the line talking to customer service who is pretty upset that they can’t buy our featured trip to Costa Rica for $74.99. What is going on?! We need to halt all testing immediately.”


Instead of my testing parameters applying only to the backpack products we were piloting our experiment on, they had been applied site-wide. To clothing, snowboards, bikes, kayaks, and even trips. The test was running perfectly for our intended products – but also everywhere else.

The go-fast-and-break-things approach had … broken things. While we had massive wins, we also now had a sobering misstep. The experiment was only live for about 25 minutes, but it had caused some need for damage control with several customers. Luckily, customers were understanding, and with some exchanged store credit, everything was smoothed over fairly quickly.

That mistake, however, significantly shaped my professional approach and how we approach our testing methodology at Roboboogie. The experience has proved invaluable time and time again. For it is out of failure, that our best growth and maturity comes.

My attention to detail has never been the same since – an approach we now incorporate into our testing process. We have a fully immersive, multi-disciplinary approach to each step of the testing process – involving thoughtful strategy, smart UX, tight UI, pixel-perfect development, methodical data engineering, and double-and-triple-check QA before launch. Each team member has an eye on the bigger picture goal – launching smart tests, free from errors, with the right balance of speed, strategy, and attention to detail.

For us, our only “failed” tests are the ones that are launched broken.

We embrace the mantra of “go-fast-and-BUILD-things” now. We believe our clients and end-users deserve better than broken tests.

Not every first test we launch results in an immediate net-positive CRO impact (it’s experimentation, after all). But we work to ensure that every test we launch is a winner – either driving revenue/leads, elevating the brand experience, unearthing user insights, or unlocking new user segments.

And we’re incredibly proud of our ability to do so.

Would we be where we are today without that mistake? I’m not sure. But I do believe it catapulted us forward. Looking back at that mistake 10 years ago – almost accidentally selling a $3500 trip to South America for $75 – I dare say the mistake may have been worth it.

That failure resulted in a decade of better tests – for dozens of clients and millions of tested visitors since.

  • Jedidiah Fugle, COO @ Roboboogie

Top 5 Things You Need to Know about GA4

On July 1st, 2023, Google will sunset Universal Analytics and replace it with Google Analytics 4 (GA4). This new platform is designed from the ground up to be fully compliant with the General Data Protection Regulation (GDPR) and is built for the future, allowing organizations to seamlessly integrate desktop, mobile, and app traffic in one centralized environment. There are numerous reasons to be excited about this shift in data collection, but digital organizations around the globe are faced with the challenge of migrating their analytics in an efficient and accurate manner. As a digital optimization consultancy, analytics is central to our mission of making the internet a better place through data-backed recommendations. Whether you’re an analytics pro or just getting started, here are our top five tips on preparing for GA4:

1. Event-based data collection, rather than session-based 

GA4 takes an entirely new approach to data collection than previous versions. Whereas in Universal Analytics (UA), events were captured and reported on within a duration of time called a “session”, GA4 processes each individual interaction as a standalone event. This allows for the seamless tracking of a user as they browse on their phone, research on their tablet, and convert on their home computer (for example).

2. New UI, new reporting 

GA4 isn’t just a new name and data structure, it also features a new and simplified user interface. You don’t need to be a professional analyst to get actionable insights right out of the box. Simply browse the pre-built reports and dive in! If you’re a seasoned vet looking for a familiar experience, you may find some of your tried-and-true reports look or behave differently, but with GA4’s Explorations you can slice and dice your data for more powerful custom reports, leveraging a gallery of robust templates or free-form analysis.

3. Events are simpler than ever

Implementing and analyzing events to understand your digital experience is easier than ever with GA4. Under Universal Analytics, events include a three-tiered structure, which relies heavily on Google Tag Manager (GTM). However, GA4 tracks unique interactions under a single event name, with necessary additional details being captured as event parameters. For example, “click_video” may be the event, with an event parameter of “Top 5 Things to Know About GA4”, which is the title of the video. You can even create events and conversions based off of either event or event parameter, allowing a page view for your confirmation page to fire a unique event to track conversion performance.

4. GA4 puts privacy first

GA4 is fully GDPR compliant and uses first-party cookies, but is also designed to leverage machine learning to work in a cookie-less future. The inclusion of machine learning-powered artificial intelligence offers deep insights and quality of life improvements to your reporting—like understanding when a user is sent off-site to complete a transaction but sent back to your site for confirmation.

5. GA4 is not retroactive 

While the mandatory switch to GA4 goes into effect on July 1st, 2023, it’s important to understand that analytics data is not retroactive. The sooner GA4 is up and running on your property, the better its machine-learning capabilities will be able to serve your business goals. Running GA4 and Universal Analytics—or GA360—will not result in any breakages, so there is no harm in running them in parallel. It’s likely GA4 will continue to see updates before July 1st with more features being released or improved. There’s no better time than now to jump in and start learning!

In Closing

With just five months until the switch to GA4, do you have your migration plan ready to go? From auditing your set-up to configuring events, we’re here to help create a custom solution for you. Don’t wait until July, connect with our team today to get started!