FAQs: Testing and Experimentation in the UX Process

Lacie Webb

As experts in the testing and experimentation space, we are frequently asked about the importance of User Experience design (UX), and the role it plays in optimization campaigns: What does a UX Designer do exactly? What challenges do they face when developing A/B tests? We’re here to answer these questions and more, to illustrate why UX is a critical component of almost every project we take on. Check out the following responses from our very own UX/UI Designer, Lacie Webb.

Lacie Webb drawing wireframes on a chalkboard

Describe a typical workday for you.

I usually start my day by getting to work before everyone else (typically around 7am, but hey, I’m weird) so that way I have some time to spend getting caught up on any blog reading or finishing up an episode of a design podcast. Some mornings I’ll work on my own personal design projects as a lightweight warm up exercise to get my brain in the “making” mode. Then I’ll create my daily “Most Important Tasks” which is (what it sounds like) a prioritized list of the tasks I must finish that day. Those tasks typically consist of updating designs/wireframes based on client or internal team feedback. I will usually have 1-2 client presentations scheduled each week, in which I walk our clients through an interactive prototype of a fully designed comp explaining our reasoning and recommendations. I try to strike a balance between how much of my day is spent in meetings vs. maker time.

What research methods do you find most effective?

A competitive analysis is a great way to see who in adjacent industries are interacting with their users or research problems to see if there’s already an established best practice for a particular flow/interaction. But a lot of time there is no concrete answer and you have to use your intuition and have a deep and empathetic understanding of the users you’re designing for.

Lacie Webb draws wireframes in a notebook

What unique challenges do you face in designing towards A/B tests?

As designers, we just naturally have a lot of ideas on all the ways a design could be different. There’s no shortage of ideas. But when you’re given a certain allotment of tests to run you have to be very intentional about what you’re testing and why. What hypothesis are you testing? What are you hoping to learn from this test? Sometimes it’s hard to take that step back from your design and ask those questions before jumping right in and making stuff. If you do approach testing with the mindset of “What can I learn as a designer?” and, “What can I learn about these particular set of users?” you’ll have a much better understanding of the tests and more importantly, the results.

How frequently are you able to test various stages in your design process to make sure you’re on the right track?

It really depends on what client and how deep they want to go. Most of our testing happens after the design is made and developed. You could test prototypes and even static designs (and that’s what a lot of in-house teams will do, oftentimes with apps and products) and all that information would be very helpful but also slows down the process, which is a hard thing to get clients on board for. But regardless of when you test I would say all that information is helpful even if it’s just learnings you take with you onto the next project. You’re always learning about users in that way.

What metrics do you use to measure a design’s success?

We use “conversion rates” to measure a design’s success. Conversion rates can be calculated actual monetary values that are related to checkout totals in an online store or conversion rates could mean total number of people who signed up for a newsletter or created a profile. Basically we measure how often a user was able to reach and be motivated to interact with our design.

Lacie Webb shows us a wireframe on a window

What software is most critical to your daily practice and why?

It is important to not rely too heavily reliant on software in general, but the software I use most often in my day is Axure (an interactive prototyping tool) and Adobe Photoshop (for Visual/UI Design). But I could just as easily do my job with just a pencil and paper and a lot of the time I prefer those tools because I feel less pressure to come up with the perfect solution the first time.

What are your favorite online resources (news, blogs, tools) and why?

I keep up with news and design blogs such as Fast Company and online design communities such as Designer News to stay up to date with current best practices and technologies. I also listen to podcasts such as Creative Pep Talk which never fails to put me in a great mindset to create awesome things and to kick my energy to make into high gear.

We hope our thoughts on these FAQs help answer some of your own curiosities about testing and experimentation in the UX process. Want to know more? Drop us a line, we’re always happy to chat.

Interview with Roboboogie Sr. Producer, Jesi Wu

Jesi Wu

Here at Roboboogie, we firmly believe that every project needs a dedicated producer. From start to finish, the PM is responsible for making sure all projects are completed to Roboboogie standards, and that each contributor has exactly what they need to succeed. Jesi Wu is our own master of production, and this week we’re picking her brain to see just what makes her tick.

How do you describe what you do?

Simply put, I get shit done. I am the bridge between the client and my internal team. I bring teams together to make beautiful things happen (while in scope, on time, and under budget).

While crushing to-do tasks and project milestones, I strive to provide the best possible customer service to any client interaction and aid in building strong personal relationships (sometimes even friendships) alongside an account manager with our clients and project stakeholders.

What is a belief people have regarding the industry that you might not agree with?

Get the right people and you won’t need producers.”

Every time I hear something along those lines, I cringe. In my opinion, it’s crucial for all projects to have a dedicated producer. If you choose to not have one, your project will be at risk. Someone will have to assume the “producer” role, and it might as well be someone who has the passion, training, and time for it. Projects NEED a “neutral party” to manage the day-to-day havoc, so other project contributors can focus on fun parts.

What is a skill set that you use every day, but never thought you would have to?

Empathy. I feel like I’ve become a better person because of this. My job involves me having to connect with many different types of people. It’s impossible to know everything that’s going on in your colleagues’ or clients’ personal lives, and sometimes these factors can affect project tasks or scope. Through empathetic listening, I’m able to create a better understanding of the other person’s perspective, as well as create better awareness within myself. Applying this skill set to my daily interactions with my project teams has helped my relationships grow stronger and overall has helped me gain respect and trust with these individuals. This relationship allows me to have a firm hand on the project.

How do you approach solving problems?

When I feel a problem arising, I first take a deep breath or meditate (I’ve been loving the Headspace app). When I force myself to pause and breathe, rather than react prematurely,  I can save myself from behaving emotionally and striking out in a way that might make things worse or cause unnecessary stress to my team members.

Once I have reflected on the situation, I will create an open line for communication so everyone can have their say. As a natural mediator, I like to solve problems by being transparent, proactive and positive. One team, one dream...

What is your advice for turning a strategic idea into reality?

As a PM, a lot of people talk to me about potential ideas for new projects and tests. This is one of my favorite parts of the job. I love watching/listening to people voice their passions and seeing their creative juices flowing. It’s my job to turn those passions into a reality.

I’ve got three bits of advice for you:

  • Ask a million (or a few) questions! In doing so, you will become an expert in their idea. Live and breathe it.
  • Involve the right people for execution. In return, it will help avoid potential blockers.
  • Be realistic. Listen to your team, give them sensible deadlines and a practical view on what can/can’t be achieved within your current resources.

What advice do you have for brands who are interested in getting into experimentation and iterative design from a producer perspective?

Establish a process for sharing wins internally. Wins are meant to be shared. Publish your team’s ability to achieve business goals for your term based on iterative testing. It’s a great way to show the true value of experimentation to your higher-ups while helping them make some serious dough.

Thanks Jesi for giving us an inside look at the world of Production and what it takes to excel in that vital role. Stay tuned next month for another interview with a Roboboogie pro.

Working with AngularJS via Optimizely without Manual Activation

Darius Capizzi

Introducing: Darius Capizzi

Role: Developer
Time at roboboogie: 2 years and counting…

I’ve been part of the roboboogie team that optimizes web experiences through multivariate and A/B experimentation for just over two years. In that time, I’ve had the opportunity to solve for and build highly integrated experiments using a variety of testing platforms: reworking a checkout funnel top-to-bottom, dynamically altering content based on unique events, and utilizing complex targeting algorithms to power personalized experiences. At roboboogie, we work with an ever-evolving array of clients spanning a number of different industries with unique challenges, technologies, and goals. Part of my role is to identify wholistic test development solutions that maximize efficiency and effectiveness of test builds based upon the plugins, platforms, frameworks, and tools the client has available. One of the biggest hurdles I have encountered is when developing tests interacting with AngularJS.

Angular is Javascript framework, allowing for quicker, more immersive user experiences. It does this by loading content asynchronously. The result is a snappy, clean experience for the user. However, it creates challenges when attempting to load outside technologies for testing, like the script loaded through Optimizely to trigger experiments, track goals, and load content.

Ready for the good news? I’m here to share what I’ve learned and what’s worked for us at roboboogie!

The Scenario:

You’ve built a rockin’ A/B Test for a site built with Angular using Optimizely. Credit where credit is due: nice job.

You spin up the experiment to test it and it works locally: double nice job. But when you test it live, oh jeez…. Something is wrong. For some reason, AngularJS hasn’t loaded that component when your Optimizely code tried to trigger the change. After triple-checking your test code, everything checks out within Optimizely, yet the experiment isn’t triggering. Awww, of course. Angular loads content asynchronously and will probably finish updating elements after Optimizely has finished its setup.

Ready to give up? Don’t! People do this stuff. Let’s unpack some solutions to address the issue.

The Solution(s):

Below I’ve outlined what I have found to be the best methods to work through the issue. You will have to customize and troubleshoot based upon your stack, but the following should get you what you need to run your tests successfully.

(Warning: the following content is about to get very techy. So hold on to your coder pants.)

Solution 1: Using Conditional Activation

This boils down to triggering your experiment within your Angular codebase. If you are the owner and operator of the web property and have access to make changes within the Angular implementation, this is the ideal solution. It involves building each variation into your codebase and conditionally activating the experiment and its variations. However, if you do not have this luxury (working without direct contact with development team, or are servicing a client), the requirement for another team to deploy code every time you want to run an experiment usually creates too much overhead for the in-house team..

For further reading, check out Optimizely’s documentation for Classic and X.

Solution 2. Wait for AngularJS by Polling for Window.angular

If you are working through a snippet alone, without access to the codebase, and need an alternate fix polling for Window.angular can be a good solution. Here is some code to get you started polling:

var tooManyLoops = 0, doOnce = true, viewModel = "", maxLoops = 3000, timeoutDelay = 300;
var observeAngular = function(inElement, callback) {
var elementToCheck = $(inElement);
if (
!window.angular ||
!window.angular.element(elementToCheck) ||
!window.angular.element(elementToCheck).scope() ||
!window.angular.element(elementToCheck).scope().vm ||
) {
if (tooManyLoops > maxLoops) {
setTimeout(function(){observeAngular(inElement, callback)}, timeoutDelay);
} else if (doOnce) {
doOnce = false;
viewModel = window.angular.element(elementToCheck).scope().vm;
} else {
observeAngular('.some-angular-element', yourDomManipulation);

This is a recursive solution where we poll for an element and its associated scope. Angular binds data to elements, so to get the data, we grab the element. Once we have the scope returned, then we know DOM is ready for manipulation. The scope associated with an element might also provide you a slew of relevant information, such as out of stock status or original vs. reduced pricing. It should be as simple as targeting the troubled/refreshing elements, or their parent elements. Sometimes it can be tricky to find $('.some-angular-element'). If you encounter this, look around the DOM for ng- attributes, like ng-repeat or ng-show, and check those elements.

Tip: If running window.angular.element($('.some-angular-element')).scope() returns undefined, then debugInfoEnabled is probably set to false. Run angular.reloadWithDebugInfo(); in the console, then try to grab scope again and you should see the scope being returned. You may have to ask the client to set debugInfoEnabled to true for some functionality.

vm is a commonly used object to reference the element’s controller and in this case stands for “View’s Model.” It is described in the controller as syntax, so it’s usually there. If not, you might have to dig around in the scope for another object to poll for or an alias to vm. Use window.angular.element($('.some-angular-element')).scope() in the console to check for vm, and try checking its .$parent if the vm object is not present. Finally, in the code above, “desiredFeature” can be any property. You can expect things like an add-to-cart method on an add-to-cart button or product comparison tools on product pages.

So now you have the initial DOM manipulation working, and it looks super great. For pages that do not reload elements based upon user interaction, you can give props to your designers and you’re done. Otherwise, if you do have refreshing elements and these $('.some-angular-element')‘s refresh on click of a button, they will strip your styles from the DOM and reset the page to some hybrid of the original and your current test build. You probably should have caught this in initial development, but it’s never too late! It’s time to…

Solution 3: Observe the Element with a Mutation Observer

The disadvantage to the setInterval solution is recurring flashing, since the setTimeout will (likely) not run immediately after the element was added, like a Mutation Observer that is listening for changes on specific elements will. Before testing, check this caniuse, which namely excludes Opera Mini and IE (<=10). Don’t forget to adjust your audience settings accordingly.

Here is a simple example of a mutation observer:

var observeDOM = function(elementContainer, elementSelector){
var MutationObserver = window.MutationObserver || window.WebKitMutationObserver;
var options = { childList:true, subtree:true };
var observer = new MutationObserver(function(mutations){
console.log(mutations) //here to get a better picture of how they come in
for (var i = 0; i < mutations.length; i++) { if( mutations[i].addedNodes.length > 0) {
if ($(mutations[i].addedNodes[0]).is( elementSelector )) {
observer.observe(elementContainer, options);
observer.observe(elementContainer, options);

Okay, so maybe not that simple, but it is a simplified version. Now let us work through the observer to define it in a way that works for your project.

Try to be narrow when choosing your elementContainer. Select the direct parent of the refreshing element. If you get Failed to execute 'observe' on 'MutationObserver': parameter 1 is not of type 'Node' it means the elementContainer is likely either not there on runtime or you passed it a jQuery object. What you need is a native JS object. You can pull a native JS object from the DOM in a few ways. Using jQuery array notation it would look something like $('#refreshing-element-container')[0] or with native JS it could be pulled using an id like this document.getElementById('refreshing-element-container').

Look out it’s a stack overflow! (No, not the site). The observer will run EVERY time your element is added. That includes if you append it using your secondaryDomManipulation function. It is important to call observer.disconnect(); before detaching or removing of a $('.refreshing-element'). If your secondary manipulation of $('.refreshing-element') is async, you should move observer.observe(elementContainer, options); to the end of that call. Also, having two separate asynchronously loaded views on one page can get messy fast, as it can easily produce false positives, leaving you erroneously re-appending your elements.

This Mutation Observer is only watching for nodes being added, but if you want to observe removed nodes you can add mutation.removedNodes to the list. See David Walsh’s post on Mutation Observers for other configuration options including attribute mutations.

If you would rather use a plugin or script you can check out this CROmetrics article where they share a sweet jQuery plugin which defaults to a setInterval solution if MutationObserver is missing from a Browser’s default feature list. Also, the Optimizely docs have example code to source external scripts, as well as a number of other useful helper functions.

Not here to preach, but I think there’s something to writing these type of scripts yourself. Maybe so that you aren’t just wildly downloading libraries or so that you can make changes to the script as needed and only use the functions that are necessary for your code base.

In Conclusion…

You’ll run into a bunch of errors and it’ll be hard, but this is what developing is all about! Each Angular test you will build will need a solution that is a little different from the last and the next, but keep pushing forward. If you do get stuck or just want to say, “Hi!”, feel free to reach out to us by leaving us a note on our contact page. We’d love to help where and when we can. Good luck and happy testing.

Written by Darius Capizzi