(NOTE: This post will be updated as new information and solutions are available from Optimizely, Google, VWO, Convert and others).

Update: April 1, 2019

As of March 25th, iOS 12.2 and Safari 12.1, on macOS High Sierra and Mojave, are now live.

Considering the impact ITP 2.1 will have on analytics and the continuity of experiences in testing, we are anticipating announcements from each testing platform soon. Adobe has released the first statement in which it outlines a solution using HTTP cookies set by a shared web-service. if we assume this to be the trend, and it seems to be in discussion within analytics circles, we can expect the solution to go like this:

  1. A company looking test will need to set up a CNAME record that is a subdomain. As a subdomain it exists technically as same-site, reference as eTLD+1. (eg. tracking.yourdomain.com)
  2. When this CNAME record is called by yourdomain.com it will redirect to a subdomain such as “tracking-fix.analytics.google.com”, which responds with a HTTP cookie (eg. _ga). Because the HTTP cookie is handed back to the subdomain, which is considered same-site, coming from a url on the same domain, it will not be deleted in 7 days.

This puts the majority of the work on the web service, and only a little bit more on each client looking to perform testing.

In a discussion on Twitter between Simo Ahava and John Wilander, Simo, speaking about using CNAME’s to set first-party HTTP cookies for analytics asked, “Do you think this is future-proofed (as much as anything can be)?”

John Wilander answered: “The only ‘future proofing’ to be had is to stop cross-site tracking. If trackers are allowed to repurpose scripts in the first-party context to facilitate cross-site tracking, we have no choice but to prevent it.”

Another notable viewpoint worth reading through is a recent proposal from Mike West, who works on the Chrome security team. In this tweet, West says we should deprecate javascript cookies all together. He points out over 84% of cookies are not using the built-in security features released over two decades ago and we cannot trust that any future security enhancements will be taken seriously. Notably it was retweeted by John Wilander, Apple WebKit Engineer behind Safari’s Intelligent Tracking Prevention.


First: What is happening? In the upcoming release of Safari, cookies will only last 7 days. This includes iOS. Firefox to follow.

Second: Why should you care? This will break test integrity. A user who comes back after a week will not necessarily see the same experience. Unless the test is concluded within 7 days, it would be difficult to have a consistent experience, or any reliable user-level metrics. Personalization? BAH.

But let’s not freak out, it is currently a small audience that we are able to exclude from testing, and there are already some potential solutions. There has yet to be an official response from Optimizely, Convert, VWO, or even Google about how they are planning on handling this. Hopefully they will have a solution on their end. In the meantime let’s talk about what’s happening and what we can do.

(Update #1 Tuesday March 26th @4:55pm): There has been an update from Adobe (read here). Although they only mention their testing platform briefly (Adobe Target), they do identify their proposed solution for user level identification… CNAME! We have also been in direct contact with one of our testing partner technologies who mentioned they were headed in this direction as well. We will be going further into the benefits of this solution soon!

Background:

With the introduction of iOS 12.2 and Safari 12.1, expected to roll out at the end of March, both browsers will be capping all client-side cookies to 7 days. This particular feature is called Intelligent Tracking Prevention (ITP) 2.1 (source). Firefox has began testing a similar behavior in a near future release (source).

The motivation behind this is user privacy, and they have been moving this way for some time. Last year, Safari limited third party site cookies from persisting in an effort to limit the tracking of users across sites. This in turn caused analytics tools, advertisers, and anyone looking to leverage a users journey, to start using first party cookies to track them. Currently third party services are able to set first party cookies to track users, and ITP 2.1 is Safari’s response.

Client-side cookies are where these tools store user-level identifiers. They are what make a user a User. Allowing you to say “Most users visit 3-5 times in six months before converting.” Most have expiration dates well beyond 7 days. For example Google Analytics uses a 2 year cookie (source) and Optimizely uses (by default) a 6 month cookie (source). This is to ensure that a user’s information is always continuous.

Continuity is integral for testing, advertising, and analytics. For A/B testing specifically this is a massive dilemma. Most testing platforms (Optimizely, VWO, Convert) mainly use javascript injection to load experiences. This means their only means to create cookies is through the client-side. Without continuous user IDs, users will be given new IDs each time they come back to a site after 7 days. This ultimately means users will be bucketed within different variations if a test necessitates a run longer than 7 days to reach statistical significance (which is common even if a site gets hundreds of thousands of visitors a month). Not only will this be disruptive to users who might be bumped back and forth between variations, but KPI attribution to specific variations will be impossible in these situations.

Who will this this effect?

Safari is the second most used browser in the world. This is because of mobile, which is increasing in market share. Firefox is the third (source). But how many will update, and how fast? This is something to be monitored, and will certainly be a growing audience.

What are the potential solutions?

For now, it may be simple enough to just ignore this audience. This can be done in analysis. It is expected to be small audience to start, and we will be monitoring them closely. Ultimately, we still don’t want those users to see a bad experience. Once a test is launched, we will confirm the User Agent and release details on the specific audience. We will also update this post with trends.

Other long-term, more wholistic potential solutions are updating user-level identifiers via “localStorage” or setting the cookie from the server-side such as use of an HTTP cookie.

With no news yet from the major A/B testing platforms, rather than investing the upfront costs of working around this change, simply excluding the audience may be best for the time being. (However, we are investigating both options mentioned above currently and will post details as we identify them.)