A/B Testing at the Edge with Servers Workers
In Second Meaningful Content: the Worst Performance Metric, I noted the terrible impact on page loading performance that can occur when sites introduce personalized content and A/B test logic on the client-side, using JavaScript. To recap, the delivery and application of this pattern tends to produce a “wait and switch” effect; delaying both initial page rendering, and the subsequent re-rendering of portions of the page, as illustrated in this visual timeline:
The article finished with some tips for either mitigating the impact of the approach, or moving the logic away from the browser. However, while the latter is ideal, it may not be an easy change to make. Recently, we’ve been exploring an approach that gives us all the performance benefits of a server-side approach with the familiarity of doing the work with JavaScript.
Servers Workers to the rescue?
Permalink to 'Servers Workers to the rescue?'Delivering HTML that’s ready to render on arrival is a worthy goal. However, if that HTML needs to vary its content for different users, achieving that goal will require running some logic on the server or in middleware between the server and the browser. Using middleware for content transformations like this is not a new thing, but recently CDN-provider Cloudflare introduced support on their middleware CDN servers using JavaScript Service Workers to handle the logic, which is a new idea indeed.
For those not already familiar, Service Workers are JavaScript files that are typically sent to the browser where they act as a proxy to handle tasks like caching, content modifications, and offline fallbacks. In typical use, Service Workers have some differences in browser support, and can be used only after downloading, meaning they’re more helpful after an initial page visit. However, running Service Workers in a server environment (or middleware like Cloudflare’s CDN) eliminates many of those technical concerns. For those familiar with authoring Service Workers, we can use the same patterns and syntax when running service workers in the browser but take all the advantages of being on the server.
More like servers workers… right?
Working with Servers Workers
Permalink to 'Working with Servers Workers'Given that we have clients who use Cloudflare already, and that we know a bit about Service Workers ourselves, we were interested in trying this tool out. On Cloudflare’s site, there’s a nice playground page to test how these servers workers work. You can try plugging in most any URL and practice modifying its response text, swapping image URLs, changing headers and more. To get a little familiarity, I’d suggest going to the playground site and browsing the example workers in their documentation.
Let’s explore a couple of examples of how we’ve been trying to use this tool to move dynamic content logic out of the browser to improve performance.
A Delicious A/B Test
Permalink to 'A Delicious A/B Test'At the recent Performance.Now() conference, I demonstrated a couple of service workers that we built to drive A/B tests. I’ll start with a simpler example before moving on to a more complex, but more useful one.
Pictured below, we have two variations of a link to a site’s “Treats” section that we could serve to our users to see if we get more clicks when the text is either “donuts” or “stroop wafels” (the conference was in Amsterdam, after all!).
In this A/B test, we can consider the users who get “donuts” to be in the A group, while the B group will be the stroop group. To drive this, we could start with some HTML containing a link that looks like this, containing the “donuts” text:
<a href="/treats">Try our delicious <i>donuts!</i></a>
The following Service Worker script, when running on Cloudflare and pointed at that page, would change the first instance of the text “donuts” to instead say, “stroop wafels”. Note the line where a standard JavaScript replace method is used to swap the text.
addEventListener("fetch", event => {
event.respondWith(fetchAndModify(event.request));
});
async function fetchAndModify(request) {
const response = await fetch(request);
const text = await response.text();
const modified = text.replace("donuts", "stroop wafels");
return new Response(modified, {
status: response.status,
statusText: response.statusText,
headers: response.headers
});
}
Concise and standards-based!
Of course, to actually be useful as an A/B test, we’d want that worker to do a little more.
- First, it should only transform the text for 50% of our visitors.
- Second, we’d want it to only run on requests to a particular HTML file on our site (rather than running on every request to files on the site - including requests to CSS, images, etc!).
- Third, we’d want to give the test a name and track which group “wins” the test, for example, by setting a cookie for the group that the user belongs to. This cookie could later be detected by most any analytics package to correlate, for example, sales from each group. I should also note here that once a user is cookied into a given test group, the worker will continue to use that same group on subsequent visits to the same page.
Rather than dump the full code into this article, I’ll link out to a playground page containing a Service Worker that performs the above list of tasks: A/B Service Worker Example 1.
If you open that page, you’ll see some configuration at the top of the service worker for the test’s name, the text to look for, the text to potentially swap in, and the URL that the test should run on. Browsing the code further, you’ll see the parts that don’t require editing, where the worker handles the 50/50 split, setting a cookie with the test name and assigned group.
And that’s it! Here’s an example of the link that the Stroop Group will see, all pre-transformed when the browser recieves it:
A More Delicious Swap
Permalink to 'A More Delicious Swap'The worker above handles a simple A/B test just fine, but the transformation it does is very limited. More often, we’ll want to change more than just a word, but instead swap between larger slags of HTML. For example, a site might A/B test an entire hero banner on a landing page, and suffice to say, a lowly word replacement is really under equipped for matching multiline HTML like that.
So to build on our first example, let’s consider a more complex case, such as this link swap that changes not only the text but an image and some styling as well:
For cases like this that require larger swaps of HTML, we made a more robust worker. What’s neat about this one is it requires no JavaScript configuration for each test. Instead, the configuration of one or more tests occurs inside the HTML that the service worker runs on, denoted by special HTML comments that mark the start and end of each named group.
Take this initial markup for example:
<!--workertest:mytest=a-->
<a href="#" class="donuts">
Try our delicious <i>donuts!</i>
<img src="donuts.png" alt="two iced donuts">
</a>
<!--/workertest:mytest=a-->
<!--workertest:mytest=b-->
<a href="#" class="wafel">
Try our delicious <i>stroop wafels!</i>
<img src="wafel.png" alt="stroop wafels">
</a>
<!--/workertest:mytest=b-->
Given the HTML above, our service worker will determine (based on the comments that start with the workertest: prefix) that there’s an A/B test in play called “mytest”. It will perform the same logic as the first simple service worker example with regards to dividing the visitors into cookied groups, but instead of a simple text swap, this worker will actually delete the HTML for the group that does not apply to the visitor and return the HTML without that markup, so group A will receive only the markup for the “donuts” link, and the opposite for group B.
So you can see how it works, here’s a playground page running that worker on a page containing that markup above: A/B Service Worker Example 2
More Work To Do
Permalink to 'More Work To Do'This post covered some ways we can use Cloudflare’s Service Worker support (or, servers workers if you’d like!) to move costly clientside A/B tests out of the browser to a middle layer where they can happen before the page is delivered. We think this tool is pretty neat, and given the standards-based nature of the code itself, we hope other CDN providers pick up on this and support it as well.
But more generally, the takeaway here is that we should be thinking about how to move these sorts of tests out of the browser, and not necessarily with Cloudflare’s worker support (which are just one tool), but by any means available. It’s often much faster to change the initial HTML on the server than it is to use many of the popular tools that do this in the browser, and thankfully many of these same tools offer ways to use them on the server instead. We should push for the workflow that doesn’t bog down the browser every time.
Thanks for reading, and if you have any feedback, feel free to get in touch with us on Twitter.