A Case for Server Side A/B Testing
After many years of conceptualising, managing and prioritising A/B tests (both server side as client side) I thought about sharing my experience. Most companies and clients with who I’ve worked with have been conducting A/B tests. Some more than others, but all of them understood the necessity of testing hypothesis.
Although understanding the “why” is an important one, I noticed a very different approach to the “how”. Time to list some of the dangers, common mistakes and best practices of client side and server side testing methods.
When implementing an A/B test using a client-side solution, you will notice a white flickering on the page. This is called FOOC or “Flash of Original Content”. To understand where it comes from, you need to understand how client-side tools work.
This flickering can be minimised to a few miliseconds with some hotfixes but it can never be completely taken out because of its technical implementation.
Impact on Loading Time
Almost all client-side tools come with a WYSIWYG or visual editor. That makes these tools so popular among marketeers for example, since you do not require much technical knowledge to set up a simple A/B test. Changing wording, different colors and different placement are all nice but what if you’d like to test an entire new feature or a redesign of your website?
Coding and Deploying
What (not) to test
Static content can be tested with client side testing tools, such as Zarget or Visual Website Optimizer, as well as the server side solution. Dynamic content, however, should always be tested with the server side solution.
With server side A/B testing you can test, for example, database queries. You can optimise the algorithm used for search queries or test real personalised messages or banners. An improved algorithm means a better user experience.
User Restrictions for Client-Side Testing Tools
Remember that I said that client-side testing tools are very popular among marketeers? A long time ago, I started working as a UX Manager for a small startup. The marketing team was rather big, around 17 at that time, and all of them had access to the testing tool they were using. About 7 of them were actually implementing small tests, going from different online banners to button colors.
Can you imagine the results? What they had was not an online shop, but, what I call, an online color book. There was no real concept behind those A/B tests, no overall strategy, no prioritisation, no hypothesis, no QA, no decent reporting, etc.
My advice: Limit access to just a few people, centralise its implementation or integrate a full team devoted to A/B testing.
Build your own A/B testing tool
Creating your own in-house server side tool can be tricky and expensive. Preparing the server and the load balancer isn’t so much of a problem, but decent reporting can be. I strongly advice to read more about statistics and algorithms of Bayesian Bandits.
Before building your own tool, think about the following. Some of the client-side tools, such as Zarget, Optimizely or even Google Analytics offer URL split tests. In a URL split test, the variations are hosted on different domains/sub-domains and traffic is split accordingly. These pages are being pushed by the server but the reporting is done by the tool itself. Once the results are in, change the target URL to the winning variation and that’s it.
In case you’re not conducting any A/B tests at the moment, but you would like to, then try an existing client-side tool to start with. This instead of building your own tool right away. I recommend using a tool that can be tweaked to your needs!
Client side and server side A/B testing can work together in harmony. Different tools for different needs and goals. Client-side requires less technical knowledge and is quick to set up. Server side testing is more robust, more flexible and quick to deploy but requires more time to set up and deploy.
How about you?
I would love to know what kind of A/B testing process you have in place. What are the advantages or where do you see room for improvement? Please share your experience!
Subscribe to the UX newsletter
Get the best UX newsletter, full of handpicked articles, resources and ideas delivered right to your inbox every second week.