A Case for Server Side A/B Testing
After many years of conceptualising, managing and prioritising A/B tests (both server side as client side) I thought about sharing my experience. Most companies and clients with who I’ve worked with have been conducting A/B tests. Some more than others, but all of them understood the necessity of testing hypothesis.
Although understanding the “why” is an important one, I noticed a very different approach to the “how”. Time to list some of the dangers, common mistakes and best practices of client side and server side testing methods.
Flickering
When implementing an A/B test using a client-side solution, you will notice a white flickering on the page. This is called FOOC or “Flash of Original Content”. To understand where it comes from, you need to understand how client-side tools work.
Client-side tools create a variation of your test page by manipulating your browser through Javascript. You web server still sends the original, as usual, but the client-side tool pushes the changes into the visitor’s browser. Since those changes are not done on the server, but in your browser, they are called client-side tools.
So where does this flicker effect appear? It’s when, for a very short time, your visitor sees the original page before the variation is loaded by Javascript.
This flickering can be minimised to a few miliseconds with some hotfixes but it can never be completely taken out because of its technical implementation.
Impact on Loading Time
Almost all client-side tools come with a WYSIWYG or visual editor. That makes these tools so popular among marketeers for example, since you do not require much technical knowledge to set up a simple A/B test. Changing wording, different colors and different placement are all nice but what if you’d like to test an entire new feature or a redesign of your website?
At some point you will need to add javascript “to mess around” with the CSS and HTML. The bigger the change, the more code you’ll need to add to the page. This means an impact on loading time and added complexity.
Coding and Deploying
So now you have a few hundred lines of additional javascript (in most cases created by a front-end designer), your test variation comes out as the winner. You would like to deploy this asap, but your website is actually running on PHP. That’s right, you need to rewrite all of that code. This results in additional time to handover the knowledge to the back-end developer, additional time for rewriting that code, another run through QA (Quality Assurance) to check all use cases, … . It doesn’t seem very efficient to me.
What (not) to test
Static content can be tested with client side testing tools, such as Zarget or Visual Website Optimizer, as well as the server side solution. Dynamic content, however, should always be tested with the server side solution.
With server side A/B testing you can test, for example, database queries. You can optimise the algorithm used for search queries or test real personalised messages or banners. An improved algorithm means a better user experience.
User Restrictions for Client-Side Testing Tools
Remember that I said that client-side testing tools are very popular among marketeers? A long time ago, I started working as a UX Manager for a small startup. The marketing team was rather big, around 17 at that time, and all of them had access to the testing tool they were using. About 7 of them were actually implementing small tests, going from different online banners to button colors.
Can you imagine the results? What they had was not an online shop, but, what I call, an online color book. There was no real concept behind those A/B tests, no overall strategy, no prioritisation, no hypothesis, no QA, no decent reporting, etc.
“The danger of client side testing tools lies in its simplicity of setting up A/B tests!”
My advice: Limit access to just a few people, centralise its implementation or integrate a full team devoted to A/B testing.
Build your own A/B testing tool
Creating your own in-house server side tool can be tricky and expensive. Preparing the server and the load balancer isn’t so much of a problem, but decent reporting can be. I strongly advice to read more about statistics and algorithms of Bayesian Bandits.
Before building your own tool, think about the following. Some of the client-side tools, such as Zarget, Optimizely or even Google Analytics offer URL split tests. In a URL split test, the variations are hosted on different domains/sub-domains and traffic is split accordingly. These pages are being pushed by the server but the reporting is done by the tool itself. Once the results are in, change the target URL to the winning variation and that’s it.
In case you’re not conducting any A/B tests at the moment, but you would like to, then try an existing client-side tool to start with. This instead of building your own tool right away. I recommend using a tool that can be tweaked to your needs!
Conclusion
Client side and server side A/B testing can work together in harmony. Different tools for different needs and goals. Client-side requires less technical knowledge and is quick to set up. Server side testing is more robust, more flexible and quick to deploy but requires more time to set up and deploy.
How about you?
I would love to know what kind of A/B testing process you have in place. What are the advantages or where do you see room for improvement? Please share your experience!
Good post! We here at Optimizely have news that might warrant an update of the post: we recently released server-side SDKs for the most popular programming languages (with more to come) so that customers don’t have to pick between server-side and frontend / native mobile testing, but can use whatever works best for each experiment with the same valid stats that don’t allow for misinterpretation. If you’re interested, check out http://developers.optimizely.com/server/reference/index.html and let me know what you think about it.
I appreciate your article. I’d love to hear more from you or others about successful integration of server-side and client-side experimentation programs, or instilling experimentation into engineering teams. I’ve done client-side testing for years with a growth marketing focus, and goals around changing/aiding user behavior. Server-side experimentation is appealing to me as both a means to get experimentation into other digital channels, and for the capability to test technical hypotheses where success metrics would include system behavior as well as user behavior.