Roll Your Own A/B Tests With Optimizely and React
What are Experiments?
Experiments are site features that can be displayed to sets of users. The goal of an experiment is to better understand our customers, with an eye towards encouraging desired user behaviors. For example, if the goal is to drive signups from a landing page, you first measure how many customers hit the landing page and create an account. Next, you try different versions of that landing page, and see which one has the best conversions. By experimenting on your website’s traffic, you learn how to build a more effective customer experience.
Of course, getting the analysis right is hard. Variations in traffic and user behaviors need to be statistically significant, which sometimes means you need to run an experiment for days or even weeks at a time. Additionally, while sometimes it’s really easy to improve one metric (clicks on “sign up”) at the expense of another (clicks on “learn more”), that metric may not be the most valuable thing to optimize for your overall business. It’s really important to be driving user behavior that leads to the right long-term result. Because of these gotchas, experimenting can be a bit of a tricky science and it’s really important to make sure you’re doing it right.
To use Optimizely, you include a script tag in your document, which connects your site to Optimizely’s servers to determine which experiments to run. Optimizely allows non-technical people to modify the elements on a website through the Optimizely Visual Editor, and allows them to choose which of your experiments apply to which pages of the website. Using Optimizely, businesses can try out different ideas before they are ready to be baked in as a real part of the product.
We’ve found the Optimizely Visual Editor is best at small changes – updating a call to action button, or changing the copy of the website. We found a need for more full-featured experiments – those that create UI widgets or perform different actions as a result of user flows (creating or contributing to a Tilt, referring a friend to the site). To support these, we needed to build the concept of an experiment into our application, and then set up Optimizely to activate it.
An Experiment: Moving The Activity Feed On The Homepage
Normally the Tilt homepage is very sparse – a navbar, a logo, and a call to action.
We also have an activity feed of things that are happening on our site – what events people are pooling money for. For public campaigns, your contributions will show up in the global activity feed, so seeing what’s in the feed is a great way for people to understand examples of how they can use the Tilt product for their own needs. A few months ago we ran an experiment that was ultimately unsuccessful – what if we moved the activity feed above the fold on the desktop version of the homepage? Maybe that would spur more people to understand the product and click on the main “getting started” call to action.
This version of the page would look something like the following – activity feed on the left, call to action on the right:
Instead of using Optimizely’s Visual Editor to launch the homepage-feed experiment, we’ll build this experiment into our application. We’re going to use the Flux application architecture – an application architecture that’s different than traditional MVC web applications. A Flux application renders the page based on the contents of the stores. A store is a global set of data that’s associated with some concept in the application – some examples from our codebase at Tilt are a
UserStore (with data about the logged in user) and a
CampaignStore (with data about a crowdfunding campaign).
Here we’re going to render the page differently based on the contents of the
ExperimentStore: if the
You can think of our experiments setup as a series of lightswitches – depending on the configuration of the lightswitches, the page looks and responds differently. Optimizely has the ability to switch individual lightswitches on and off, immediately updating the rendered state of the page.
Looking At The Code: The Homepage
Like much of our site, the homepage is rendered through React.js. Although React lets you build components that have dynamic behavior through React’s event system, the homepage mostly just renders static markup. Here’s the code for the
Homepage component. Without any experiments running, this is the version of the homepage that gets displayed to all visitors of the site. Here we’re using a Bootstrap grid.
You can see that the Homepage mostly just calls out to other components. The
HeroContent component shows the “Collect money from your group and make something happen” copy, along with the Tilt logo and a “Get Started” call to action button. The
IntroVideo components are placed side-by-side with an extra spacer between them using a Bootstrap grid. The
Footer components are static components with links to other resources for people to learn about Tilt.
To make the homepage optionally render the feed above the fold, we add current state of all experiments to the homepage component’s state (as
this.state.experiments) and ensure that the component state stays in sync with the value in the store when the store changes. Next, we make the
render method conditional on the value of the
homepage-feed experiment from
this.state.experiments. The current state of all experiments is an Immutable Map, so the value of a key inside the map must be accessed through the
(In our production code, we use an
ExperimentMixin that manages the experiments map as part of a component state, both initially and on store update.)
Looking At The Code: The Experiment Store
At Tilt we use the dispatchr library, that provides a
createStore helper method to easily create stores that can be used in a Flux application. Using this helper, the
ExperimentStore has the following code:
Looking At The Code: Global Functions for Optimizely
To set up the experiment in Optimizely, we use
After running this experiment on 50% of our homepage traffic for a week we determined that the experiment was a failure – there was more engagement (more people looking on and clicking the feed), but less people were driven to the ‘get started’ call to action button. We disabled the Optimizely experiment and removed the homepage feed conditionals from our code.
Experiments As Feature Flags
To easily enable developer testing of experiments, we added some glue code so that developers and product managers can opt in to running experiments through a query parameter. By specifying the URL argument
?tilt-experiment=homepage-feed, the experiment
homepage-feed will be activated. We’ve also added logic so that a set of active experiments can be saved into a browser’s session storage, allowing for testing experiments that may span multiple pages – for example, showing the tagging lightbox immediately after campaign creation. Together, this allows people to easily share links that conditionally render different parts of the site.
This easily allows for our product teams to develop and test features on a variety of devices without needing to worry about long QA cycles on feature branches or in a staging environment. We recently released a feature where people can tag friends to their tilt – a lightweight way to let your friends know about cool things that they can help make happen (for example: a block party, a tailgate, a local concert). Rather than create a feature branch and release tagging to all users of Tilt at once, we shipped features daily to tilt.com for production testing behind a feature flag.
Next, we opted Tilt employees into tagging using an Optimizely audience to solicit feedback from the rest of the company. Finally, when releasing the feature to customers, we rolled the feature out for all users of the site, again using Optimizely – this way, if a critical bug were discovered in production, we could easily disable the feature. A few days after tagging went live we turned the tagging experiment fully on (by removing the
else branch of the experiment conditionals) and disabled the Optimizely experiment.
With past experiment systems I’ve used, logic for displaying different variations of the site was controlled by the server. In order to see the application with different features on or off, you’d have to update a configuration file or change a value in the database, then refresh the page. With React views listening to a centralized
ExperimentStore, it’s really easy to toggle between different the versions of the site – all without a page refresh. (While I’ve stuck to a frontend setup for experiments here, it would be straightforward to extend it to the server using React’s built-in server-side rendering.)
Our experiments infrastructure has been a big win for us in terms of simplifying our development process around conditional versions of the site. Additionally, by using them as feature flags, they’ve allowed us to easily test features in production prior to launch – a must have to release high quality software while avoiding long and costly manual QA cycles against unintegrated feature branches.
I’ve extracted the key concepts from this post into a standalone repository that I’ve published on Github – flux-feature-flags. This repository uses the standalone Facebook flux library and webpack to create a demo app with different render behaviors based on the contents of the
Finally, if you’re on your laptop, you can check out “Homepage Feed” experiment at tilt.com by following this link! (Because of the lack of horizontal space, the mobile homepage looks about the same when the experiment is on or off.)