Optimizely A/B Testing Tools: Knowing Which is Right for You

Optimizely has grown to be one of the major A/B testing tools. It’s suitable for companies of all sizes — from startups and small businesses to major corporations. Optimizely’s tools and integrations allow anyone to quickly learn more about their audience and experiment with design and content changes non-destructively through A/B testing. With A/B tests you can confirm, for example, which CTA (call to action) leads to the most clicks, or which version of a checkout leads to the most online sales.

Optimizely also integrates with mobile apps and TV apps, meaning that you can analyze metrics and improve conversions on virtually any platform.

It sounds ambitious (and maybe even a little complex), but you can start out small with their simple visual editor, which lets you experiment with fairly simple A/B tests (no code necessary). When you feel you’re getting the hang of Optimizely (it’s actually really easy to use), you can integrate Optimizely more natively using their many SDKs. In this article, I’ll summarize the tools Optimizely offers and identify a suitable scenario for when they could be used.

Since it’s an ambitious tool with a huge offering, let’s break down the features.

What You Need Before Using Optimizely

While there are no official requirements for using Optimizely, you should already have your KPIs established beforehand. A KPI (key performance indicator) is a metric that you’re tracking to measure success — such as the number of user registrations, sales, newsletter signups and so on. In short, what are your website’s goals? Have them noted down.

(If you need some background to website goals, analytics and more, check out our recent articles on using analytics to improve UX, such as our introduction to Google Analytics.)

Your website (or app) should also have a reasonable amount of traffic, so that the data collected is informative enough. I would also recommend a minimum of 1000 visitors over 1–2 days on the page you’d like to optimize, to justify the costs of Optimizely services. If you’re not quite at these levels yet, consider using Google Optimize as a free alternative.

Optimizely offers a standard 30-day free trial for all of their services, so that you can experience the benefits for yourself and learn how to use it like a pro before investing any money. Like I said, to make the most of the 30-day trial, have your KPIs established and experiments thought out beforehand, to make the most of each free day.

Okay, so let’s dive into the features now.

Web Experimentation

Web Experimentation is essentially a visual editor for creating A/B tests on your website. With the editor, you can make content, color and layout ordering changes to create one or more variations of a web page.

Then, using the audience builder, you can target specific segments of your audience based on geolocation, time, gender and more. With these powerful targeting features, you can then test your variations on a very specific audience. The results are fed into a live reporting system, which gives you detailed feedback on all of your variations, to help you identify the one with the highest conversions, which you can then implement permanently.

Changes to code are not required, and your original version is not disturbed.

Web Personalization

Optimizely’s Web Personalization tool is similar to its Web Experimentation tool, but with a focus on A/B testing content rather than visuals, where the aim is to display content that’s relevant and interesting to the audience.

For example, consider the home page of a sportswear store. Visitors with a user persona matching that of a hiker could be shown a mountain-view background with the latest hiking shoes. To take this a step further, you could show hikers in colder locations thermal clothing and a snowy scene. With Optimizely’s exceptional targeting features, the options are endless.

By tailoring content to audiences based on their location, current weather conditions and more, your business can maximize sales and stand out from the competition by giving users exactly what they’re looking for.

Web Recommendations

With the Web Recommendations tool, Optimizely uses intelligent machine learning software to show the right things at the right time, to maximize engagement and conversions by displaying the content and products that users will likely want to see, based on a variety of factors. This recommendation engine allows you to experiment with algorithms, the placement of the recommendation widget, and the visual design of it.

Considering that Amazon’s recommendations engine is responsible for over 35% of Amazon’s sales, you’d be astounded by how useful this tool is!

Optimizely for Mobile Apps

Optimizely’s mobile tools allow you to integrate your A/B experiments directly into native mobile applications, using code. It encompasses a lot of Optimizely’s web features, but the focus on code-based experiments over the use of a visual editor means that you can instantly publish app updates without having to wait for approval from the iOS/Android app store.

Experimentation is full stack, so A/B tests can span both front end and back end, using any of Optimizely’s mobile SDKs (Android, Swift, Objective-C).

Full Stack (for Deeper Experiments)

Speaking of SDKs (and code-based experiments), Optimizely allows you to create your A/B experiments (for both web and mobile) using any of the programming languages we use today, including the three mobile app languages mentioned above. Aside from those, there’s also Python, Java, Ruby, Node, PHP, C# and JavaScript SDKs, which are all open source.

Regardless of what language your app or website is built in, Optimizely can be integrated into it, giving your team more control and total customization when compared to tests made with the visual editor. As mentioned before, you’ll be able to experiment with much more than visual and content changes.

If you were building a tool or feature for SaaS startup, you could create two entirely different versions of it to see which converts better.

Full stack integrations hook into the same targeting engine that Web Personalization uses, so you can utilize that same power to target audiences at a high level while having much greater customization within your website.

OTT (for TV apps)

Optimizely has created an OTT tool to allow creators of TV apps to A/B test variations across various OTT platforms such as tvOS and Chromecast. OTT stands for “over the top”, which means delivering video over the internet without the need to subscribe to a subscription service such as Sky or HBO.

Optimizely’s new OTT tool allows designers and marketers to experiment with visuals and content for different TV platforms — for example, showing a slightly different theme or layout for Apple TV than Vimeo or YouTube, to ensure that the visual experience looks and feels like an Apple experience.

Like with the mobile SDKs, instant feature rollouts mean that users will be shown the latest variations and experiments without needing to update their TV application, meaning that you’ll get valuable feedback instantly.

Analytics

Regardless of what Optimizely tools you’re using, the Analytics Statistics Engine is there to help you evaluate results. Traditional statistic engines like Google Analytics only give relevant results upon experiment completion. This means that if you look at your results mid-way through an experiment, they may be inaccurate by up to 30%! This can lead you to preemptively make the wrong design decision and end your experiment way too early.

Optimizely’s engine uses a new algorithm to show you live statistics mid-experiment, with an error margin of only 5%. This means that you’re much less likely to make the wrong decision regarding your experiments.

Use detailed reporting to evaluate your experiments’ performance. These reports utilize the unique analytics method created by Optimizely to deliver the most accurate results as quickly as possible, even mid-experiment.

Conclusion

Optimizely is really pushing the boundaries, providing A/B testing tools to markets that previously didn’t have them (such as OTT) while revolutionizing analytics and reporting to be more informative than ever before.

It’s suitable for roles of all kinds — designers, developers, marketers, managers, and all combinations of those working together. Optimizely ensures that whatever your skill level, there’s a tool that allows you to experiment with A/B testing for your app or website.