5 Myths About Data-driven Design

It’s not unusual for designers to omit data-driven design from their workflow, simply because of the unfair myths surrounding analytics and the way that data-driven design is conducted.

Psychology studies and UX studies are mistakenly thought to be sufficient enough as an informant, since the results are tested in a controlled environment and defined by large sets of data. But while the information is certainly useful (I mean, it’s fine to regard the results of user studies as best practices, for example), what’s missing from these experiments is the fact that the test subjects aren’t necessarily our users.

Without data collected from our audience, it’s easy to make assumptions or be without the other half of the story.

Let’s take a look at 5 myths about data-driven design that might change the way you look at using analytics for UX design.

Myth #1: “Analytics Aren’t My Responsibility”

Oh, hell no 🙅‍♂️. Data is definitely 100% your responsibility, regardless of what your primary role is in a team. If you’re a marketer, you need to know conversion rates to see if your marketing strategy is working. If you’re a developer, you need to know conversion rates to see if your code is bugging out somewhere. If you’re a designer, you need to know conversion rates to see if the user experience is optimal. What marketers, developers and designers all have in common is the same business goal of helping users convert.

Product managers have to care about all of those things, although each team member collectively has a responsibility to improve metrics using the skill and knowledge of their field.

TL;DR: analytics are everyone’s responsibility.

Myth #2: Data = Numbers

Data can be the heatmap from a user test, the answers from a customer survey, a scrap of feedback from a teammate. Anything that informs our decision-making is data. However, many of us make our first mistake when we assume that usability testing and surveying are alternatives to analytics, where in fact analytics lives and breathes at the center of all user research.

Analytics helps you ask the right questions for your customer survey, define the right user groups for usability testing, and identify where exactly the UX is falling short so that you can run A/B tests on those areas and find design solutions.

TL;DR: data isn’t just numbers, but analytics comes first.

Myth #3: Data-driven Design Requires a Finished Product

It’s very wrong to assume that we can only collect data on finished products. Product design today is leaner than it’s ever been before, with “shipping” becoming something of a spectacle thanks to discovery apps like BetaList and Product Hunt.

Nowadays, it’s not unusual to ship an app, website or feature multiple times as it naturally moves through its alpha, beta and launch stages. With each ship, data and feedback can be collected on a massive scale while even acquiring subscribers, customers and/or a social following along the way. These early adopters can be what make or break your initial success.

TL;DR: data can be collected at any time.

Myth #4: Data-driven Design Kills Creativity

Analytics tells you where the best opportunities are leaking out of your site, but it doesn’t outright tell you how fix them. Most of the time there are multiple winning strategies, and that’s where classic creativity comes into play. Creativity is needed to come up with effective solutions to turn a losing game into a winning game, using analytics to drive smart decisions. Here’s an insightful way to think of analytics:

Analytics puts you in the position of a football manager, allowing you to see things from a sideline perspective. However, it’s up to you to draw conclusions on how the game is progressing and find ways to turn the game around if needed.

So, while data-driven design doesn’t let you run wild, it doesn’t hinder your creativity either. It reduces the number of feasible options and tightens the boundaries, which is actually a good thing, because when there are too many directions to choose from, it can be hard to reach a destination (scientists call this analysis paralysis in their psychology studies).

TL;DR: data eliminates bad choices, not all choices.

Myth #5: Conclusions Are Objective

Data holds the clues, but it requires a detective to analyze those clues and solve the mystery of what your users are looking for. Let’s say, for example, that your website has a high bounce rate. What’s fact is that many users are entering your website and leaving without visiting another web page. What’s not fact is the assumption that users hate your site.

It may be that users find exactly what they’re looking for and leave satisfied within seconds (👌👌👌). Perhaps they googled a question, and your website answered it perfectly. It’s up to you to dive deeper and look into referral data, search terms and user demographics to try and identify what the user came to achieve, and then decide whether or not they achieved it.

Data is objective, but the conclusions drawn are subjective.

With analytics, it’s easy to see what you want to see. No data is impervious to bias, and even the most intelligent analytics software can’t explain exactly what your users want.

Now that you know this, you can be more consciously aware of what analytics can and can’t do. It’s a tool, not a solution.

To learn in-depth about UX Analytics, check out SitePoint’s book Researching UX: Analytics.