Could you be drawing unsupported conclusions and making suboptimal business decisions because of your data? Sound familiar?
A department head schedules a meeting with their team and other lateral department heads. After thanking everyone for coming, the organizer dives in. He begins pulling up dashboards with various metrics, ostensibly pointing to a departmental goal. He points out a few statistics and arrives at the damning conclusion: “Based on the data, we need to double down on live events, incentivize the sales team to push more of our flagship product, and 5x the amount we’re spending on paid ads.”
But there’s a problem: That’s not what the data says at all. If he had dug 2 layers deeper (yes, exactly 2 – I’ll explain in a moment), this decision-maker would have realized a few things:
- The company has maxed out their ROI from live events. There are no more relevant events in their industry to attend. If they spend more on live events, the ROI will decrease drastically.
- Selling more of the company’s flagship product will bring in more short-term revenue, but it risks devaluing the brand and removing focus from more important, longer-term sales goals.
- Paid ads have been working because the company has dialed in their re-targeting efforts. They’re still having trouble converting on cold traffic. So any increase in ad spend will likely see a correlating decrease in conversion rates.
Way too often, we’re guilty of relying on data to make decisions for us. And who can you blame us? Data is sexy. It’s tangible. It makes it easy to support our argument. But without context, we can’t replicate the successes or fully understand the shortcomings that the data conveys.
People can find data to suggest anything if they ask the wrong questions and extrapolate conclusions, often missing the real value of early traction and customer growth. As valuable as customer-generate revenue is, the real value of early customers lies in the feedback they share to help us scale a viable business. As Brendan Dell says, “Most companies aren’t playing to win. They’re playing not to lose.” The risk aversion often overshadows our ability to create meaningful experiments and draw actionable insight from the data we collect.
Why Data-Driven Decisions Often Lead Us Down The Wrong Path
Imagine you’re planning to build a house. What’s the first thing you would do? My guess is that you wouldn’t head out to the home improvement store for bricks and concrete mix. You’d likely create a blueprint to help guide your building efforts, because if you build without a clear blueprint, your house probably won’t be standing for long. But before that, there are a few other things you should do as well:
- Research the landscape (How deep can you dig before running into rock or water?)
- Determine weather patterns (Do you really want to put those windows on the East side of the room?)
- Identify local regulations, so you can build to code (Wouldn’t want the local government dissolving your efforts before you even hang your drywall!)
When it comes to building a go-to-market strategy for your business, you want to do something similar. While it might be exciting to start pulling data and getting to work, doing this too early in the process risks building upon a shaky foundation. Consider Josiah Warren’s warning: “It is dangerous to understand new things too quickly.”
The fact is, while we like to proclaim to the world that we’re data-driven, the reality is that many of us can be assumption-driven…with data support. What’s even more challenging is that we often have no idea that our assumptions are driving us.
And a strategy built on assumptions is one likely to crumble in the face of a brisk wind. Or worse – it might hold up long enough to run its course. And after the company has spent tens of millions of dollars, the team realizes the strategy was flawed from the beginning.
Even big brands aren’t immune. Consider this confession from Adidas’ global media director in regards to the company’s over-investment in digital advertising: “We had a problem that we were focusing on the wrong metrics, the short-term, because we have fiduciary responsibility to shareholders.”
Even if Adidas was showing short-term growth (and I’m not sure if they were), they were doing so at the expense of longer-term brand-building, because they failed to pay attention to the right metrics.
But what are the “right” metrics? And how should you use them effectively? I recommend what I call “The Reverse Blueprint Model.”
It’s surprisingly simple, and when used correctly, will help not only identify the right metrics for your business to track, but also help you effectively use those metrics to drive better business decisions in both the short- and long-term.
How The Reverse Blueprint Model Can Help You Make Better Data-Driven Decisions
Using data to help your business instead of harm it starts with The Reverse Blueprint Model. It’s broken up into 3 parts.
- Understand the Market
- Build the Strategy
- Use Metrics as Support
- Understand the Market
You can understand a market from two types of resources: primary and secondary.
Understanding The Market: Secondary Resources
Interestingly enough, secondary resources are what you should examine first, before even beginning to develop your strategy. Secondary resources include:
- Size of the market
- Market momentums
- Willingness to spend (including number of people in the market and the market’s gross revenue)
The phrase to remember here is: “Where before Share.”
Where you choose to position yourself in the market is much more important than the share of the market you’ll be able to dominate. Here’s a basic example:
Imagine you’re comparing two markets. One is a $2 billion per year market. The other is a $1 billion per year market.
In Scenario A: you’re able to take up 50% of the $1b market.
And In Scenario B: you’re only able to take 30% of the $2b market.
In Scenario A: you have an annual revenue potential of $500,000,000.
In Scenario B: you have an annual revenue potential of $600,000,000.
Now, which one would you go for? I’m opting for Scenario B, because even though there’s less market share there’s much more earning potential. Here’s the point: You have to understand the broad context of the market you’re entering into. This will help you identify which metrics are important to measure.
Understanding The Market: Primary Resources
Primary resources are where you get qualitative data, which is vital to understand the context of the quantitative data you’ll get in Step 3. Primary resources include:
- Open-ended conversations with customers
- Survey responses
- Anything else?
But again, context is key. If you want to glean as much valuable information as possible from these primary resources, here are a few things to keep in mind: When performing customer interviews for market research, say as little as possible…and avoid focus groups.
If an interviewee knows you work for the company, their responses will be biased toward the positive. And focus groups aren’t reflective of how people actually buy. (Unfortunately, they tend to be biased toward the loudest person in the room.)
Be Deliberate In Choosing Interviewees
It’s important to understand the difference between demographics and psychographics here. Let’s consider a relationship therapist for example.
A young, married couple living in the city between the ages of 30 and 35 might share similar demographics, but consider this: If one couple is considering divorce and the other is looking for ways to handle an upcoming move healthily, their needs and psychographics differ wildly even though their demographics look the same…
To help combat this, I recommend segmenting the market 4 times, and then another 4, so the psychographic behavioral profile pares down characteristics a total of 16 times.
Use Your Interview Insights To Create A Survey
Once you’ve performed your interviews (In my experience, I’ve found around 40 interviews to be the sweet spot), it’s time to analyze them, identify themes, and create a survey to use with a larger audience.
The purpose of this survey is to ensure that you’re not extending the viewpoint of early adopters or interviewees to the general public.
Build The Strategy
At this point, you’ve done the foundational work to understand that market. Now it’s time to put together your strategy based on that qualitative data. Too often, metrics and quantitative become the reason for a strategy, instead of support towards a strategy. And this can lead to serious problems down the road.
For example, back before Blockbuster bit the dust, they had concrete evidence to suggest users would never prefer streaming over the in-person rental experience. “Neither RedBox nor Netflix are even on the radar screen in terms of competition, “Jim Keyes, CEO of Blockbuster in 2002. Of course, this is laughable now. But remember, at the time, Blockbuster had concrete evidence to support this statement.
Another example was Sony’s pre-digital domination. To Sony, digital seemed like a passing fad. And they had the data to support it.
When their engineers pointed out that digital audio compresses and reduces sound quality, they set about asking the market:
“Do you care about audio quality?”
“Well, of course!” the market responded.
With that, Sony was able to come to the conclusion that digital was just a fad. If the market cared about sound quality, they certainly would not jump on the digital bandwagon and sacrifice their precious sound quality!
But what Sony failed to realize was this: While sound quality was important to the market – convenience was more important.
Sony asked the wrong questions and pieced together a narrative driven by availability bias and their own assumptions. And they didn’t pick up on what now seems like an obvious market trend.
So what’s the solution? Build your strategy based off of your qualitative understanding of the market and the priority of its customers. Then, bring in the data.
Use Metrics as Support
Mike Duboe of Greylock Partners, says: “Before launching any feature, experiment, or initiative, it’s important for the lead to clearly articulate & quantify “what success looks like” – and then ensure it is possible to measure success, before going live. Any metric in isolation can be gamed – pairing metrics (e.g. monitoring cohort retention alongside acquisition volume) can help mitigate.”
So, based on the strategy you developed in Step 2 – what does success look like? What metrics would actually indicate growth toward your desired goal? There are some important questions to consider here:
- Who is going to use this data?
- How will they use it?
- What will they use it for?
- Is it actionable?
Answering these questions will help you choose metrics that actually help drive toward your long-term goals – rather than away from them.
There are, however, some metrics I advise against using to measure success. In general, I’ve found these metrics to be meaningless at best and misleading at worst.
- Cumulative charts
- Registered users or total downloads
- Use DAU/WAU/MAU and cohort retention instead
- “5 Year Roadmap” or revenue projections 3+ years from pre-launch
- Extrapolating cohorts across unlike segments
- ARR, MRR, or CARR for a non-subscription product
- Use “annual revenue” instead
- Direct or organic traffic
- Blended CAC
The Meaningful Metrics Litmus Test
If you want to determine whether you’ve chosen metrics that are actually meaningful, I recommend a quick litmus test:
Ask yourself “How do you know that?” and then “why is that important?” for each metric.
For example, let’s say you’ve decided that the conversion rate on a free trial opt-in page will indicate “success” for your strategy.
How do you know that?
“Because our #1 goal is to grow the number of active free-trial users.”
Why is that important? “Because the more active users we have, the more people we can convert into paid members.”
Keep this context close whenever analyzing the data.
It’s not enough to observe numbers on a dashboard in isolation. Zoom out to understand the qualitative circumstances surrounding the quantitative data being analyzed. These two gauges will work in tandem to keep your strategy focused and effective.
Data Should Support Decisions, Not Make Them
When used in a vacuum, quantitative data can drive worse decisions than a lack of data altogether. It’s way too easy to draw unsupported conclusions based on what “the data says.” Especially now, it’s important not to extrapolate shifting customer trends into predictive forecasting, as it’s uncertain which will persist after we’ve contained the COVID-19 pandemic.
That being the case, it’s more important now than ever for us to learn how to use data as a tool to support our strategy, rather than a crutch to retrospectively support our decisions.
Stefania Olafsdottir, CEO and cofounder of Avo, puts it this way. “I believe data is not the same as information, [but] most people don’t realize that. So we’ve done “big data” for a while, but are now entering a wave of quality over quantity.
That means we need to think through how we will use the data, before we start gathering it. And if we don’t, we’ll be piling up analytics debt.”