Vibes, Oxygen, and Big Wins: How To Judge The Success Of Your Data Team
For when someone inevitably asks about the ROI of the data team
Everyone Wants To Know
In every single data leader interview I've ever had there’s been a question about how I measure the success of the team.
If you’ve ever worked in data before, you know it’s not a simple answer!
Having led data teams for many years, and faced this question many times, I thought I would write up all my lessons learned over the years, and how I’ve developed a strategy that has worked for me and could be very helpful to others.
All the lessons in this article were developed through experience and mistakes.
When people ask me this question, I like to start with the following joke:
"It's like a meme, data teams are very good at judging other teams performance with data, but we are very bad at judging our own performance with data."
But really - how do you measure it? It's a valid and important question.
What Doesn’t Work
Precise ROI Calculations
Some data leaders do their best to try and justify exact dollar amounts and returns for every initiative. I think this is foolish, and frankly doesn't pass the sniff test.
Your "validated with the business $4 million of positive business impact added up across all our initiatives" isn't believable.
It’s like a meme, data teams are very good at judging other teams performance with data, but we are very bad at judging our own performance with data.
You might have a great data team, but your ability to take 20 different things you've done, put a dollar impact amount next to all of them in a spreadsheet, then add it up at the end of the year, is too specific about something you cannot be that specific about.
Ignoring It and Hoping Everything Will Work Out
If you ignore this, everyone will come to their own conclusions, and it will always be bad.
That’s right, EVEN IF you’re doing a good job, if you don’t help the business understand how to think about the data team, they will often come to the wrong conclusions.
This is because executing data work always has problems, and so if you don’t help people put those problems in perspective, you’ve got a high risk that people see those problems as evidence the team isn’t doing a good job.
Similar to if you didn’t realize that sales teams don’t win every deal, you might have a team crushing their numbers while some people say: “sure, but look at all these deals we lost, maybe our revenue should be 3X what it is right now!”
Vibes, Oxygen and Obvious Wins
Over the years I've developed my own strategy for to measure the success of a data team, which I call: "Vibes, Oxygen & Obvious Wins."
When you have these three elements working together, you can have a massively successful team where it’s clear as day to everyone in leadership that the team is driving significant ROI.
When you’re missing even one of these pieces, your team can be in trouble.
Terms
First I’ll define terms and then get into the details:
Vibes: Happy stakeholders.
Oxygen: Business literally can’t function without it
Big Wins: Wins so obviously valuable they justify the entire team on their own
I’m going to go out of order in explaining each of these, because it’s easier to understand if you start with Oxygen, move on to Big Wins, and then finish it all off with Vibes.
Oxygen
Some things the data team does are so obviously necessary that people would think you were insane if you started talking about getting rid of the data team.
For example, the other day my wife was going over an inventory report put together by a data person at her company in order to decide what products to reorder from their manufacturers.
There would be literally no way her company could continue operations if there wasn't this type of inventory report, and anyone who suggested that the company doesn't need that report would be insane.
Or imagine a financial company that didn’t have a reliable governed source of market data. You just can’t do it!
And anytime people wonder: “Do we really need the data governance team?” it’s clear the person asking just doesn’t understand the financial industry, because everyone else knows that in finance if your data is incorrect you can lose $440 million in 58 minutes.
When companies successfully integrate data into the core functioning of the business, data becomes like oxygen and there's no functioning without oxygen.
Big Wins
But serving as oxygen for the business to function isn't enough.
Because oxygen is only enough to justify data as a cost center that gets squeezed to the bone to provide that oxygen at the lowest possible cost.
That's why you ALSO need obvious big wins that you can point to as a data team to answer: "What have you done for me lately?"
As I wrote above, I don't like the spreadsheet with every data initiative and a "Share of value creation" metric next to it about how much value the data team created vs. the business team on a specific initiative.
Nobody believes those and I haven’t seen them be effective. It makes you seem naive as a business leader because everyone knows that the numbers are pretty shaky, and if you’re the data leader who is supposed to figure out how to make decisions with data, you don’t want to be leading the charge with shaky numbers.
What I’ve found DOES work are stories about how the data team was crucial to the success of something that actually moved the needle for the business at a level that makes it obvious the data teams is helping the company win big.
For example:
We're commercializing our data externally and making more money than the data team costs to operate (e.g. see Why Sell Data)
Without the data team putting together that custom analysis we would have lost our biggest customer, instead we got them to expand their contract
The insights marketing we're doing has built our brand in the market. The entire industry follows our work, and our thought leadership comes up in every single customer conversation.
The marketing optimizations the data team is doing helped us improve 40% on our digital marketing ROI
That customer journey analysis helped us reduce churn by 15% this year, finally bringing net revenue retention over 100%
etc...
Any one of these initiatives more than pays for the data team, and by consistently delivering these types of wins, people will want to keep you around and give you more resources to see what you can do next vs. try to squeeze you into doing the required work for the minimal budget.
And if your data team isn’t pursuing the types of projects that would provide these types of wins - that are so obviously big they pay for the entire data team - I think you should find some places to swing for the fences.
At a minimum, find the one place your company is already strategically swinging for the fences and make sure that the data team is highly engaged in making sure that initiative is as successful as possible.
Results Are Not Enough
Between the "oxygen" style obvious need for data, and the big wins that the data team continues to deliver - you should have a strong case for why the data team is doing a great job.
But that's only half of it.
Because data impacts the business THROUGH OTHER TEAMS - so what those teams think is an incredibly important factor.
I've previously described this as "Data Is The Sidekick" and the business teams are the heroes.
And it's tough to be a good sidekick if the heroes don’t like you you.
Enter Vibes
Which is where the Vibes come in.
Are the business teams happy with the data they're getting? Are they constantly pulling you to do more work with them vs. pushing you away?
If so, you’ve got some good vibes going.
I once put together this diagram of the virtuous cycle you get when the vibes are good and the data team is doing good work supporting the business:
In this case, the data team does good work, which helps the business teams be successful with data, which makes them want to pull more work from the data team which makes them more successful, and the virtuous cycle goes on.
You can generally tell when this virtuous cycle is off!
It manifests as:
Business doesn't trust data
Business doesn't use or care about data
Business is trying to avoid the data team or hire their own data people vs. rely on the actual data team
Data people are mad "the business just doesn't get us"
Data people build things that aren't used / important
Data people try to boss around the business people
etc...
When these things happen the vibes get all messed up, and everyone can tell that things aren't cranking the way they should.
You might have data as oxygen in certain ways. You might be able to point to some big wins. But if the vibes are off you can tell, and they cause things to get even worse!
Not About Perfection
From a data leader perspective, it's actually OK to have SOME of these bad things happening at any time.
Data work is hard, and if things aren’t always going wrong, it should be a major red flag that you’re not doing much :)
What makes the vibes good or bad though despite things going wrong is whether business leadership and data leadership are on the same page about making progress to fix these things if they're major, or everyone seeing them as minor issues to an overall highly successful data team.
As long as the virtuous cycle keeps going you are in good shape.
Feedback Welcome
I’m sure many data leaders have thoughts on this framework. Feedback is welcome, let me know what you think!