The value of big data should be focused not on how much an organization collects but on how much it can use effectively. That’s making big data small. Barr Moses, co-founder and CEO of Monte Carlo, discusses the key questions to ask to understand an organization’s data maturity.
Everyone loves to talk about how much data there is, but the context is almost inconsequential. CEOs should work smarter, not harder, when it comes to becoming more data-driven. Somehow the amount of data in the Library of Congress has become a meaningful measuring unit – like football fields or the power generated by a horse.Â
When presenters pitch how data is the new oil, the takeaway is often that valuable data should be hoarded rather than the extensive infrastructure required to monetize it by extracting it from the ground, transforming it into fuel, and loading it for consumption.
The good news is the conversation is starting to change from a terabyte measuring contest to one more focused on the art of possibility.Â
Gartner predicts Opens a new window that by 2025, 70% of organizations will shift their focus from big to small and wide data focusing on contextual analytics and more intelligent machine learning. This trend from big to meaningful data requires us to shift our expectations and perspectives.Â
Four Steps to Data Maturity
Here are four ways CEOs can think outside the terabyte to determine the maturity and value of an organization’s data operations in the era of smaller data.
1. Instead of headcount, ask how much time your data and engineering team spends building vs. fixing
A flex that leaders like to trot out, especially in today’s tight labor market, is the size of their data teams. It can be seen as a competitive advantage as recruiting is the number one challenge by far I hear from data leaders.Â
According to DICEOpens a new window , tech unemployment dipped to 2.1% in October, and the tech job postings in Q3 of 2021 were up nearly 40 percent over the same period in 2020.
However, the more headcount on the data team, the larger the potential waste. One CDO I spoke with told me that his 20-person team spends 200 cumulative hours per week tackling data quality issues – that’s over eight days scrambling to understand why GOOD_DATA_USE_THIS_SPREADSHEET_V2.csv isn’t actually all that “goodâ€!
In my opinion, the efficiency of technical teams varies more widely today than at any other point in recent history. I’ve seen three-person teams with modern tooling and processes move from the ideation of a new algorithm to production in a week, whereas other companies measure this in months or quarters.
Before launching my startup, I spoke with more than 150 different data teams across industries and based on these conversations, I estimate that they spend 30-40 percent of their time handling data issues instead of working on revenue-generating activities. ForresterOpens a new window also puts this at about 40%.
So, instead of seeing the size of the data team as an asset, probe deeper to determine its efficiency–particularly how much time they spend fixing data quality issues versus building business value.
2. Instead of how much data, ask how much data is documented
Data teams have found that the data they collect is big, but the data the organization actually uses is small. The same key tables tend to get repeatedly leveraged while most quickly fall by the wayside.Â
Part of the transition away from big data involves teams focusing more on that small number of key tables rather than boiling the data ocean.
According to Gartner, by 2025, 80% of organizations seeking to scale digital business will fail because they do not take a modern approach to data and analytics governance. Data documentation is a big piece of that puzzle.
Without business context for your data, teams become inefficient in trying to figure out the context of a query or which table to use.Â
No team will have all of their data documented, nor should they, but a look at the documentation of their essential data assets is a quick way to size up the maturity and scalability of their operations.
See More: 3 Big Data Challenges for Manufacturers and How to Solve Them
3. Instead of the experience of the CTO, ask how many active data consumers there are
Don’t get me wrong, an experienced and talented data-driven leader can make all the difference in how an organization derives value from data, but it’s not the whole story.
Parallel to the transition from big to small data is the transition from big, monolithic data teams to a decentralized structure (often referred to as a data mesh) that prioritizes domain expertise. This allows the data team to stay close to the business, prioritize adoption, and ensure data operations remain use-case-driven.
4. Instead of the stack, ask about the data products
As technologists, we are all a little guilty of focusing on the new shiny features of the latest and greatest. Data teams, in particular, like to talk about the composition of their pipelines and modern data stack.
Another way big data is getting smaller is the shift in emphasis from the data stack to how it enables data products for both internal and external customers.Â
That shift has ramifications for technology (data reliability must be high), processes (codify expectations in data SLAs) and people (the emergence of the data product manager).
For example, as a former data product manager on Uber’s Product Platform, Atul Gupte led a project to improve the organization’s data science workbenchOpens a new window to enable data scientists and others to automate validating and verifying worker documents that were required when applying to join the Uber platform.Â
An external data product would be how point-of-sale providers like Square or Toast separate themselves from the competition based on their business insights to their customers. Through Toast, restaurants get access to hundreds of data points, such as how their business has done over time, how their sales compared to yesterday, and their top customers. The average cash register simply can’t compete.
There are several other examples of this type of “data product†across FinTech, e-commerce, digital services, and literally every other industry that claims to prioritize data. What separates the good data products from the bad ones, then, is a focus on the outcome, not the tools.Â
Size Does Not Matter
These complementary macro-trends illustrate that the data industry is starting to move in the right direction. Lightbulbs are starting to go off that it’s the use, not the collection, of data that matters.
And if that’s the point, shouldn’t companies start there – with the use cases? Is that perhaps a better way for executives and investors to determine how data and data teams add value?
Hopefully, focusing on the right questions will help us get the right answers.
Where are you on your path to data maturity? Tell us about your journey so far on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We’d love to know!