Your enterprise has invested in myriad analytics tools. You have capable data scientists and analysts on board. You believe in the power of letting your data guide your business. You think you’re making data-driven decisions. Then you see these decisions go awry. Michael Amori, CEO and co-founder at Virtualitics, discusses how to find out if your organization may still be guilty of decision-making by hunch and strategies to remedy that.
The amount and complexity of data that’s now collected and analyzed is too vast for even experienced data professionals to sort through. So, they resort to what’s worked passably well in the past: They create a hypothesis to test.
Further, this educated guesswork is often skewed toward the highest-paid person’s opinion (HIPPO). Foundational questions get formulated around whatever company leadership believes to be a priority. Right out of the chute, you may be trying to answer the wrong questions.
Guess-and-test approaches to data exploration leave out important, unexamined insights. Here’s how to avoid assumptions that take you in the wrong direction and better leverage AI to follow where all your data is actually pointing.
Avoiding Bias in Data Projects
Bias and best guesswork are closely linked. Take the example of HIPPO bias. An executive may be convinced any productivity dips during COVID-19 were due to the workforce not being in the office. Any analysis done at this executive’s request would be narrowed before it’s even underway because the data analytics team keys in first on the exec’s preconceived notion. They may miss factors that are actually more important: the engagement impact monitoring software has on employee productivity, childcare challenges, and remote work technology difficulties. And instead, investigate a very narrow scope to justify a policy requiring a return to in-person work.
The first step in avoiding bias is recognizing much is hidden in the huge universe of data available. The second step is in knowing what we choose to investigate and also injects bias. The solution is to acknowledge humans can’t really begin to figure out what’s going on within modern datasets without the help of AI, used for exploring data objectively up front and drawing our attention to what matters.
Intelligent Exploration Leaves No Data Behind
Think of a first responder examining an unconscious accident victim who can’t say what hurts. The medic looks at obvious trauma, yes, but they don’t stop there. If they did, they might overlook a hidden injury that poses an even bigger threat to the patient.
This is what intelligent data exploration looks like. It allows you to be sure you’re looking at all important contributing factors to a scenario, even when some are far more obvious than others. It also ensures your personal observations aren’t limiting your scope. Back to the first responder example, they don’t limit their assessment to what they have the most experience with or have seen most often. They approach each patient assessment with fresh eyes.
To illustrate with a supply chain example, say a large e-tailer is concerned about delays in shipments of certain goods. There could be hundreds of metrics to consider to discover why some deliveries are on time while others aren’t, everything from shipment and destination locations to weight, package dimensions, type of product, time of year, cost, and more.
A data analyst might think, â€œI’ll start by looking at origination locations and follow that path; maybe India is underperforming China.â€ If the hypothesis turns out to be correct to some degree, the analyst might stop there.
But there may be a factor impacting delivery delays that is, in reality, more important. Intelligent exploration provides open-ended inquiry at the front end of the analysis and uses AI, not someone’s best guesses, to make sense of complex datasets and process them into understandable interactive 3D visualizations.
These visualizations are capable of revealing all the connections going on in a dataset. They can incorporate up to a dozen different dimensions of data, all types of data (categorical, numerical, IoT-sensor-based, etc.),Â and literally hundreds of millions of individual data points.
This allows the supply chain analyst to discover groups that are experiencing more delays than others, with AI describing the differences between the groups in natural language. All the factors associated with the most delays â€“ packages originating in India that used a certain shipper, were of medium size, weighed between eight and 12 pounds and were destined for Europeâ€”are apparent. And it turns out package size is a bigger factor than shipping origination point.
Decision makers in this example now know to make solutions for larger packages a top priority, not something that a hunch or limited data exploration would have revealed. By looking at all the relationships in the data, they know where to intervene.
Removing Bias Is Key for Truly Data-driven Decision-making
There’s a natural inclination to gravitate toward what we already know. But limiting what’s considered at the outset of a data analysis project is risky. It leaves out insights that can incorrectly confirm biases. It can fundamentally change how you understand business problems. Data can appear outsized because other data providing context has been excluded. It can lead you to act in the wrong direction.
Intelligent exploration is the antidote to guess-and-test. It leverages artificial intelligence not to supersede an analyst or SME, or CEO but to lead them to consider new ideas and challenge assumptions. With more thanÂ 2.5 quintillion bytes of dataOpens a new window Â being generated each day, this automation is the only way to make sense of data in all its complexityâ€”and achieve true DDDM.
Image Source: Shutterstock
MORE ON DATA-DRIVEN DECISION-MAKING
- Top 6 Analytics Trends To Drive Data-Driven Decision Making
- Decision Intelligence and the Future Beyond BI
- How Real-time Analytics Creates Operational Intelligence and the Future of Data-driven Decision-making
- Why AI Should Not Be Compared to Humans in Regard to Decision-Making
- Overcome the Data Decision Gap With AI and Automation