Business Intelligence (BI) software was meant to solve a common problem: How can an enterprise benefit from an analysis of its current and historical data without requiring expertise in statistics, query languages and specifics of physical data access? Have these tools delivered on their promises, and, if not, what were the underlying issues?
Current Business Intelligence Tools
Companies rarely throw data away, especially if it relates to profit-generating operations. As their historical databases grew into enterprise data warehouses and then into big data applications, the critical question was always, “How do we analyze this information to generate actionable intelligence?†It started with power users having knowledge of statistics and query languages querying the data warehouse or big data applications.
Then new applications and new data types appeared, along with growth in customers, products, services and transaction volumes. These issues made analysis more complex. Companies saw the need to employ data scientists, specialists who had both knowledge of the business data as well as analytical and statistical techniques. As data and applications grew, understanding all of the various data associations, relationships and correlations became increasingly difficult.
The answer was BI. Many vendors created BI software solutions for business users that provided graphic methods of visualizing data and data relationships. Users could request analysis of business data from multiple data sources including on-premises, external data and cloud. The software would generate the appropriate queries against data sources, merge, match and sort the intermediate results, and finally aggregate and subset the final results for the requestor.
Learn More: Bolster Your In-House Talent to Meet the Growing Shortage of Data Scientists
Business Intelligence Software Issues
As BI software matured, several common issues arose that reduced the benefits of BI. These issues fell into the following categories.
- Know your data. BI software can sometimes deduce data relationships from database structure definitions such as primary keys. However, most business rules and relationships must be entered into the BI software. For example, what is the difference between the CurrentCustomer and ActiveCustomer files, and which one(s) refer to customers that have ordered products? This requires that someone (the BI user, the data scientist, the enterprise data modeler) define data elements and relationships to the BI software; otherwise, analysis is impossible. Hence, you must know your data and define it properly to the software in order for it to work properly.
- Know your value proposition. Many first-time users of BI software acquired it in order to analyze specific problems. This “low-hanging fruit†provided the justification for software purchase and user training. However, after solving this first set of problems, how will the BI software be used? What categories of business problems in your organization will generate correlations, trends and predictions that will either generate profits or reduce costs? Will you analyze customer behaviors, sales data by geographies, prices and shipping costs?
- Know your sizes, speeds and feeds. Most BI users begin with relatively simple analyses of a few data sources. Eventually, interesting and useful analyses graduate from one-time queries to regular reports. The number of BI users increases, data volumes grow, query complexity grows and historical data accumulates. Clearly, you must coordinate with your IT support organization to do resource capacity planning for your BI software, data and associated hardware. BI queries will consume more CPU cycles and intermediate data storage, and you should plan for this.
- Users or applications. Most initial BI data analysis is done on historical data looking for patterns and trends. This kind of data usually resides in a data warehouse or one or more data marts, perhaps even in the cloud, and is typically executed by a data scientist. Initially, queries are ad hoc, as the data scientist searches for correlations. Eventually, one or more BI queries will emerge as being extremely useful for inclusion in applications, sometimes even transaction processing applications. One example is an on-line customer-facing application that executes a BI query to determine the probability of financial fraud. BI queries in applications need to be reviewed for potential performance issues, especially if they access current production data.
Learn More: Safeguarding an Essential Business Asset: Enterprise Data
Are BI Users Satisfied?
A new Business Intelligence Emotional Footprint ReportOpens a new window from SoftwareReviews found that 54% of users say they are dissatisfied with BI vendors who over-promise and under-deliver.
Why are so many users dissatisfied? Are the issues primarily defects in the software, issues with data, or possibly issues with users themselves?
As Igor IkonnikovOpens a new window , Research & Advisory Director of Info-Tech Research Group noted, one of the biggest implementation challenges for BI users is that, “Everyone wants an ‘improvement’ – very few stakeholders can formulate exactly what that looks like or how that improvement could be measured …â€. This is a combination of knowing your data and knowing your value proposition.
Similarly, Ikonnikov points out that the BI software market is saturated with drag and drop tools. “Drag and drop is not everything that a user expects from a BI/Analytics tool – they need to accomplish a business goal: provide a new insight, find an opportunity for improvement or a new value creation.â€Â
The tool, therefore, should help the user from understanding the data, through iterative solution creation, up to finding the proper visualization for their findings. And in the process, the users expect the vendor to provide answers to their questions or help them overcome technical challenges, adds Ikonnikov. Â
Best Practices for Business Intelligence
How can organizations avoid the traps mentioned above? Here are some best practices for building out in-house solutions that include BI software.
- Accommodate different BI query speed lanes: Some queries are part of mission-critical applications (think of the financial fraud example above, or real-time monitoring tools). The data must be of high quality (no missing values, internally consistent, and so forth), and you may need to dedicate high-speed and high-volume resources to ensure that queries execute quickly. Other queries in less critical applications or ad hoc queries by BI users are lower priority, and can usually run on slower hardware against lower quality data.
- Copy data from source production systems into a staging area: Data staging allows you to avoid direct access to source data (possibly locking the data or negatively affecting production performance). Data can then be cleansed and pre-aggregated in the staging area.
- Move staging data to multiple data marts based on business needs: Some BI analysis may involve sophisticated statistical analysis such as cubes and rollups. Others may only need specific subsets of data based on limited time-series or geographies or other dimensions. Some data relationships may be in the form of star schemas or snowflakes that may then require specific query optimizations in order to get good performance.
Learn More: 5 Big Data Analytics Trends for 2020
Selecting BI Tools
You may find that multiple tools meet the needs of different BI communities. Indeed, it is possible that you may need several tools. Ikonnikov advises that you should, “Start with your major use cases and usage patterns – do not feel embarrassed if you realize that more than one tool is needed, e.g.: one for Data Scientists, another one for Business Analysts, and, possibly, a real-time dashboarding tool working off of streaming data – this is typical for many organizations.
Striving to reduce the number of vendors is, of cause, an excellent practice, but BI & Analytics might be a good exception to this rule – since this capability is expected to save your company money by identifying operational improvements and yield new money by creating new business opportunities (conversely, trying to coerce drastically different usage patterns into a single technology might eventually cost more time and, consequently, money).â€
 Another area to consider is coordination with your other software tools such as enterprise data model, monitoring tools, and data movement tools. You may already have these tools in-house from a variety of vendors, and it is sometimes difficult getting multiple software packages to work together effectively.
 Ikonnikov also points out that a wide variety BI tools exist having different combinations of functions for different categories of users. “Some tools are designed to be extremely business user friendly, but may lack the sophistications of other categories … [other] tools are designed to serve technically savvy users who don’t mind writing code to solve problems of high degree of complexity … [others] are designed to be “all-in-one†platforms with built-in data preparation pipeline capabilities … [and some] have very narrow specialized applications, but require minimal effort to create reports and dashboards in their specific niche,†he said in closing.Â
Are you satisfied with your BI tools? Comment below or let us know on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We’d love to hear from you!