All companies, large or small, face data challenges. Acquiring relevant information is no longer the hurdle it was ten years ago. Today, the challenge for business is how to integrate all their data and perform the analytics necessary to drive the business.
This can involve significant effort, particularly for the many companies adopting a code-centric approach to the problem. Let’s explore the problems associated with a code-centric approach to data integration, and present alternative strategies that will provide greater business value.
A code-centric approach is characterized by the necessity of having engineers develop custom software to solve the problem. This path is often taken by companies that have already made a significant investment in software engineering resources. Unsurprisingly, when software engineers are asked how to best solve a problem, they will tell you to write software. While this can provide a highly customized solution, it comes with several significant drawbacks.
Time Consuming and Expensive
Whenever an organization develops custom software, or even custom software components to modify or extend an existing framework, they are also shouldering the burden of software development and maintenance (see Figure 1 – Standard Software Development Life Cycle). Should the organization have a new analytic need, wants to integrate a new data source, or modify an existing data process, they must also take on additional software development work. In the best case, where an existing component or process is being modified, the required effort may be limited to a subset of the full phases of the lifecycle. However, creating a new component or process, and adding that to an existing system, increases the complexity of that system. This increase in complexity comes with an increase in the cost required to maintain and upgrade the system.
Figure 1 – Standard Software Development Life Cycle
An example of a code-centric approach that is frequently deployed in an attempt to solve data integration challenges is the Hadoop Data Lake. This approach gathers all the data as files stored in HDFS (Hadoop Distributed File System). In order to perform data enrichment or analytics, software engineers write Map / Reduce jobs. While there are many open source frameworks and commercial products available to make this process easier, ultimately, each time the company wants a new analytic or new data source made available, software engineers have to write, test, and deploy new software, spending valuable time and resources.
Lack of Transparency
Businesses need, as much as possible, transparency in the information technology (IT) processes which are driving business decisions. The data model in place within the analytic processes must be understandable to business leaders, and ideally should match the way in which these business leaders think about the company’s data and data processes.
Typically, in a code-centric approach, the data model is developed by engineers to conform to the technical infrastructure. This model is often misunderstood by non-engineers and contributes to a significant disconnect between the business and its IT systems. When an analyst’s conceptual understanding of the data differs from how it has been encoded into the IT systems, this can lead to incorrect analytic results.
Problems will also result when the data processes, inherent in the joining and analysis of multiple data sources, are not easily understood by the business. This is a common problem when custom software provides ETL, or other data matching behavior, in order to load data from disparate systems. When business leaders and analysts are unable to clearly understand these processes, it contributes to a lack of confidence in the overall result.
Summary of Code-Centric Approach
While custom code-centric approaches to solving data problems provide the most flexibility, it comes at a price:
- Time and cost of implementation are high
- Continuous maintenance of the solution is required
- Development is required for every new capability
- The complexity of the approach lowers the transparency of the approach to the business
Codeless Solutions
Fortunately, another path to solving data integration challenges exists. A growing number of low code and no code solutions now allows organizations to connect and analyze data without writing custom software. While codeless options do require Operations & Maintenance (O&M) funds to acquire the software, investments in human capital (software engineers) to develop code-centric approaches are not. The ability to deploy this type of solution also provides the organization with numerous benefits:
Near immediate ability to access the data for reporting and analytics.
Typically, the time required to install and configure codeless solutions is far less than comparable code-centric approaches, often saving months of time.
Codeless solutions are usually designed to be easily used and understood by non-engineers.
Increased visibility into data models and processes allows all of the organizational stakeholders to participate in a common understanding of the data flows. Within larger organizations, this common understanding can also serve to facilitate important conversations amongst business units in regard to how they each understand and use the business data better.
New data sources, reports and analytics can be integrated and created with minimal effort.
This significantly increases the organization’s agility, allowing it to react to new requirements at the speed of business. It also provides the business with more freedom to experiment with new solutions. The shortened time and effort to deliver new solutions reduces the cost of failure, allowing for greater experimentation.
Looking Forward
Developing custom software to solve mid to large scale data challenges including integration, reporting, and analysis, has been, and continues to be, a common approach. Unfortunately, this solution is costly and burdensome. Recent advances in technology have led to growth in new low code and no code data solutions. These solutions tend to be significantly easier to deploy and maintain, while simultaneously offering additional benefits to the organizations in which they are deployed. Companies looking to tackle their data challenges could benefit through serious investigation of this new category of solutions.