Optimization of the total cost of ownership (TCO) has become extremely crucial, especially in the current scenario. This article by Joydeep Sen Sarma, co-founder and CTO of Qubole, highlights the effective use of cloud data lake platforms to address the rising concerns of organizations towards reducing TCO.
In today’s business environment, companies around the world have prioritized cost reduction and optimization. In data organizations, cost optimization has also been added to the objectives and key results (OKRs) of data leaders and data professionals. While cost has always been important, the time has come for data platforms to lead the way in helping data organizations manage this key business lever.
When it comes to data platforms, the time for data lakes has truly come, not just for helping companies optimize TCO, but also to help them build a modern data architecture as they scale up their analytics and machine learning initiatives. The rise of data lakes can also be seen in market projections where the data lake market is expected to clock a 27% CAGR across the 2019-2024 period.
Learn More: Don’t Let Your BI Tools Limit the Value of your Data Lake
What Is a Data Lake?
A data lake is an architectural pattern for collecting and storing data in its original format, and in a system or repository that can handle various schemas and structures until downstream processes need the data. The primary utility of a data lake is to have a single source for all data in a company — ranging from raw data, prepared data, and 3rd party data assets. Each of these is used to fuel various operations, including data transformations, reporting, interactive analytics, and machine learning. Managing an effective data lake requires systems for ingestion, organization, cataloging, and governance of data.
Enterprise data lakes are most commonly split into two general industry trends: on-premise data lakes and cloud-based data lakes. In an on-prem data lake, companies must manage both the software and the hardware assets that house their data. If their data volume grows beyond the capacity of the hardware they’ve purchased, companies have no choice but to buy more computing power themselves. In cloud data lakes, companies can pay for only the data storage and compute they need. They can scale up or down as their data requires. This scalability has been a huge breakthrough in the adoption of big data, driving the increased popularity of data lakes.
Learn More: How to Choose the Right Platform to Manage Your Data?Opens a new window
TCO Optimization Capabilities of Cloud Data Lakes
Enterprises run their interactive analytics, streaming analytics, and ML use cases in public clouds as cloud data lakes provide significant cost advantages, agility, and scale from the get-go. Proof of concept (POC) for data-driven initiatives starts easily and without any huge upfront bill. But over time, as projects mature or ad-hoc queries take long or model iteration cycles increase, there is wasteful expenditure on computing and resources due to on-demand availability. Further, they have cost unpredictability and lack financial governance leading to undesired TCO. TCO optimization ensures that wasteful spending is identified and eventually eliminated.Â
Cloud data lake platforms help enterprises lower TCO in three ways:
1. Control and shape the infrastructure spend at will by applying policy overrides and by leveraging autonomous self-learning.
2. Provide built-in capabilities to optimize underlying infrastructure usage to lower the spend, monitor total costs at the application, user, account, cluster, cluster-instance level to drive transparency and accountability across teams.
3. Identify areas of cost optimization to drive maximum performance for the lowest TCO.
Users expect a data lake platform to understand what is currently happening from a cost point of view and build a financial profile of their cloud spending. They expect the platform to help put measures in place to control spending and optimize by interacting with underlying infrastructure on a continuous basis. They also expect cloud data lake platforms to allow isolation of workloads in different environments and clusters, automatically stop clusters when they are not needed, leverage a high percentage of spot/preemptible nodes using the cheapest hardware possible, and identify and optimize data organization based on usage patterns.
Another vital attribute of a data lake is openness. The data should be stored in an open format and accessible through open standards-based interfaces. The platform should adhere to an open philosophy aimed at preventing vendor lock-in, which ensures openness to data storage, data management, data processing, operations, data access, governance, and security while supporting a diverse range of analytics.
Ultimately, openness makes cloud data lakes cloud-agnostic and portable across any cloud-native environment. It enables administrators to leverage the benefits of a specific public cloud from the point of view of services offered, economics, security, governance, and agility. It allows them to use software from different vendors to realize different objectives and use cases. Ultimately all of these flexibilities lead to desired gains on TCO and beyond TCO on the innovation front.
Learn More: Five Skill Set Essentials for Data Management and SecurityOpens a new window
What are your thoughts about cloud data lakes? Did you enjoy reading this article? Comment below or let us know on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We would love to hear from you!