There is no doubt that the cloud is the new data and analytics platform. As organizations move massive amounts of data to the cloud for analytics and AI modeling, the need to manage this data is influencing and revolutionizing how raw data is transferred from the source system to a cloud target – resulting in a shift from the traditional approach of moving data through extract, transform, load (ETL) to extract, load, transform (ELT). Clara Angotti, President of Next Pathway, shares why enterprises must make the shift.
The ETL model is representative of legacy systems, powered by expensive, inflexible licensing models that don’t adjust costs based on usage – and have limited compute horsepower.
The cloud costing model is extremely attractive to data consumers. Users have access to almost unlimited computing power and scalability at a meager cost due to capacity metering, lower licensing costs based on using a single integrated data platform, and the separation of the compute and operating system layer from the application and data layer.
On-prem, legacy systems require users to pay for peak capacity usage or transformations, even when the volume need has decreased. It’s a big reason why legacy systems are difficult to manage economically, and hence the massive migration to the cloud where you’re “renting†the space and only need to pay for the time that compute power is in use. Loading the raw data onto the cloud platform is much more cost-efficient than transforming the data and then loading it.Â
ELT vs. ETL: The Concept of “Transformâ€
There’s a pivotal nuance to understanding the concept of “transform†when evaluating ETL and ELT. In the ETL model, transform refers to a technical process to modify data (reformatting and mapping) before loading it to the cloud. However, the ELT model is a proper migration to cloud-native languages. It describes a process that permanently transfers data to the cloud, where it remains the target to be leveraged as needed. Source feeds come directly into the cloud with minimal changes and can then be transformed as part of the ELT process on the cloud.
See More: Cloud-Native Architecture: The Modern Way to Develop Software
Key Factors Prompting the Shift from ETL to ELT
Here are the five key reasons why the shift from ETL to ELT makes sense.
1. The congestion problem
Considerable increases in the use of both unstructured data, such as images, audio and video, and semi-structured data, such as email and texting, slow down the ETL process. The ETL process is riddled with congestion because of the time and compute power necessary to transform the data before loading.
It is far more efficient to transform unstructured data using cloud computing. With ETL, you often must wait until all the transformations are complete for an end-use case before loading. Also, you need to know all the end-use cases for that data before the transformation. Housing unstructured data in the cloud enables a more flexible and agile approach to data management, allowing you to manipulate it as applications or business use cases arise.
2. Modernization of data platforms
The ELT model uses the cloud’s power, cost, and capacity to incentivize customers to modernize their enterprise data warehouse (EDW) and enterprise data lake (EDL) platforms. This is particularly true with users managing vast amounts of data on legacy platforms, where the available bandwidth limits you. There’s a compelling advantage to moving applications to the cloud and then computing through millions of records of data in real-time.Â
With limitless compute power, the cloud provides the processing ability to transform the code used to manage legacy ETLs and the ongoing ingestion and mapping of the data itself. The ELT approach enables consumers to move their data to the cloud with very little modification, heralding the use of cloud tools to transform that data then.Â
3. Increased speed of integration and reduced complexityÂ
Cloud data platforms provide excellent low/no-code development, self-service and out-of-the-box tools that enable easy integration of new and different data sources for business-use applications. Complexity is reduced when using ELT, as the cloud provides a single integrated platform for data ingestion, data integration and data quality.Â
4. Boost end-use cases
Most organizations have – or wish to have – modern business intelligence and analytics tools to derive increased value from their data. These tools are much less effective when deployed against legacy databases, partly because of the higher cost of capacity required to operate them, but primarily because modern tools developed with modern languages work better with cloud-based data and code. Data analytics and BI tools are built to be optimized using cloud-native data in the current cloud environment.
5. Data source transparency
When data is transformed and then loaded to the cloud data platform, users – including analysts and developers – don’t have transparency into the source or integrity of the data. Without visibility into the logic applied during the transformation process, they can’t offer DevOps intelligence if there is an issue with the code or if the transformation pipeline doesn’t work.  Â
Transforming the Future
As investment in cloud-native architecture increases exponentially, the shift from ETL to ELT will follow suit. The tools, processes, capabilities, and ability to move to the cloud more efficiently and more easily will also significantly improve. Conversely, the accompanying risk for businesses that continue to use legacy systems will also increase substantially. In the next couple of years, we expect the shift to ELT to be nearly complete, as organizations understand that to take full advantage of the cloud-native architecture, an ELT approach is essential.
Do you think the shift to ELT is mandatory? How can enterprises make the change easier for their processes and people? Share with us on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We’d love to hear your ideas!