Why Automated ETL Isn't ETL at All

By Special Guest
Taylor Barstow, CEO & Co-Founder, Bedrock Data
April 03, 2018

As the number of SaaS applications ascends ever upward, so too has the desire to integrate the cloud data nested within them. SaaS data is rich with insights about customers. The trick is how to unify the data, so it can be fed into reports, analytics, and business intelligence (BI) tools where you can use it to draw valuable insights.

Since the 1970s, businesses have used ETL (extract, transform and load) technology as a first step towards moving data between systems. Most IT professionals consider ETL complex and time-consuming — more turtle than hare. Half a century later, ETL has evolved to the extent that it’s finally stood upright, morphed into a far more advanced species.

This “new ETL” automates the process of data ingestion, preparation, and storage. At the click of a button, anyone can have all their SaaS application data unified, and a cloud data warehouse built to feed their analytics, reports, & dashboards.

Undergirding this evolution is a true revolution in real-time analytics — historically difficult to achieve. With automated ETL, businesses can finally combine data sources and trust their accuracy. Developers are free from having to write code that extracts data from applications via APIs. And IT teams that have had to manually standardize data formats and learn how multiple data models relate to one another, no longer have to.

For decades, these roadblocks have either stalled analytics projects or postponed them indefinitely. Now businesses can analyze customer records instantly, since system relationships may now be automatically mapped, matched, modeled, and merged into a single database. That 80% of analytics projects is spent on data ingestion and prep could very soon become a stat of the past.

What does all this mean for the SaaS software & IoT spaces?
The rise in devices has also produced more data sources. Customers engage with businesses through multiple channels, often simultaneously. One day a customer might visit your website on desktop, the next on their iPad. After they file a support software, they might talk about their experience with customer service on Twitter.

With an automated ETL approach, making use of all this data — regardless of origin — is no longer a multi-day, -week, or -month project. Every activity, event, deal, and opportunity — also called “objects” — can be automatically combined within a simple UI. It’s also possible to customize relationships, or mappings, between said objects. Customer data, once seen as an analytical non sequitur, now coheres, bestowing businesses with a more holistic understanding of user behaviors and preferences.

Modern ETL also builds a data warehouse that connects easily to reports, analytics, and BI tools. For developers, it’s a relief just to have a unified schema of SaaS data. For analysts, transforming many data sets into one is just shy of a miracle, since data is in near real-time. Executives getting ready for a board meeting can report their top KPI and refresh dashboards, assured that data is up-to-date, not weeks old.

According to a survey of 502 IT professionals, traditional ETL has long undermined organizations’ ability to achieve real-time analytics. Once moved via ETL, data is at least five days old by the time it reached an analytics database. These delays also grow as data sets do. The more data there is to process, the more time IT must exert on each step, transformation especially. These are just some of the obvious reasons why so many IT professionals are on the hunt for better alternatives.

One of the chief advantages of automated ETL is speed. As the number of data sources added increases, the time it takes to unify and warehouse is relatively unaffected when compared to non-automated antecedents.

Although predictions about new technology tend to err quickly, at play are two parallel trends which reflect the same intrinsic demand — the need to do more faster, and with fewer resources. At the office, SaaS has quenched our thirst for higher productivity. At home, the IoT era has engineered a spate of gadgets which renders home and daily life more convenient.

In light of these truths, we should expect traditional ETL to fade, leaving room for a superior solution, one whose automation so radically outpaces the current one that it isn’t ETL at all.

About the author: Taylor Barstow is CEO & Co-Founder of Bedrock Data. He is a technology innovator who has a proven track record as both a developer and an engineering executive.

Edited by Ken Briodagh

Related Articles

AT&T and Los Angeles Explore Partnership

By: Ken Briodagh    9/20/2018

AT&T and the City of Los Angeles are looking to team up to make LA one of the smartest cities in America and to Drive Traffic, Public Safety and Disas…

Read More

SAS IoT to power China's Wuxi High-Tech Zone

By: Ken Briodagh    9/19/2018

Partnership builds on SAS' position as the IoT analytics leader in smart cities arenas.

Read More

AT&T, G+D and Altair Team Up to Spur IoT Deployment

By: Ken Briodagh    9/19/2018

Next-Generation Integrated SIM Will Meet the Needs of an Expanding Global Internet of Things Market

Read More

Top 5 Reasons Why APIs Lead to Blockages in B2B Operations

By: Special Guest    9/18/2018

These days, enterprises are increasingly adopting API led solutions for Business to Business (B2B) connectivity.

Read More

Technology Companies Join to Reduce Water Consumption for Commercial Farms

By: Cynthia S. Artin    9/18/2018

Three companies have announced an Industrial IoT (IIoT) "stack" combining sensors, signals, semiconductors, algorithms, expertise in social moisture m…

Read More