Modern Analytics: Data Lakes, Data Warehouses, and Clouds

Issue link:

Contents of this Issue


Page 2 of 3

D B TA . C O M / B I G D ATA Q U A RT E R LY 13 Research and the Independent Oracle Users Group (IOUG), 40% expressed concerns about the benefits being too small to justify the investment required to migrate. There are many compelling business benefits for making the move to the cloud that go beyond simple upfront cost savings. There will no longer be constraints on the ability to pursue data- driven business ideas. The organization may be able to move more aggressively toward AI and more powerful vehicles of data analyt- ics with the additional capacity available through cloud-borne data warehouses and data lakes. There will also be an enhanced ability to store data from various enterprise applications—ERP systems, financial systems, IoT-based systems—within a common environ- ment. Cloud proponents will need to illustrate the vastly expended business pursuits that cloud environments support. Determine the best architecture. There are several types of approaches to data warehouses and data lakes in the cloud. In the DBTA survey on cloud adoption, 59% of data managers said they are deploying applications and data to public cloud services, 55% are maintaining their data environments within private clouds, and 36% report using hybrid cloud arrangements. A hybrid data warehouse or lake enables an organization to maintain data or applications on-premise while either gradually moving to the cloud or delineating functions that remain on-prem- ise versus those that are cloud-based. Of course, this requires retaining skills for both cloud and on-premise systems. At the same time, it ensures greater resiliency, as well as greater flexibility in data placement. A multi-cloud data lake or data warehouse leverages more than one platform while also requiring skills to integrate and manage two platforms that likely have differing protocols. Assess your skills and staffing requirements. Within on-prem- ise data warehouses and lakes, there is a need for database adminis- trators as well as software engineers to build and scale such environ- ments. While cloud reduces the need for such skills, it requires new skill sets, such as those of cloud engineers or cloud architects, as well as professionals with insights in cloud-based security and resource management. New roles need to be defined, and appropriate train- ing provided. In the Unisphere-IOUG survey, one in five, 19%, cited skills availability as a challenge to moving to cloud environments. Establish ongoing data movement. Getting a cloud data ware- house or lake up and running is only the beginning of the process. Data needs to be continuously synced and moved between sources and data environments. This ties into the overall performance of the data environment as it becomes increasingly cloud-borne. In the Unisphere-IOUG survey, 24% said concerns over maintaining the required level of performance in the public cloud was a challenge. Rethink security. Moving to the cloud does not mean out- sourcing security to a third-party provider. Security needs to remain a top priority for enterprises, regardless of how much data and how many applications are managed through cloud services. Due diligence is important, and enterprises and their data man- agers need to hold vendors closely accountable for the security of their corporate data assets. In the Unisphere-IOUG survey, 36% expressed concerns about data security as they moved data envi- ronments to the cloud. Reorient your cost structure. While on-premise systems typi- cally involve upfront capital expenditures, a cloud-based approach spreads costs across subscription plans. However, subscription costs can quickly add up, requiring a different methodology for calculat- ing expenditures. Upfront investments will be minimal, but costs associated with increasing usage—as well as skills and resources still required to plan and build out capabilities—may escalate, resulting in sticker shock. In the Unisphere Research survey of IOUG mem- bers, 32% cited worries about hidden or unforeseen costs of cloud subscriptions as a challenge to cloud adoption, making this one of the leading concerns. Similarly, 28% cited higher licensing costs to run in a public cloud than in current on-premise solutions. Consider storage requirements. The storage requirements for a cloud-based data warehouse or data lake may be massive. The increase in data and analytical capabilities can mean exponential increases in storage. It has always been challenging to scale onsite storage to meet the requirements of a booming data environ- ment, which were often met by deploying strategies such as data compression, along with additional hardware. Cloud reduces the need for such strategy but also introduces new challenges, such as latency, integration, and the potential costs due to charges, whether they be per megabyte of data transfer or monthly fees. Overall, cloud computing is proving to be a boon to adoption and expansion of data warehouses and data lakes. The key is preparing the organization for the endless possibilities this architecture brings. —Joe McKendrick Best Practices Series

Articles in this issue

Links on this page

view archives of Reports - Modern Analytics: Data Lakes, Data Warehouses, and Clouds