The Data Lake has been a great place to take large quantities of semi-structured data and allow data scientists to run ad-hoc analyses and provide insights into the data. However, a large volume of ad-hoc work can result in a data swamp that quickly becomes unmanageable and ungovernable with potentially sensitive data exposed and undocumented data decaying, reducing its quality.
Many organizations are now applying a little more structure to their lakes by creating Data Lakehouses. These offer:
Data Lakehouses address the limitations of traditional data lakes by enhancing structure, documentation, and data quality, allowing organizations to govern their data more effectively and meet business needs with greater efficiency.
The Data Lakehouse is really just a Data Warehouse that includes data from the lake. The same principles as data warehouses should apply. The above process means that we have layers in the Lakehouse.
There are multiple approaches to naming and structuring these layers. The following list describes some of the more popular approaches.
This approach has been around for many years, developed by Bill Inmon. It centers around a central data store well-organized using highly normalized third-normal-form (3NF) models. The central data store is designed to align with the structure of the business.
There is then a data mart or presentation layer that contains structures aligned with the final output, using the Kimball approach with denormalized star schemas that make reporting much easier.
The central data store loads data from sources, potentially using external tables pulling from lake-hosted files, and may also pull data from structured source systems using ETL tools, perhaps maintaining historical versions of that data.
This approach was originally promoted by Databricks and consists of Gold, Silver, and Bronze layers. For all intents and purposes it is identical to the Inmon approach above.
The Data Vault approach, developed by Dan Lindstedt, is designed to handle changes in data structures efficiently.
The Data Vault centers around a core structure consisting of Hub and Link tables, which align with the business’s understanding of its data.
The reconciliation of records is done early in the process, using an accepted Business Key for each record held by the Hubs.
Final Presentation Layer
The final presentation layer is typically a set of star schemas.
So that’s the technical process and structure, but this requires some planning. All three of these approaches will only succeed if the central structures are well-defined and aligned with the business. ER/Studio is a tool that supports Enterprise Data Architecture and centers around the use of an Enterprise Data Model.
This model is really an understanding of the business and should be non-technical. It has two layers: the Enterprise Conceptual Data Model and the Enterprise Logical Data Model.
This is the mile-wide inch-deep view of the organization and can be built very quickly. It should be a list of the fifty to two hundred concepts that are important to the organization. These may be key actors in the organization and the key transaction types performed within it. There will be a simple description for each and some lines that represent the relationships between the concepts.
This view allows the business to prioritize these concepts in terms of importance, sensitivity, etc.
This provides a more detailed view of the concepts and gives us standardized definitions of data entities. Concepts may be defined using taxonomies to show the different types of entities. Entities will have the accepted ways of uniquely identifying instances of them. Standard descriptions and attributes of them will be provided. And again, the relationships between them are described.
We hear many organizations tell us, “our business is too complex to do this; it would take forever.” This doesn’t have to be an exhaustive model and can take the important concepts from the conceptual model and go deep on those. Perhaps just have a standard definition of “Customer” and its related information.
The Enterprise Model may also store governance information such as policies and security classifications.
This Enterprise Model is used for many purposes including:
ER/Studio allows architects to define the Enterprise Model and publish it through its web-based front end to the entire organization. It then allows this model to design data assets ensuring they include standardized structures and are well documented. It then allows what-if analysis to be performed.
The three approaches to building an effective data lakehouse are all common in that they rely on the central body of the lakehouse to be aligned with the business. This ensures that requirements for new outputs from the warehouse can be met and changes managed quickly. The Enterprise Data Model provides this business knowledge in a form that can be quickly used in the lakehouse.
In ER/Studio, entities from the Enterprise Model can be added to the design of the warehouse, maintaining links back to it for traceability. For Data Vault, tools are available to transform 3NF entities into Hubs and Links so that the structure is aligned with the accepted enterprise model.
In the past, approaches to the warehouse have been “build it and they will come.” Just loading data into the warehouse and hopefully new requirements will be able to use that data. More recently, we have accepted that we should start from the business requirements. Thus, the process should be:
This approach has become popular of late and allows a decentralized approach to designing and governing the outputs of the analytics world that allows domains of genuine subject matter experts to define Data Products.
The domain teams review requests and maintain well-documented product catalogs and can set classifications and policies on them.
Data Mesh really focuses on the consumption model of the data products. The delivery can also be decentralized but will have more benefits if it is delivered through a well-managed platform, in this case, the Data Lakehouse.
Now Data Products can be designed quickly with consistency, and the data for them sourced, staged, and processed quickly as part of an effective data supply chain. Throughout all data is well understood with governance policies considered.
ER/Studio has been designed to support this process centered around the Enterprise Data Model to ensure consistent and high-quality lakehouses can be built that support the delivery of data requirements quickly with governance baked in.
To learn more about ER/Studio’s solutions, connect with our experts for a customized demo and see how we can support your goals.
