The demand for data accessibility with built-in governance is a common goal for organizations. Achieving this requires a collaborative approach involving Data Architecture, Data Governance, and Data Analytics teams working together within a shared data ecosystem. The Medallion Architecture, ER/Studio, and Databricks are key tools in this process. They play a crucial role in implementing Data Mesh programs, delivering Data Products within a Medallion architecture on the Databricks platform, and forming a comprehensive Data Fabric.
Effective data architecture and standardized data models are fundamental to overcoming challenges like data swamps and the uncontrolled “Wild West” of data. Balancing decentralization with rapid delivery while maintaining governance, standardization, and orchestration is essential.
Data Mesh represents a set of principles aligning with decentralization, autonomy, and accountability, forming the core of the Data as a Product (DaaP) concept.
Data Mesh is integral to Data as a Product because it operationalizes the principles of decentralization, autonomy, and accountability in data management. By embracing Data Mesh, organizations can incorporate data agility and ensure the quality and consistency of their data assets, ultimately driving value and innovation across the organization.
While Data Mesh promotes decentralization, it can lead to data silos if not managed properly. Consistency across domains is vital to ensure data interoperability.
Ensuring consistency across these domains reduces ambiguity and improves interoperability. If a “Sales Figures” data product from the “Sales” domain refers to products that are different from the “Products” described by the “Product Management” domain, then we will have problems.gement” domain, then we will have problems.
At its core, Medallion Architecture, proposed by Databricks and adopted as part of Microsoft Fabric, embodies a multi-tiered approach to data model design, fostering data quality within a data lakehouse:
Raw data enters the Medallion Architecture at this stage, where it undergoes minimal processing and remains in its original form.
Data is refined and transformed in the Silver Layer, ensuring cleanliness and structure before it progresses to the next stage.
The pinnacle of the architecture houses data optimized for analysis and business use, enriched with additional insights, and aggregated for deeper insights.
For a more detailed breakdown of how these layers are structured and connected, explore ER/Studio’s approach to Bronze-Silver-Gold architecture.
This is not a new approach to warehousing. Bill Inmon advocated Staging, Enterprise Data Warehouse, and Data Mart layers decades ago and has been used by many organizations.
Maintaining data in its raw form improves governance and allows historical records to be held. The silver layer allows standardization of reconciled, cleansed, and verified data that is well documented, governed and conforms to accepted standards.
The Silver layer ensures that data is reusable and well-understood. Standards on data quality can be ensured so that the data is reliable. The Gold layer ensures that data is fit for purpose and in a form that is consumable by the stakeholders.
Medallion Data Architecture and Lake House Architecture are both modern approaches to managing large-scale data, but they serve different purposes and excel in distinct areas.
Medallion Architecture is a layered data architecture primarily used to ensure data quality and governance within a data lake house environment. It organizes data into three main layers: Bronze, Silver, and Gold.
The Bronze Layer contains raw, unprocessed data, which is then cleaned and transformed in the Silver Layer. The Gold Layer holds data that is fully refined, enriched, and optimized for business analysis. This structured approach not only enhances data reliability and consistency but also ensures that data governance and compliance are maintained throughout the data lifecycle.
Lakehouse Architecture combines the benefits of data lakes and data warehouses into a single, unified platform. Key characteristics include:
Lakehouse Architecture excels in environments where agility and the ability to handle diverse data types are crucial, offering a more flexible approach to data management and analytics.
In summary, Medallion Architecture is a structured, tiered approach focused on data quality and governance, while Lakehouse Architecture is a more versatile framework that merges the strengths of data lakes and warehouses to support a broader range of analytical workloads.
Our objectives of delivering Data Products into the Gold Layer as quickly as possible with governance baked in needs some coordination. Data Models can help in several ways.
Having an up-to-date data catalog makes the system ready to deliver. Physical data models of source data assets are the first step, but the metadata within the asset is often insufficient. We need to understand the nature of the data in these assets. We will need to understand each field to determine if this holds valuable information for the business and then attach this to an accepted definition.
In practice, we will map to business terms but, better still, produce a logical data model that clearly shows the schema of the asset in business language with helpful metadata about the asset. Now, the asset is well understood and is ready if there are demands for that data.
The Enterprise Data Model is the framework that ties everything together by creating standardized entities for the organization’s vital information. Standardizing the naming of these entities and understanding their taxonomies is crucial for hardening the requirements for Data Products. For example, when defining a data product that describes individuals in a clinical trial, it’s essential to clarify whether it refers to the “Human Subject,” the “Healthcare Professional” involved, or the “Investigator” supporting the trial—each of which has distinct data sources.
After standardizing names and definitions, we must agree on how entities are identified. This process involves understanding multiple identification methods and choosing the primary one, which is critical for reconciling data from diverse sources. Additionally, understanding key attributes, definitions, and the nature of these entities helps reduce ambiguity and accelerates product delivery.
Once standardized, these entities are cataloged as foundational data products associated with different domains. Connecting these entities through relationships provides a map that simplifies understanding data connections, making it easier to build aggregate data products.
The two primary challenges are reconciling various data sources and coordinating Data Products across domains. If all data sources are cataloged and mapped to the Enterprise Data Model, sourcing data becomes straightforward. Standardized, reusable foundational data products ensure consistency across domains.
The Enterprise Data Model doesn’t have to be built all at once. It can start with the most important entities and grow over time, allowing for scalable and manageable implementation.
As we stated above, the Silver layer of the warehouse needs to be well structured with reusability and governance baked in. The Enterprise model can provide the core structure of this layer, whether it uses a third-normal form or a Data Vault schema system. This makes it easier to reconcile source data assets and understand what needs to be created to deliver against requirements.
The designs or our data products will drive the Gold layer. The logical models of these products can be used to create structures in the Gold layer, which may be realized as star schema tables or views. The logical models ensure that they are well documented and easily governed, and traceability back to the enterprise model makes sourcing data much more accessible.
ER/Studio is specifically designed to support and enhance enterprise data models. It offers an intuitive approach to building and mapping models to data sources and business terms, making the process straightforward and user-friendly.
The Team Server component within ER/Studio enables easy visualization and tracing of relationships within data models. This feature ensures that both business and technical users can effectively collaborate and leverage the models.
ER/Studio provides robust support for Databricks, including essential features like nested structures. This integration allows for the repeatable production of physical models of data warehouses directly from enterprise models, simplifying the process.
ER/Studio can automatically generate the necessary code to create Databricks assets, reducing manual workload. Additionally, it connects seamlessly with Data Governance tools, ensuring that models are always correctly classified and can be published back to the main data catalog.
The design features in ER/Studio, such as those integrated with Microsoft Purview, make it possible to turn data product designs into actionable insights, further enhancing the tool’s utility for businesses.
Want to see more? Please register for a demo so our engineers can walk you through your use cases.
Medallion Architecture is a layered design pattern—Bronze, Silver, and Gold—that helps structure data pipelines for quality, governance, and business-readiness. It’s key to delivering trusted data products in Databricks.
ER/Studio aligns with Data Mesh by decentralizing ownership while preserving governance. It enables domain teams to define and manage data products, while standardizing definitions across the enterprise.
The Enterprise Data Model (EDM) standardizes entities, definitions, and relationships across domains. It ensures that foundational data products are reusable and aligned—avoiding data silos and inconsistencies.
ER/Studio supports Databricks features like nested structures and can auto-generate code to build warehouse assets. This enables consistent physical model generation straight from your logical and enterprise models.
Yes. Logical models in ER/Studio are mapped to business terms and source systems. This traceability ensures that Gold layer assets align with enterprise standards and stakeholder expectations.
