Data Mesh and Data Product Delivery – A Response to to Zhamak Deghani’s Manifest

By Hans Lux, the leading expert and innovative pioneer of enterprise data architecture, responsible for creating cutting-edge data solutions, designs and strategies.

Data Mesh for Effective Data Management

Currently, the most popular data topic, the Data Mesh (“DM”) applies product thinking to analytical data and puts the business domains who know “their” data best, in charge. A self-service data platform supports the teams. Governance is lightweight and mostly automated. A potential game changer, the DM could.

• greatly improve user experience and data quality
• alleviate bottlenecks due to monolithic architectures and centralization
• much better support for AI, highly distributed architectures and agile delivery

The DM is loosely presented as a vision, as a “sociotechnical paradigm”.  For success, competent and diligent practitioners need to interpret it, fill in the gaps, combine it with best practices and avoid potential pitfalls – some specific to the DM and some typical for large data projects in general. 

Enhancing Data Modeling for Success

For high chances of success, DM concepts must be understood, evaluated, in part fixed and augmented with best practices – in particular an effective approach to data modeling and data element management.

1. Understand the Data Mesh

A much-improved set of materials is needed just to understand what DM proposes, to evaluate the suitability of the DM for a project and to consider an implementation. Such materials

• fix the convoluted language (homonyms, terminology)
• prune the repetitive and extraneous content and restructure it
• fill in gaps based on underlying resources 

2. Fix flawed concepts and ideas

Flawed concepts and ideas create confusion, lead to endless yet fruitless philosophical discussions and make implementation without many exceptions difficult.

Data Product (“DP) Types

These types are oversimplified, incomplete yet overlapping.

• Don’t stick slavishly to them as proposed – it doesn’t work.
• Split up “source-aligned” and “aggregate” DP types
• Use a multidimensional approach, e.g., “this is a consumer-aligned DP, which also has a source and a derived component.”

That way, DP can be accurately defined and clear rules applied. Staff is optimally deployed, and responsibilities are clear.

Market and Product Paradigms

Take advantage of such paradigms only where they are suitable, e.g., where competing datasets are available on the market and avoid them or use a more differentiated approach where they are not:

• structured data from single in-house sources
• data that are only used as components together with other data. 

Joining Data (“Interoperability”)

The DM uses a haphazard method for joining data from different DP, relying on “polysemes” that allow guessing potential foreign keys from matching names. Provide guaranteed foreign key – primary key pairs to

• stop wasting analysts’ time for guessing and testing
• avoid errors or poor decisions due to faulty analyses

3. Take Advantage of Best Practices

Presented as a radical paradigm-shift, the DM ignores or rejects best practices, which risks cost-overruns and reduced benefit. Ensure that the DM delivers on its promise by augmenting with best practices, e.g., the well-honed analytic data journey from operational data through landing, staging, harmonization to data marts.

4. Optimize Modeling

An optimized data model is essential for effective data work, for understanding semantics in context and for data governance. Because intentional enterprise modeling is usually too expensive and too cumbersome, Domain-Driven Design (DDD) – the foundation of DM Principle 1 – pushes modeling to the domains.

Domain-driven modeling puts staff who lack modeling skill and experience in charge of modeling. Help them and support them:

• standardize modeling and provide reusable components
• simplify models and keep them tight by optimized abstraction
• incorporate into the self-service data platform the best data modeling tools that you can get your hands on 
• provide effective training and support

This will give you happy and productive domain teams, where competent staff can be rotated in-between. They will produce consistent and effective models that drive the DPs.

DM proposes point-to-point cross-model mappings (translations) between domain models, which are costly to maintain, create bottlenecks and are brittle and error-prone. Keep most modeling in the domains; use a small hub-and-spoke kernel for shared classes, concepts and the most essential data elements.

• Foreign-primary key pairs in the kernel guarantee reliable joining across DP.
• The enterprise-wide model emerges from the kernel, which is the unifying core and the domain models that line up to it and provide the detail.

Focus on data quality

Quality and trustworthy data are the very purpose of the DM: data must be easy to find, easy to understand – semantically and syntactically -, easy to use, easy to combine and worthy of trust – in terms of completeness and accuracy. Effective design, an optimized model and quality metadata are indispensable for assuring data quality. This requires competence as well as effort and money – hard to maintain under delivery pressure. 

The “savings” of cutting corners are lost manifold as analysts keep muddling through poor data.

Optimize design, model and metadata, and make sure everyone understands, throughout the project, why these are essential for success. Stay on message regarding their importance in every meeting and stay on top of politics.

Priorities

1. Drive this from the business NOT from technology/IT

Technology-driven DM projects with the primary focus on the delivery platform risk

• failure to deliver on requirements
• wasting money on a foundation that may far exceed needs
• late-state surprises with costly and sub-optimum emergency fixes
• refusal by the domains to take charge of their DP
• creating new legacy modules that are near impossible to improve because other modules are already dependent on them

Put the horse before the cart:

• focus on the problem you are trying to solve
• optimize the conceptual data architecture accordingly; the business needs to drive it
• iteratively build, test, evaluate and improve a few data products end-to-end
• stay on existing technology
• map out DPs for the first year or two including build-out, volume and usage metrics

Only now are you ready for technology decisions: Evaluate what’s there and new options. This approach avoids over and under engineering, useless or unused components, late-state surprises, emergency fixes and refits and unnecessary replacements if it turns out that the paid-for current kit is just fine.

2. Make sure the DM fits your needs

Build a full-blown DM only if there is a clear benefit. It may be cheaper and much more successful to implement only those DM facets that are beneficial, e.g., product thinking or federating to domains.

3. Don’t bite off too much

• Focus on one topology, one straightforward technical approach and design, plan and implement only once, in one way and in one place.
• Tightly control scope to essential core data, bring additional data in only after a successful initial delivery

This will avoid big, lengthy and costly projects, potentially poor delivery or project failure. Implementation becomes predictable, focussed and successful.

For a one-on-one consultation on data mesh, speak with one of our experts.

Sources

 1 Eric Evans’ 2003 Domain-Driven Design: Tackling Complexity in the Heart of Software (“DDD”)

2 Typically, no more than 1000 DE total

3 DM Principle 2: Data as a Product