More and more of our clients are investing in Data Warehousing. Also, more are moving to cloud-based warehouses to enjoy benefits such as agility, cost reductions, and availability. We now have some great products, like WhereScape, to help automate the design and execution of data warehouses, further reducing costs and improving efficiency.
But we need to be mindful of what we put into our shiny new warehouses. We have all seen ungoverned warehouses become bloated with data that is not understood, putting us at risk. Misunderstood or poor-quality data can be very dangerous.
The warehouse can advertise data that is labeled poorly, lacks context, uses incorrectly calculated fields, or is simply of poor quality.
We also need to be mindful of security and regulatory issues. Are we exposing confidential or sensitive data? If we are working in the cloud, are we storing personal data in locations that may contravene our rules? Planning is the key.
What data assets exist in your organization? For each asset do we have useful knowledge such as the context of the data, the owner of the asset, a summary of content and any rules that may apply? ER/Studio can help you build a list of data sources and publish that knowledge to the organization.
For each asset, we need to know what data is within it. The metadata of the asset may not help, as it could consist of physical, technical names. Most database products have restrictions on naming, so compressed physical names are often used. For example:
Other systems, like SAP, may use codes for tables and columns that have zero meaningful context. There are tools available that analyze the data itself and attempt to deduce the meaning, some utilizing AI technologies.
However, there is little substitute for proper documentation. Data Architects play a crucial role here, as they design and document data assets. ER/Studio is central to this process, enabling each data asset to have both:
It is essential to define and classify your information clearly to avoid ambiguity and ensure proper management. By standardizing terminology, classifications, and usage rules, organizations can significantly reduce risks and enhance data governance.
We often use terminology assuming a common understanding, but this can lead to ambiguity. For example, if asked for “North American sales figures,” it’s essential to define what this means and how it should be calculated. A published glossary of accepted terms can help reduce misunderstandings and minimize risk.
Key Performance Indicators (KPIs) should be clearly defined along with their calculation methods. It’s also necessary to classify information based on sensitivity, confidentiality, and related attributes, such as personally identifiable information (PII) for data related to individuals.
Specific rules for data usage need to be outlined. ER/Studio’s built-in Business Glossary can help by allowing Business Terms to be defined and enabling Data Architects to classify data assets accordingly. This feature lets users see which data assets contain specific information types.
ER/Studio supports this planning process by producing and publishing a Data Catalog. This resource can answer critical questions about the data stored in the warehouse, including:
Then tools like WhereScape can automate the inclusion of that data into the warehouse.
