Monitoring and Metering the health of Enterprise Data

Executive Summary
Data governance is now seen as a priority in nearly all enterprises. Data-related in production data environments have grown to a point where they must be addressed. Yet data governance is still relatively immature and not defined precisely. Efforts at data governance have often focused on establishing councils or committees that set up procedures to access or use the data. Executive management appears to find such approaches bureaucratic and unable to address the underlying data problems.

A significant area for improvement is data quality, and executive management sponsor data governance in the hope that it will address data quality. However, data quality is itself poorly defined and usually a collection of issues that unlikely has one common fix. The whole approach of first trying to identify types of data quality issues and then figuring out ways to remediate them is questioned here. It is proposed that monitoring (detecting exceptional events) and metering (gathering metrics on the health of the data) is a logical precursor to fixing any problems. Monitoring and metering is a fundamental engineering principle that is used in process control. Yet it is hardly ever used in production data environments. This is a significant missing element in data management.