By Carol Newcomb, Senior Data Management Consultant at SAS
No one likes unpleasant surprises. When it comes time to look objectively at what has happened in terms of customer activity, business productivity or progress toward goals, everyone wants to be able to rely on the reports they get. And no one wants to be embarrassed to provide inaccurate reports, whatever the underlying reason. In situations like this, the importance of data quality is undisputed.
But how much control does a person have over the quality of the data used in reports? Who is responsible for this data? Who understands where it came from, or how, and why it may have changed? Who gets to write the business rules or quality standards? There are usually many different business components for each dataset, each of which has to tell their own story. Are you drawing straw to decide who will tailor the data to meet their specific business needs? And who gets to say about the data it is wrong?
Of course, there are software tools that can help with data correction and error analysis. But tools alone do not solve the problem. Business users must first have a plan that helps them identify quality problems, track the underlying sources, develop mechanisms to solve problems, and then create a process for monitoring and marking any new issues that arise.
A sustainable plan
Managing data quality is not necessarily simple. When considering the data life cycle – from data creation / collection to archiving – there are many steps along the way, including:
- Rules for collecting / creating data.
- Data quality standards, thresholds, and rejection criteria.
- Data standardization and summary rules.
- Rules for data integration with other data sources.
- Hierarchy management (relationship management).
- Ongoing triggers to detect outliers during updates.
- Data correction rules.
An effective, sustainable data quality plan will resonate with business users and should include the following five elements.
Increase the visibility and importance of data quality
Poor data quality has a significant business cost – in time, effort and accuracy. Quantify the cost of bad data and build a credible business case that demonstrates the negative impact of current data quality issues. Illustrate how data quality affects different parts of the business. This becomes an important part of your rationale for why a plan that encompasses the importance of data quality is a business critical one.
Formalize decision making through a data management program
Data correction should not be done in a vacuum, nor should each analyst have his or her own rules of error correction. Avoid allowing too many people to make one-time data quality decisions that do not meet a common business purpose. Give authority to develop business rules and standards with a decision-making data management team that has perspective across business areas. These rules must be monitored and approved to ensure that they are valid and reusable. Only then should the data quality process be used.
Document the data quality issues, business rules, standards, and data correction policies
A boss once told me, “If it’s not documented, it didn’t happen.” To combat a firefighting culture with data fixes – and to prevent ongoing inefficiency caused by individuals inconsistently correcting data – you must document each topic, publish it, and communicate the remedy. This encourages users to avoid costly and time-consuming experiences in correcting data in ways that cannot be reused or shared across the organization.
Clarify accountability for data quality
Develop a process where business users can report data quality issues and then work with data providers to investigate the source of error and develop a solution. Release business analysts from the burden of researching data quality issues – freeing them to do their job as analysts. Identify data quality specialists, both data providers and data quality professionals, who are responsible for solving data quality issues. These issues can range from analysis of root causes, metadata management and policy definition to documentation and monitoring. This approach – which recognizes the enormous importance of data quality – is a huge saving for most organizations.
Applaud your successes
If you have prepared your plan carefully, you will collect basic statistics and then measure data quality improvements over time. Show business value to users through your own case studies. The importance of data quality is superb – better data is translated directly into better business value. Make sure you understand how they measure themselves in different parts of the business and tie the improvements in data quality to improvements in their overall success. Then communicate how to share the value of better data across business areas.
When you recognize the multifaceted nature of questions asked by people with different business interests and perspectives, you will position yourself to design a sustainable approach to data quality. As you elaborate on the data quality plan, you need to include technical experts to work with your business analysts. Together, they can explain the full spectrum of data definitions and properties of the data while constructing ways to correct and maintain them. The investment pays off in the short term and in the years to come.