Data Quality Assessment
Data Quality Assessment, by Arkady Maydanchik
Imagine a group of prehistoric hunters armed with stone-tipped spears. Their primitive weapons made hunting large animals, such as mammoths, dangerous work. Over time, however, a new breed of hunters developed. They would stretch the skin of a previously killed mammoth on the wall and throw their spears, while observing which spear, thrown from which angle and distance, penetrated the skin the best. The data gathered helped them make better spears and develop better hunting strategies.
Topics
Part I – Data Quality Overview
Chapter 1 Causes of data quality problems
Initial Data Conversion
System Consolidations
Manual Data Entry
Batch Feeds
Real-Time Interfaces
Data Processing
Data Cleansing
Data Purging
Changes Not Captured
System Upgrades
New Data Uses
Loss of Expertise
Process Automation
Summary
Chapter 2 Data Quality Program Overview
Data Quality Assessment
Data Cleansing
Monitoring Data Integration Interfaces
Ensuring Data Quality in Data Conversion and Consolidation
Building Data Quality Meta Data Warehouse
Summary
Chapter 3 Data Quality Assessment Overview
Project Team
Project Plan Overview
Planning Phase
Preparation Phase
Implementation Phase
Fine-Tuning Phase
Ongoing Data Quality Monitoring
Summary
PART II – Data Quality Rules
Chapter 4 Attribute Domain Constraints
Introduction to Attribute Domain Constraints
Attribute Profiling
Optionality Constraints
Attribute Format Constraints
Valid Value Constraints
Precision Constraints
Summary
Chapter 5 Relational Integrity Rules
Relational Data Model Basics
Identity Rules
Reference Rules
Cardinal Rules
Inheritance Rules
Summary
Chapter 6 Rules for Historical Data
Introduction to Historical Data
Basic Data Quality Rules for Historical Data
Advanced Data Quality Rules for Historical Data
Data Quality Rules for Event Histories
Summary
Chapter 7 Rules for State-Dependent Objects
Introduction to State-Dependent Objects
Identifying State-Dependent Entities
Profiling State-Transition Models
Rules Derived from State-Transition Diagrams
Timeline Constraints
Advanced Rules
Summary
Chapter 8 Attribute Dependency Rules
Introduction to Attribute Dependency Rules
Identifying Dependencies through Analysis
Identifying Dependencies through Data Profiling
Identifying Dependencies Across Data Sources
Summary
PART III – Data Quality Assessment
Chapter 9 Implementing Data Quality Rules
Project Scope and Rule Design
Selecting Optimal Rule Design
Rule Cataloguing
Rule Coding
Summary
Chapter 10 Fine-Tuning Data Quality Rules
Rule Imperfections
Rule Fine-Tuning Process
Identifying Rule Imperfections
Analyzing Imperfection Patterns
Eliminating False Positives
Handling False Negatives
Handling Uncertainty in Error Location
Summary
Chapter 11 Cataloguing Errors
Error Catalogue Basics
Recording Missing Records
Errors Affecting Multiple Records
Error Groups
Subject-Level Error Tracking
Error Messages
Summary
Chapter 12 Measuring Data Quality Scores
Introduction to Aggregate Scores
Score Tabulation Process Overview
Building Score Catalogue
Tabulating Record-Level Scores
Adjusting Scores for Rule Imperfections
Tabulating Subject-Level Scores
Summary
Chapter 13 Data Quality Meta Data Warehouse
Data Quality Assessment Meta Data
Data Quality Scorecard
Other DQMDW Functions and Reports
Summary
Chapter 14 Recurrent Data Quality Assessment
Basics of Recurrent Data Quality Assessment
Data Quality Changes on Atomic Level
Adding Time Dimension to DQMDW
Executing Assessment Runs Against Production Data
Summary
Appendix & Index
Appendix – Charts and tables
Quality data is the key to any advancement, whether it’s from the Stone Age to the Bronze Age. Or from the Information Age to whatever Age comes next. The success of corporations and government institutions largely depends on the efficiency with which they can collect, organize, and utilize data about products, customers, competitors, and employees. Fortunately, improving your data quality doesn’t have to be such a mammoth task.
DATA QUALITY ASSESSMENT is a must read for anyone who needs to understand, correct, or prevent data quality issues in their organization. Skipping theory and focusing purely on what is practical and what works, this text contains a proven approach to identifying, warehousing, and analyzing data errors – the first step in any data quality program. Master techniques in:
- Data profiling and gathering metadata
- Identifying, designing, and implementing data quality rules
- Organizing rule and error catalogues
- Ensuring accuracy and completeness of the data quality assessment
- Constructing the dimensional data quality scorecard
- Executing a recurrent data quality assessment
This is one of those books that marks a milestone in the evolution of a discipline. Arkady’s insights and techniques fuel the
transition of data quality management from art to science — from crafting to engineering. From deep experience, with thoughtful structure, and with engaging style Arkady brings the discipline of data quality to practitioners.
David Wells, Director of Education, Data Warehousing Institute
About Arkady
Arkady Maydanchik is a recognized practitioner, author, and educator in the field of data quality and information integration. He is a frequent speaker at conferences and seminars, and teaches data quality courses through the Data Warehousing Institute and through his company, Data Quality Group LLC.
Bestsellers
Faculty may request complimentary digital desk copies
Please complete all fields.