What is a data dictionary and what should be in it?
A data dictionary describes each column in a dataset: what it means in business terms, its type, what proportion of rows are populated, its range or its set of valid values, and who owns it. Coverage is the most useful entry and the most often omitted, because a rate computed on a column that is sixty per cent populated has a denominator nobody intended.
Section 01
What belongs in it
The column name as it appears in the file, and separately what a person calls it. Those differ, and the gap between net_amt and what finance means by net amount is where most misunderstandings start.
Its type, its role, whether it is something to total, something to group by, a date, or an identifier, and its coverage as a percentage of rows.
Then either its range, for a quantity, or its set of valid values, for a category. A category column's value set is often the most informative thing in the whole document, because it tells a reader what distinctions the business actually makes.
Section 02
Why coverage earns its place
Every other entry is documentation. Coverage is a finding: a rate computed over a column that is populated in sixty-two per cent of rows has a denominator of the rows that happened to be filled in, which is not the population anybody thinks they are measuring.
Putting the figure beside the column name is the cheapest way to make that visible, and it takes seconds to compute.
A column at one hundred per cent coverage and a column at ninety-four per cent support very different claims, and nothing in a spreadsheet distinguishes them.
Section 03
Leave the values out
A dictionary that reproduces sample rows cannot be sent to the people who most need it: the developer writing the importer, the analyst inheriting the file, the reviewer who is not cleared to see personal data.
Describing the data rather than reproducing it is what makes the document portable. Ranges and category sets carry almost all of the usefulness and almost none of the disclosure.
It is also worth recording which columns are sensitive, because that decision otherwise gets made again, differently, by whoever next handles the file.
See it on a real project
The 22.4% rate combined three different events; the unrecovered no-show rate was 11.4%.
Keep reading