Skip to content

No account required

Start a project

The data dictionary, and why inheriting a file without one costs a week.

The cheapest document in analytics to produce, and the one whose absence causes the most repeated work.

  • Read5 min
  • Sections3

What is a data dictionary and what should be in it?

A data dictionary describes each column in a dataset: what it means in business terms, its type, what proportion of rows are populated, its range or its set of valid values, and who owns it. Coverage is the most useful entry and the most often omitted, because a rate computed on a column that is sixty per cent populated has a denominator nobody intended.

01

What belongs in it

The column name as it appears in the file, and separately what a person calls it. Those differ, and the gap between net_amt and what finance means by net amount is where most misunderstandings start.

Its type, its role — whether it is something to total, something to group by, a date, or an identifier — and its coverage as a percentage of rows.

Then either its range, for a quantity, or its set of valid values, for a category. A category column's value set is often the single most informative thing in the whole document, because it tells a reader what distinctions the business actually makes.

02

Why coverage earns its place

Every other entry is documentation. Coverage is a finding: a rate computed over a column that is populated in sixty-two per cent of rows has a denominator of the rows that happened to be filled in, which is not the population anybody thinks they are measuring.

Putting the figure beside the column name is the cheapest way to make that visible, and it takes seconds to compute.

A column at one hundred per cent coverage and a column at ninety-four per cent support very different claims, and nothing in a spreadsheet distinguishes them.

03

Leave the values out

A dictionary that reproduces sample rows cannot be sent to the people who most need it: the developer writing the importer, the analyst inheriting the file, the reviewer who is not cleared to see personal data.

Describing the data rather than reproducing it is what makes the document portable. Ranges and category sets carry almost all of the usefulness and almost none of the disclosure.

It is also worth recording which columns are sensitive, because that decision otherwise gets made again, differently, by whoever next handles the file.

See it on a real project

The 22.4% rate combined three different events; the unrecovered no-show rate was 11.4%.

Why is our no-show rate 22%?

Related

Try it on your own file.

Every check described here runs automatically, and your first findings arrive in full before you pay for any of it.

No account needed to start. You only pay when you like what you see.