Skip to content

No account required

Start a project

5 min · 3 sections

The data dictionary, and why inheriting a file without one costs a week.

The cheapest document in analytics to produce, and the one whose absence causes the most repeated work.

By Data Analysis App team

What this covers

Jump to the part that matches what you are looking at.

What is a data dictionary and what should be in it?

A data dictionary describes each column in a dataset: what it means in business terms, its type, what proportion of rows are populated, its range or its set of valid values, and who owns it. Coverage is the most useful entry and the most often omitted, because a rate computed on a column that is sixty per cent populated has a denominator nobody intended.

Section 01

What belongs in it

The column name as it appears in the file, and separately what a person calls it. Those differ, and the gap between net_amt and what finance means by net amount is where most misunderstandings start.

Its type, its role, whether it is something to total, something to group by, a date, or an identifier, and its coverage as a percentage of rows.

Then either its range, for a quantity, or its set of valid values, for a category. A category column's value set is often the most informative thing in the whole document, because it tells a reader what distinctions the business actually makes.

Section 02

Why coverage earns its place

Every other entry is documentation. Coverage is a finding: a rate computed over a column that is populated in sixty-two per cent of rows has a denominator of the rows that happened to be filled in, which is not the population anybody thinks they are measuring.

Putting the figure beside the column name is the cheapest way to make that visible, and it takes seconds to compute.

A column at one hundred per cent coverage and a column at ninety-four per cent support very different claims, and nothing in a spreadsheet distinguishes them.

Section 03

Leave the values out

A dictionary that reproduces sample rows cannot be sent to the people who most need it: the developer writing the importer, the analyst inheriting the file, the reviewer who is not cleared to see personal data.

Describing the data rather than reproducing it is what makes the document portable. Ranges and category sets carry almost all of the usefulness and almost none of the disclosure.

It is also worth recording which columns are sensitive, because that decision otherwise gets made again, differently, by whoever next handles the file.

See it on a real project

The 22.4% rate combined three different events; the unrecovered no-show rate was 11.4%.

Why is our no-show rate 22%?

Keep reading

Every check in this guide runs on every upload.

Drop the file in and the problems described here are tested before anything is reported: with the rows behind each repair kept, so you can see exactly what changed.

  • Subtotal rows detected and excluded
  • Duplicates matched on identity, not on whole rows
  • Text-typed numbers found, coerced and counted

No account needed to start. You only pay when you like what you see.