How do I document my analysis so somebody else can reproduce it?
Record three things alongside every result: the exact filters applied before the calculation, the definition of each derived measure in one sentence, and what the finding does not establish. Most reproduction failures are not arithmetic errors: they are an unstated exclusion or a differently defined variable, and neither is visible in the output.
Section 01
Filters are claims
Every exclusion applied before a calculation is an assertion about which population the result describes. Dropping incomplete responses, removing outliers, restricting to a date range: each is defensible and each changes the answer.
A filter nobody can see is an invisible claim, so the list of them belongs next to the figure rather than in a method appendix nobody reaches.
Section 02
Define derived measures once, in words
A derived variable defined in code and never in a sentence is a variable two people will implement differently. One sentence per measure prevents more reproduction failures than any amount of version control.
Where you changed a definition partway through, record both and which results used which. That is the situation most likely to produce a table that cannot be regenerated.
Section 03
Report the results that did not hold
A comparison that came back null is a result, and omitting it makes every reported result harder to interpret, because the reader cannot tell how many comparisons were made.
State how many things were tested. One striking finding out of twenty is roughly what noise produces, and the honest version of that sentence is short and costs nothing.
Write the definition down before looking at the result. The temptation to adjust it afterwards is strong and completely invisible in the final write-up.
Section 04
Keep the file that carries its own codebook
SPSS and Stata files hold variable labels, value labels and declared missing codes alongside the data. Exporting to CSV first throws all three away, and re-typing a codebook is both slow and where errors enter.
It also removes a specific failure: a survey coding a refusal as 99 produces a mean satisfaction of 14 on a five-point scale once the missing codes have been forgotten.
See it on a real project
The apparent 0.42-point gap fell to 0.07 once partial responses were included.
Keep reading