The .parquet file.
The format to ask for when a CSV export is too big, too slow or too lossy. It is all three of those problems solved by storing columns instead of rows.
What it actually is
Parquet stores a table one column at a time rather than one row at a time. Every value in a column has the same type and usually similar values, which makes it compress extremely well and makes reading three columns out of two hundred cost almost nothing.
It also carries a schema. A date is a date, a decimal is a decimal with a stated precision, and a string that looks like a number stays a string. Everything a CSV loses at the boundary is preserved, which removes an entire class of analytical error before any analysis starts.
The trade is that it is not human readable and not editable in a spreadsheet. It is an interchange format between systems, which is exactly what an export to an analysis tool is.
At a glance
- Layout
- Columnar, with row groups
- Types
- Declared in the file, including decimals and timestamps
- Compression
- Snappy, gzip or zstd, per column
- Typical size
- Roughly a fifth of the equivalent CSV
- Human readable
- No
- Read here as
- Direct, with the file's own types kept
- Status
- Ready
- Fidelity
- Read directly
- Largest file
- 250 MB
Read columnwise, with the file's own types kept.
Upload one
The analysis runs before there is anything to pay for. You see what it found, and the evidence behind it, first.
No account needed to start. You only pay when you like what you see.
What goes wrong, and what is done about it
01
Timestamps without a timezone
Parquet distinguishes a timestamp that is a point in time from one that is a wall-clock reading, and many writers emit the second while the reader assumes the first. A day boundary then moves by hours, which is enough to shift a daily total into the wrong day.
The declared logical type is used rather than inferred, and where a file stores local timestamps with no zone that is recorded as an assumption you can see and override.
02
A folder of files that is really one table
Warehouses write partitioned datasets: a directory of many .parquet files, often with the partition key encoded in the directory name rather than in the data. Uploading one part gives you a slice with a missing column and no indication that either is true.
Zip the whole directory and upload the archive. Parts are recombined into one table and the partition keys in the paths are restored as columns.
Questions people ask
How do I get a Parquet file out of my database?
Most warehouses export it directly: BigQuery, Snowflake, Redshift and DuckDB all do. From a local CSV, DuckDB will convert one in a single statement. It is worth the step for anything above a few hundred thousand rows.
Is Parquet better than CSV for this?
Meaningfully, yes. Types survive, so leading zeros, long identifiers and dates arrive intact, and the file is small enough that a dataset which exceeds the upload limit as CSV usually fits comfortably.
Can I mix Parquet with spreadsheets in one project?
Yes. Sources are joined on content rather than on format, so a Parquet transaction export and an Excel cost sheet are matched the same way two spreadsheets would be.