Parquet files for ticks: why CSV is the wrong format

[Amanhasnoname]

CSV is universal and slow. For tick data, it wastes storage, parses slowly, and loses type information. Reading a few gigabytes of CSV can take minutes, and every analysis repeats the cost.

Parquet stores data in a columnar, compressed binary format. File sizes often shrink by five to ten times compared with CSV, and reading specific columns or date ranges is fast. Types are preserved, including timestamps with precision.

Partitioning by date or symbol allows loading only what is needed. A query for one week of one pair touches a small fraction of the data.

Tools such as pandas, Polars, DuckDB, and Arrow read Parquet efficiently. DuckDB can even run SQL queries directly over Parquet files without loading them into memory.

Converting an existing archive takes a one-time effort. The payoff comes in every subsequent analysis.

For recording live ticks, write to a lightweight format first, and convert in batches to Parquet during quiet hours.

Better infrastructure makes more analysis possible, which leads to better decisions.

What storage format holds the tick archive, and how long does a typical query take?