Storing tick data without drowning the disk

[Fairman]

Storing tick data without drowning the disk

A year of tick data for a single major pair can run to tens of millions of rows. Several pairs and several years add up to hundreds of gigabytes in plain text.

Efficient storage helps. Binary formats like Parquet or HDF5, compressed with a fast algorithm, can shrink files by an order of magnitude and read far faster than CSV.

Storing only what is needed is another saving. Timestamp, bid, ask, and perhaps volume are enough for most purposes. Extra columns multiply the size.

Partitioning by day or month makes it easy to load only the period under study. A database designed for time series can also handle queries quickly.

The unglamorous side of research is data management. Traders who set up a clean, compact, well-labelled store find that testing becomes much faster and mistakes much rarer.

A useful rule is keeping raw data untouched and storing cleaned versions separately, so that a mistake in cleaning never destroys the original.

What format is used for tick data, and how big has the archive grown?