Storing tick data without drowning the disk
A year of tick data for a single major pair can run to tens of millions of rows. Several pairs and several years add up to hundreds of gigabytes in plain text.
Efficient storage helps. Binary formats like Parquet or HDF5, compressed with a fast algorithm, can shrink files by an order of magnitude and read far faster than CSV.
Storing only what is needed is another saving. Timestamp, bid, ask, and perhaps volume are enough for most purposes. Extra columns multiply the size.
Partitioning by day or month makes it easy to load only the period under study. A database designed for time series can also handle queries quickly.
The unglamorous side of research is data management. Traders who set up a clean, compact, well-labelled store find that testing becomes much faster and mistakes much rarer.
A useful rule is keeping raw data untouched and storing cleaned versions separately, so that a mistake in cleaning never destroys the original.
What format is used for tick data, and how big has the archive grown?
Storing tick data without drowning the disk
Storing tick data without drowning the disk
It’s Fairman 
-
Amanhasnoname
- Posts: 2060
- Joined: Fri Oct 30, 2026 9:45 am
Re: Storing tick data without drowning the disk
My archive started as CSV files, one per pair per month, and after about three years and seven pairs it was well over a hundred and fifty gigabytes. Loading a single month for a test took long enough that I avoided running tests I should have run, which is the real cost of messy storage.
I converted everything to Parquet with zstd compression, partitioned by pair and day, and kept only timestamp, bid and ask. The same data came down to under twenty gigabytes, and reading a month of EURUSD went from several minutes to a few seconds. Being able to query only the hours I care about, such as the London open, made a bigger difference than I expected.
The rule about keeping raw data untouched saved me once. I wrote a cleaning step to remove spikes that accidentally dropped valid ticks around a central bank announcement, and I only noticed because a backtest result looked suspiciously smooth. Because the raw files were separate, I fixed the filter and rebuilt the clean set in an afternoon.
I converted everything to Parquet with zstd compression, partitioned by pair and day, and kept only timestamp, bid and ask. The same data came down to under twenty gigabytes, and reading a month of EURUSD went from several minutes to a few seconds. Being able to query only the hours I care about, such as the London open, made a bigger difference than I expected.
The rule about keeping raw data untouched saved me once. I wrote a cleaning step to remove spikes that accidentally dropped valid ticks around a central bank announcement, and I only noticed because a backtest result looked suspiciously smooth. Because the raw files were separate, I fixed the filter and rebuilt the clean set in an afternoon.