How much disk a month of tick data really takes

[Amanhasnoname]

Anyone planning serious tick-level testing runs into storage quickly. A single liquid pair can generate several million ticks in a month, and the files add up.

Raw data in text format is large. A year of EURUSD ticks with bid, ask and timestamps can run to tens of gigabytes uncompressed. Binary or compressed formats cut that by a factor of five or ten.

Multiply by the number of pairs and years wanted, and an external drive becomes necessary. Solid state storage speeds up the testing itself, since tick tests read data constantly.

Memory matters as well. Loading very large files into a spreadsheet or a naive script can freeze the machine. Tools designed for large data, such as databases or columnar formats, handle it better.

A sensible approach is to store the cleaned data once, in a compact format, and keep raw downloads only until verified.

Planning the storage before downloading a decade of data saves a lot of frustration.

What format and tools do you use to store and query tick data?