Handling tick data in pandas without running out of memory

[Amanhasnoname]

A month of tick data for one pair can contain tens of millions of rows. Loading everything into a dataframe at once often exhausts memory, especially with default data types.

Several practices help. Use compact data types: float32 for prices where the precision suffices, or integers for pipette-based prices, and categorical or integer codes for symbols. Store timestamps as 64-bit integers rather than strings.

Read in chunks, process each chunk, and aggregate results. Operations like computing per-second statistics can be done incrementally.

Filter columns and rows early. Load only the needed time range and fields.

Consider tools designed for larger data, such as Dask or Polars, which handle out-of-core processing and multithreading more efficiently.

Be careful with copies. Chained operations in pandas can create temporary copies that double memory use.

Profiling memory use at the start saves hours later. A well-structured pipeline can analyze years of ticks on an ordinary laptop.

Which tools or data types made the largest difference in handling tick datasets?