Writing ticks to a file from an EA without slowing OnTick

[Fairman]

Writing ticks to a file from an EA without slowing OnTick

I log every tick from a couple of symbols for later analysis, and the first version slowed the EA badly during busy minutes. Here's what fixed it.

The slow version opened the file, wrote one line, and closed it on every tick. FileOpen and FileClose are expensive compared with everything else in OnTick. At the London open, EURUSD can produce dozens of ticks a second on some feeds, and the EA was lagging behind.

The faster version:
Open the file once in OnInit with FILE_WRITE, FILE_READ, FILE_CSV and FILE_SHARE_READ, and seek to the end.
In OnTick, append the tick to a string buffer rather than writing directly.
Every 100 ticks, or every second using OnTimer, write the buffer with FileWriteString and call FileFlush.
In OnDeinit, write anything left and close the file.

Each line holds time in milliseconds from SymbolInfoTick, bid, ask, and the flags field.

The other improvement was rolling the file daily. One file per symbol per day keeps sizes manageable and makes it easy to load a single day into Python.

On a busy morning the buffered version keeps up easily. The trade logic in the same EA is no longer delayed by disk writes.

If you don't trade from the same EA, a separate logging EA on its own chart is even safer, since a slow write can't affect trading at all.

[Fairman]

CopyTicks is another option if you don't need real-time logging. It pulls historical ticks from the terminal's own store for a time range. You can run it at the end of the day and save everything in one go. The catch is that the terminal's tick history depends on what the broker provides and how far back it keeps it, so check that the data you get back is complete for the hours you care about. Comparing the count against your live log for a day is a quick test. If your broker's history goes back only a few days, the live log becomes the only complete record you have.

[Fairman]

The flags field in MqlTick tells you which values changed on that tick, bid, ask, last or volume. Logging it lets you filter later for ticks where only the bid moved, for example. On forex feeds most ticks change bid and ask together, but not always. It's a small extra column and saves having to infer what changed by comparing with the previous row. For example, a tick where only the ask changed can be a sign of the spread widening, which you might want to study separately. It's the cheapest way to see spread behaviour in detail, with no extra data source needed.

[Fairman]

For millisecond timestamps, use time_msc from MqlTick rather than TimeCurrent. TimeCurrent only has second resolution, so several ticks in the same second would all get the same time. With time_msc you can see the order and spacing of ticks within a second, which matters for anything about latency or fast moves. Converting to a datetime in Python is just dividing by 1000 and treating it as server time. Note that time_msc is in server time like everything else in MetaTrader, so convert it with the same offset you use for bars. Mixing a converted tick file with unconverted bars puts trades in the wrong minute.

[Fairman]

Disk space adds up, so I compress files older than a week with gzip. pandas reads gzipped CSV directly, so nothing else changes. A month of two symbols compresses to a fraction of the raw size. If the VPS has limited disk, a small scheduled script to move old files off the machine avoids the day it fills up and the terminal starts failing in odd ways. A full disk can also stop the terminal saving its own logs and settings, which makes the failure harder to diagnose. A simple disk space check in the same script avoids it.

[FXS HongKong Scalper]

Opening the file once and writing through a buffer is the right fix. Another option is to collect the ticks in an array and write them in batches, for example every second or every 100 ticks, using a timer. That keeps OnTick very light. During busy moments the batch gets larger, but the work is done outside OnTick. It also helps to flush the file in OnDeinit, so no data is lost when the EA is stopped. With MqlTick and CopyTicks, collecting ticks in batches can be simpler still.