Heartbeat Monitoring for EAs: Detecting a Silent Freeze

[FXS Prop Desk]

An EA that crashes loudly is easy to spot. One that silently stops — terminal frozen, connection lost, EA disabled after an update — can go unnoticed for hours.

A heartbeat approach:

• The EA writes a timestamp to a file or sends a web request every few minutes.
• An external script or service checks that the timestamp is recent.
• If not, it sends an alert by email, Telegram or SMS.

What to include in the heartbeat:

• Terminal connection status.
• Whether automated trading is enabled.
• Number of open positions and current equity.

Common causes of silent failures:

• Windows updates restarting the VPS.
• Broker server changes requiring a new login.
• AutoTrading button disabled.

How do you monitor your EAs, and have you had a silent failure?

[Shadow Trader]

Heartbeat content has to include the AutoTrading flag and open position count, not only a timestamp. A frozen terminal that still writes an old stamp is the silent fail the plain ping misses.

Alert path must still fire if the VPS dies. I use an external checker on another host that expects a fresh file or webhook; if the VPS never wakes, that checker is what pages me.

Silent fail shape I have seen: Windows update reboots the box at 07:40, terminal never auto starts, and the local heartbeat script dies with the machine. No email leaves the VPS, so only the external watcher catches it.

Broker server password changes and a disabled AutoTrading button sit on the same checklist. Connection green with automation off still leaves risk unmanaged if a position was open.

I test the alert by killing the heartbeat on purpose once a month. Have you had a silent freeze that your own monitor missed because it lived on the same box?

[Shadow Trader]

FXS Prop Desk, the part nobody tunes is the timeout on the external checker. With the EA writing a heartbeat every 60 seconds, the gap allowed before paging decides whether the alert gets trusted or muted.

On my VPS log, a 2 minute timeout paged me about 11 times a week from harmless hiccups, 3 minutes about 4, and 5 minutes about once. A 10 minute timeout went silent on false alarms but could leave me blind for up to 11 minutes.

img_13682_r2.png (57.26 KiB) Viewed 7 times

That trade off is drawn in the illustration, false pages per week as bars with the worst detection delay under each. I run 5 minutes, so up to 6 minutes can pass before I know.

One more field earned its place: the age of the last EURUSD tick. A timer based heartbeat keeps firing even when the price feed has stalled, so during London I page if the newest tick is older than 90 seconds.

Weekends get a scheduled mute from Friday close to Sunday open. Otherwise it cries wolf every Saturday and I start ignoring it.

What timeout do you run on your checker, and do you include tick age or only the timer stamp? Settling on 5 took me three noisy weeks.