Dead-Man Heartbeats: Knowing an Automated System Is Still Alive
Posted: Wed Oct 28, 2026 2:05 am
The most dangerous failure of an automated system is silence. A crashed process produces no errors, no alerts and no trades, and a trader checking later may find hours of missed opportunity or, worse, an unmanaged open position.
A heartbeat reverses the logic. The system sends a small signal at regular intervals to an external monitor, and the monitor raises the alarm when a signal fails to arrive. Because the alarm depends on absence, it works even when the whole machine, network or power supply has failed, which internal error handling cannot cover.
A basic heartbeat shows only that the process runs. More useful ones attach content: the time of the last received price, the number of open positions, current equity and the last order result. Watching these values reveals cases where the process is alive but broken, such as running with a frozen feed or with every order rejected.
Thresholds need thought. Too tight, and every brief network hiccup produces a notification; too loose, and a real outage goes unnoticed. Separating warning from critical levels helps, as does a second notification channel in case the first fails. Periodic review of the alerts that fired, and those that should have, keeps the monitoring honest.
A heartbeat reverses the logic. The system sends a small signal at regular intervals to an external monitor, and the monitor raises the alarm when a signal fails to arrive. Because the alarm depends on absence, it works even when the whole machine, network or power supply has failed, which internal error handling cannot cover.
A basic heartbeat shows only that the process runs. More useful ones attach content: the time of the last received price, the number of open positions, current equity and the last order result. Watching these values reveals cases where the process is alive but broken, such as running with a frozen feed or with every order rejected.
Thresholds need thought. Too tight, and every brief network hiccup produces a notification; too loose, and a real outage goes unnoticed. Separating warning from critical levels helps, as does a second notification channel in case the first fails. Periodic review of the alerts that fired, and those that should have, keeps the monitoring honest.