Broker APIs and data providers limit how many requests a client can make per second, minute or day. Exceeding the limit produces errors, throttling or temporary bans. For automated strategies, rate limits are a design constraint.
Reading the documentation is the first step. Limits may differ by endpoint, with order placement handled more generously than market data queries. Some APIs return headers showing remaining capacity.
Design patterns help. Cache data that doesn't change quickly. Batch requests where supported. Use streaming connections instead of polling for prices. Implement exponential backoff on errors, waiting longer after each failure rather than hammering the server.
Logging every request and response code reveals how close the system runs to limits. Alerts on repeated errors catch problems early.
Politeness also matters ethically. Free data providers survive on goodwill, and abusive clients ruin it for others.
Unexpected behaviour at peak times is common. A strategy tested on a quiet day may hit limits during a news spike, exactly when it needs data most.
A simple pattern is a token bucket: the program keeps a count of available requests that refills at the allowed rate, and waits when the count reaches zero. A few lines of code prevent nearly all rate-limit errors, and the logic is easy to test.
Has an API limit ever caused a strategy to misbehave, and how was it redesigned?
API rate limits and polite data fetching
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
-
FXS Prop Desk
- Posts: 1689
- Joined: Wed Oct 07, 2026 9:52 am
Re: API rate limits and polite data fetching
Order endpoints usually have far tighter limits than market-data endpoints, and that asymmetry should shape the design: cache quotes locally, poll positions on a timer, and reserve order calls for actual orders. A strategy that hits a limit during a volatile minute can find its close order throttled, which is a risk problem, not just a coding one. Exponential backoff with a hard cap and an alert covers most of it.
FXS Official Team | The Idea Lab & Strategy
-
Amanhasnoname
- Posts: 2060
- Joined: Fri Oct 30, 2026 9:45 am
Re: API rate limits and polite data fetching
Yes, and it happened at the worst possible moment. My first semi-automated setup polled the broker for prices every half second and checked open positions on every tick. On quiet days it ran fine for weeks. Then one NFP release the price calls started coming back with errors, and the close order I needed went into the same queue and got throttled for a few seconds while the market ran against me.
The redesign was mostly about separating the jobs. Prices now come through the streaming connection, so I barely make price requests at all. Position checks run on a slow timer, every few seconds, and the order calls have their own budget that nothing else is allowed to touch. If the bucket for orders gets low, the program stops opening new trades but always keeps room for closes.
I also log every response code with a timestamp. Looking back over a month of logs showed I was running at nearly 70 percent of the limit during London open, which I never would have guessed from watching the screen.
The redesign was mostly about separating the jobs. Prices now come through the streaming connection, so I barely make price requests at all. Position checks run on a slow timer, every few seconds, and the order calls have their own budget that nothing else is allowed to touch. If the bucket for orders gets low, the program stops opening new trades but always keeps room for closes.
I also log every response code with a timestamp. Looking back over a month of logs showed I was running at nearly 70 percent of the limit during London open, which I never would have guessed from watching the screen.