Unit tests for trading code: boring and invaluable

[PTScalper]

2. The Fault-Tolerant Execution Wrapper

When the algorithm signals an entry, generate the ClientOrderId, lock the state, and wrap the network call in a try-catch block designed to trap transport-layer exceptions.

Code: Select all

public async Task TryEnterMarketAsync(string symbol, TradeDirection direction, double volume)
{
    await _executionLock.WaitAsync();
    try
    {
        if (_currentState != ExecutionState.Idle) return;

        _currentState = ExecutionState.Routing;
        _pendingClientOrderId = Guid.NewGuid().ToString("N"); // Idempotency key

        // Attempt execution
        var result = await _brokerApi.SendMarketOrderAsync(symbol, direction, volume, _pendingClientOrderId);

        if (result.IsSuccess)
        {
            _currentState = ExecutionState.InPosition;
            // Map broker's server-side Position ID to your local state
        }
        else
        {
            _currentState = ExecutionState.Idle; // Rejected by broker cleanly
        }
    }
    catch (TimeoutException) // Or HttpRequestException, SocketException
    {
        // The network dropped. The order might be sitting on the broker's server.
        _currentState = ExecutionState.Reconciling;
        
        // At this point, the execution lock is released, but _currentState prevents 
        // new trades until ReconcileStateAsync() clears it.
    }
    finally
    {
        _executionLock.Release();
    }
}

[PTScalper]

3. The Reconciliation Loop

The reconciliation method fires either via a polling mechanism or immediately upon the broker API's ConnectionRestored event. It queries the broker's source of truth to determine if the _pendingClientOrderId became a live position.

Code: Select all

private async Task ReconcileStateAsync()
{
    if (_currentState != ExecutionState.Reconciling) return;

    await _executionLock.WaitAsync();
    try
    {
        // 1. Query the broker's open positions
        var openPositions = await _brokerApi.GetOpenPositionsAsync();

        // 2. Search for the orphaned ID
        var ghostPosition = openPositions.FirstOrDefault(p => p.ClientOrderId == _pendingClientOrderId);

        if (ghostPosition != null)
        {
            // Scenario A: The order executed on the server before the network drop.
            // Action: Adopt the position and resume management.
            _currentState = ExecutionState.InPosition;
            AttachStopLossAndTakeProfit(ghostPosition); 
        }
        else
        {
            // 3. Query order history (did it execute and immediately hit a stop/take profit?)
            var history = await _brokerApi.GetHistoricalTradesAsync(TimeSpan.FromMinutes(5));
            var hitHistory = history.Any(h => h.ClientOrderId == _pendingClientOrderId);

            if (hitHistory)
            {
                // Scenario B: Order executed and closed while we were disconnected.
                _currentState = ExecutionState.Idle;
            }
            else
            {
                // Scenario C: The order never reached the matching engine.
                // Action: Safe to revert to Idle. 
                _currentState = ExecutionState.Idle;
                _pendingClientOrderId = null;
            }
        }
    }
    catch (Exception)
    {
        // Connection still unstable. Remain in Reconciling state and try again later.
    }
    finally
    {
        _executionLock.Release();
    }
}

[PTScalper]

Architectural Guardrails

Idempotency Keys: Never rely on the broker's integer PositionId or TicketNumber for initial tracking, because you won't receive it if the connection drops on the response trip. The UUID generated locally is your only tether to the order.

The Execution Lock: The SemaphoreSlim ensures that if a fast-moving M15 momentum candle triggers multiple entry signals during the few seconds the connection is resetting, the system drops the signals rather than stacking duplicate positions.

Timeouts: Ensure your API client has a strict, aggressive timeout (e.g., 2000ms). In scalping, a delayed fill is often worse than no fill. If the socket hangs, fail fast, trap the timeout, and transition to Reconciling.

[PTScalper]

To implement this in modern .NET, you should use Polly V8, which replaced the older Policy classes with the highly optimized, allocation-free ResiliencePipeline.

When the broker server is struggling—such as during Tier-1 news disconnects or a broker-side routing failure—hitting the API immediately with rapid-fire loops can result in rate-limiting or further socket drops. Exponential backoff with jitter solves this by spacing out queries and adding randomization, which prevents a "thundering herd" scenario if multiple automated strategies attempt to reconcile simultaneously.

Here is how to architect the resilient reconciliation loop using Polly V8 inside the OrderManager state machine.

[PTScalper]

1. Defining the Resilience Pipeline

Build the pipeline once and cache it (either via Dependency Injection or statically within the class). The configuration below defines an exponential backoff starting at 500ms, doubling on each attempt, with randomized jitter applied.

Code: Select all

using Polly;
using Polly.Retry;
using Polly.Timeout;
using System.Net.Sockets;

public class OrderManager
{
    private ExecutionState _currentState = ExecutionState.Idle;
    private readonly SemaphoreSlim _executionLock = new(1, 1);
    private readonly IExecutionProvider _brokerApi;
    private string _pendingClientOrderId; 
    
    // The cached Polly V8 Resilience Pipeline
    private readonly ResiliencePipeline _reconciliationPipeline;

    public OrderManager(IExecutionProvider brokerApi)
    {
        _brokerApi = brokerApi;
        
        // 1. Build the fault-tolerance pipeline
        _reconciliationPipeline = new ResiliencePipelineBuilder()
            .AddRetry(new RetryStrategyOptions
            {
                // Target only transient transport or API exceptions
                ShouldHandle = new PredicateBuilder()
                    .Handle<TimeoutException>()
                    .Handle<HttpRequestException>()
                    .Handle<SocketException>(),
                
                MaxRetryAttempts = 5,
                Delay = TimeSpan.FromMilliseconds(500),
                BackoffType = DelayBackoffType.Exponential,
                UseJitter = true, // Critical: Scatters retry exact-timing to avoid rate-limiting
                
                OnRetry = args => 
                {
                    // Log the retry for your telemetry pipeline (e.g., to MS SQL / local log)
                    Console.WriteLine($"[WARNING] Broker unstable. Reconciliation attempt {args.AttemptNumber} in {args.RetryDelay.TotalMilliseconds:N0}ms");
                    return ValueTask.CompletedTask;
                }
            })
            // 2. Wrap the entire retry block in an absolute timeout
            .AddTimeout(TimeSpan.FromSeconds(30)) 
            .Build();

        _brokerApi.OnConnectionRestored += async (s, e) => await ReconcileStateAsync(CancellationToken.None);
    }
}

[PTScalper]

2. Wrapping the Reconciliation Execution

Modify the ReconcileStateAsync method to pass its logic through the ExecuteAsync delegate of the pipeline. Notice that Polly manages the CancellationToken dynamically for you.

Code: Select all

private async Task ReconcileStateAsync(CancellationToken cancellationToken)
{
    if (_currentState != ExecutionState.Reconciling) return;

    await _executionLock.WaitAsync(cancellationToken);
    try
    {
        // Execute the broker queries through the Polly pipeline
        await _reconciliationPipeline.ExecuteAsync(async ct => 
        {
            // 1. Query the broker's open positions (passing Polly's cancellation token)
            var openPositions = await _brokerApi.GetOpenPositionsAsync(ct);
            var ghostPosition = openPositions.FirstOrDefault(p => p.ClientOrderId == _pendingClientOrderId);

            if (ghostPosition != null)
            {
                // Scenario A: Ghost order is live.
                _currentState = ExecutionState.InPosition;
                AttachStopLossAndTakeProfit(ghostPosition); 
            }
            else
            {
                // 2. Query order history for instant stop-outs
                var history = await _brokerApi.GetHistoricalTradesAsync(TimeSpan.FromMinutes(5), ct);
                var hitHistory = history.Any(h => h.ClientOrderId == _pendingClientOrderId);

                if (hitHistory || !hitHistory) 
                {
                    // Scenarios B & C: It closed quickly, or never reached the engine.
                    _currentState = ExecutionState.Idle;
                    _pendingClientOrderId = null;
                }
            }
        }, cancellationToken);
    }
    catch (TimeoutRejectedException)
    {
        // The overall 30-second pipeline timeout breached. The broker is completely dark.
        // Fallback action: Trigger an emergency alert to your phone / email.
        Console.WriteLine("CRITICAL: Reconciliation failed. Broker API unreachable for 30s.");
    }
    finally
    {
        _executionLock.Release();
    }
}

[PTScalper]

Key Engineering Details

Order of Operations (AddRetry vs AddTimeout): In Polly V8, strategies execute in the order they are added. Adding AddRetry before AddTimeout means the 30-second timeout acts as a global cap for all retries combined. If you reverse them, the timeout applies to each individual attempt.

Idempotency During Retries: Because you are reading state (GetOpenPositionsAsync) rather than writing state (SendMarketOrderAsync), the retries are mathematically pure. If you use Polly to wrap entry logic, you must ensure the broker supports idempotency keys natively, otherwise a retry on a timeout could result in two live positions.

[PTScalper]

To halt execution after three consecutive rejections, you must wrap the order entry method—not just the reconciliation loop—with a shared Circuit Breaker pipeline. In Polly V8, the circuit breaker monitors a stream of executions. Once the threshold is breached, it "opens," failing fast and immediately throwing a BrokenCircuitException for all subsequent calls until a predefined cooldown expires.

In Raw ECN environments like IC Markets, a rejection is rarely a transport exception; it is usually a successful API response containing a failure payload (e.g., ERR_OFF_QUOTES or ERR_LIQUIDITY_REJECTED). The circuit breaker must be typed to evaluate your ExecutionResult object, not just caught exceptions.

[PTScalper]

1. The Circuit Breaker Pipeline

Define a ResiliencePipeline<ExecutionResult> using CircuitBreakerStrategyOptions<ExecutionResult>. To trigger exactly on 3 consecutive failures, set the FailureRatio to 1.0 (100% failure rate) and MinimumThroughput to 3 within a specific time window.

Code: Select all

using Polly;
using Polly.CircuitBreaker;

public class OrderManager
{
    private readonly ResiliencePipeline<ExecutionResult> _executionPipeline;
    private readonly IExecutionProvider _brokerApi;
    private ExecutionState _currentState = ExecutionState.Idle;

    public OrderManager(IExecutionProvider brokerApi)
    {
        _brokerApi = brokerApi;

        _executionPipeline = new ResiliencePipelineBuilder<ExecutionResult>()
            .AddCircuitBreaker(new CircuitBreakerStrategyOptions<ExecutionResult>
            {
                // 1. Define what counts as a failure
                ShouldHandle = new PredicateBuilder<ExecutionResult>()
                    .HandleResult(result => !result.IsSuccess && IsHardRejection(result.ErrorCode))
                    .Handle<TimeoutException>(),

                // 2. Configure the "3 Consecutive" trigger
                FailureRatio = 1.0, 
                MinimumThroughput = 3, 
                SamplingDuration = TimeSpan.FromMinutes(2), 
                
                // 3. Define the cooldown period (e.g., bypassing Tier-1 news volatility)
                BreakDuration = TimeSpan.FromMinutes(15),

                // 4. Telemetry hooks
                OnOpened = args => 
                {
                    string reason = args.Outcome.Exception?.Message ?? args.Outcome.Result?.ErrorCode;
                    Console.WriteLine($"[EMERGENCY] Circuit tripped! Execution halted for 15m. Reason: {reason}");
                    // Fire webhook / push notification to your phone here
                    return ValueTask.CompletedTask;
                },
                OnClosed = args =>
                {
                    Console.WriteLine("[INFO] Circuit closed. Resuming normal execution.");
                    return ValueTask.CompletedTask;
                }
            })
            .Build();
    }

    private static bool IsHardRejection(string errorCode)
    {
        // Differentiate critical structural rejections from benign ones
        string[] hardErrors = { 
            "ERR_OFF_QUOTES",        // Liquidity provider rejected
            "ERR_INVALID_STOPS",     // Stop inside freeze level
            "ERR_NOT_ENOUGH_MONEY"   // Margin exhaustion
        };
        return hardErrors.Contains(errorCode);
    }
}

[PTScalper]

2. Wrapping the Entry Execution

When you route the order through _executionPipeline.ExecuteAsync(), Polly tracks the outcome. If the circuit is open, Polly will not even attempt to call the broker; it instantly throws a BrokenCircuitException, protecting your risk capital and preventing the broker from rate-limiting your account for spamming bad orders.

Code: Select all

public async Task TryEnterMarketAsync(string symbol, TradeDirection direction, double volume)
{
    // Await your semaphore lock here as previously defined...
    
    try
    {
        _currentState = ExecutionState.Routing;
        string pendingClientOrderId = Guid.NewGuid().ToString("N");

        // Route through the Circuit Breaker
        var result = await _executionPipeline.ExecuteAsync(async ct =>
        {
            return await _brokerApi.SendMarketOrderAsync(symbol, direction, volume, pendingClientOrderId);
        }, CancellationToken.None);

        if (result.IsSuccess)
        {
            _currentState = ExecutionState.InPosition;
        }
        else
        {
            // It failed, but the circuit hasn't tripped yet (e.g., attempt 1 or 2)
            _currentState = ExecutionState.Idle; 
        }
    }
    catch (BrokenCircuitException)
    {
        // The circuit is open. The execution was blocked locally.
        _currentState = ExecutionState.Idle;
        Console.WriteLine($"[BLOCKED] Trade on {symbol} aborted. Circuit breaker is currently active.");
    }
    catch (TimeoutException)
    {
        // Transition to Reconciling state as previously architected
        _currentState = ExecutionState.Reconciling;
    }
    finally
    {
        // Release semaphore lock...
    }
}