TimeProvider: test a 30-second retry policy in 13 milliseconds

A retry policy that waits 5 seconds, then 10, then 15 has a test that takes half a minute to pass. There are two usual ways out of that and neither is good. You can shrink the schedule under test configuration, and then the test no longer covers the schedule you ship. Or you can replace the delay with a no-op, which covers nothing about the timing at all.

.NET 8 added a third option. TimeProvider is an abstract clock that lives in System, and what separates it from the clock interface your codebase already has is that the framework's own waiting primitives accept one:

await Task.Delay(delay, timeProvider, cancellationToken);
await work.WaitAsync(timeout, timeProvider, cancellationToken);

using var timer = new PeriodicTimer(period, timeProvider);
using var cts = new CancellationTokenSource(timeout, timeProvider);

The docs list all four of those as .NET 8 through .NET 11, and I have run every one of them on .NET 10. One detail is worth having early, because it is the kind of thing a reader assumes and then wastes twenty minutes on: the cancellation source takes the provider in its constructor, and there is no CancelAfter overload that accepts one. A later CancelAfter on a source built that way does reset the delay against the same provider, which the demo asserts. A source built with the parameterless constructor stays on the system clock, and a fake clock cannot touch it.

One line in the policy

The retry loop is ordinary. The only line that decides whether this is testable is the wait:

catch (Exception ex) when (ex is not OperationCanceledException && attempt <= _backoff.Count)
{
    var delay = _backoff[attempt - 1];
    total += delay;

    await Task.Delay(delay, _time, cancellationToken);
}

_time is a TimeProvider passed to the constructor. In production it is TimeProvider.System and the behaviour is what it always was. In a test it is FakeTimeProvider, from the Microsoft.Extensions.TimeProvider.Testing package (10.9.0 as I write this), whose clock starts at 2000-01-01T00:00:00Z and does not move unless the test moves it. There is an AutoAdvanceAmount property that makes every read of the clock nudge it forward, and it defaults to zero.

The test I expected to work

var clock = new FakeTimeProvider();
var policy = new RetryPolicy(clock);

var run = policy.ExecuteAsync(FailsUntilAttempt(4));

clock.Advance(TimeSpan.FromSeconds(30));

var outcome = await run;      // sits here

Four attempts, three delays, thirty seconds of schedule, one advance of thirty seconds. That test does not fail; it sits there until the runner gives up. The first time I saw it I assumed I had the package version wrong.

Advance fires the timers that already exist

When the run starts, exactly one timer exists on the fake clock: the first Task.Delay. Advancing thirty seconds fires that one and nothing else, because nothing else is there to fire. The second delay's timer does not exist yet. It is created by the continuation of the first await, and where that continuation runs turns out not to matter: on .NET 10.0.5 I measured it running inline on the very thread that called Advance, both with a plain await and with ConfigureAwait(false), so the second timer already existed by the time Advance returned. What matters is that it is created against a clock that already reads thirty seconds past the start, so the ten-second delay it registers is due at forty, and it was never a candidate for the advance that just happened. Advancing again repeats the situation one step further along.

The demo asserts that with jumps far larger than the whole schedule:

await clock.WaitForTimersAsync(1);

clock.Advance(TimeSpan.FromMinutes(5));
await Task.Delay(Grace);
Assert.False(run.IsCompleted);
Assert.Equal(2, clock.TimersCreated);

clock.Advance(TimeSpan.FromMinutes(5));
await Task.Delay(Grace);
Assert.False(run.IsCompleted);
Assert.Equal(3, clock.TimersCreated);

Ten simulated minutes against thirty seconds of backoff, three timers created, and the run is still pending. Advancing before the operation starts is the same problem one step earlier: no timer exists at all, the jump fires nothing, and the first delay is then registered against the new now. The size of the jump was never the issue. A timer that is already armed does fire on a single advance, and the demo has a test that does only that, to keep the two ideas apart.

Waiting for the timer instead of racing it

FakeTimeProvider does not expose what it has pending. Every timer is created through TimeProvider.CreateTimer, though, so a wrapper can count them:

public sealed class TestClock : TimeProvider
{
    private readonly FakeTimeProvider _fake = new();
    private int _timersCreated;

    public int TimersCreated => Volatile.Read(ref _timersCreated);

    public void Advance(TimeSpan delta) => _fake.Advance(delta);

    public override ITimer CreateTimer(TimerCallback callback, object? state, TimeSpan dueTime, TimeSpan period)
    {
        var timer = _fake.CreateTimer(callback, state, dueTime, period);
        Interlocked.Increment(ref _timersCreated);
        return timer;
    }

    public override DateTimeOffset GetUtcNow() => _fake.GetUtcNow();
    public override long GetTimestamp() => _fake.GetTimestamp();
    public override TimeZoneInfo LocalTimeZone => _fake.LocalTimeZone;
    public override long TimestampFrequency => _fake.TimestampFrequency;
}

Incrementing after the inner call rather than before is the whole trick: when a test sees the count go up, the timer is already on the fake clock's books. WaitForTimersAsync polls that counter with a one-millisecond real delay and a five-second ceiling. That is a real wait and I would rather not have it. What it waits for is the operation reaching its next delay, which is a scheduling hop rather than a backoff: the whole three-step walk costs about 13 milliseconds on my machine.

The test then reads as start, wait for the timer, advance, repeat:

// clock is the TestClock above; startedAt = clock.GetUtcNow(), taken first
var run = policy.ExecuteAsync(FailsUntilAttempt(4));

for (var step = 0; step < policy.Backoff.Count; step++)
{
    await clock.WaitForTimersAsync(step + 1);
    clock.Advance(policy.Backoff[step]);
}

var outcome = await run.WaitAsync(TimeSpan.FromSeconds(5));

Assert.Equal(4, outcome.Attempts);
Assert.Equal(TimeSpan.FromSeconds(30), clock.GetUtcNow() - startedAt);

Proving the schedule rather than a log line

Asserting that four attempts happened proves the retry count and says nothing about the timing. The claim worth making is that the operation is still waiting one instant before the last delay is due:

// timers 1 and 2 walked exactly as in the loop above
await clock.WaitForTimersAsync(3);
clock.Advance(TimeSpan.FromSeconds(15) - TimeSpan.FromTicks(1));

Assert.False(run.IsCompleted);

clock.Advance(TimeSpan.FromTicks(1));

var outcome = await run.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Equal(4, outcome.Attempts);

A tick is 100 nanoseconds, so the first advance lands the clock at 29.9999999 seconds. That Assert.False is not a hopeful check with a sleep behind it: the only outstanding timer at that point is the last delay, it is not due, and nothing else in the process is going to complete the task. The tick that closes the gap is the one that completes it.

Fourteen tests, and here is what the suite actually prints:

  Passed TimeProviderTestingDemo.RetryScheduleTests.Whole_schedule_runs_in_milliseconds_of_real_time [14 ms]
  Passed TimeProviderTestingDemo.RetryScheduleTests.Still_pending_one_tick_before_the_last_delay_elapses [15 ms]
  Passed TimeProviderTestingDemo.EarlyAdvanceTests.One_big_jump_does_not_walk_a_chain_of_delays [501 ms]

Test Run Successful.
Total tests: 14
     Passed: 14
 Total time: 1.1608 Seconds

Thirty seconds of simulated backoff, 13 to 14 milliseconds of wall clock across five consecutive runs on Windows 11. The 501 ms in the middle is the two real grace periods that test uses to give a wrongly-ordered run a fair chance to finish.

What the interface you already wrote does not buy you

Most codebases I have seen wrap time in something like IClock with a UtcNow property, which is genuinely useful for the timestamps that end up in rows and log lines. It stops at the waiting. Task.Delay(delay) has never heard of your interface:

var frozen = new FrozenClock(DateTimeOffset.UnixEpoch);
var policy = new HandRolledRetryPolicy(
    frozen, TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(100));

hand-rolled clock: 200 ms of backoff took 218 ms of real time

The frozen clock still reads the Unix epoch at the end of that run, and the test still spent the full 200 milliseconds. I scaled that one down from the real schedule on purpose, because a half-minute test in a repo whose point is that tests should not sleep would have been a bad joke.

If you are on .NET Framework or netstandard2.0, the Microsoft.Bcl.TimeProvider package backfills the type together with extension methods that stand in for the instance overloads, timeProvider.Delay(...) and timeProvider.CreateCancellationTokenSource(...). I have not tested that path, so take it as a pointer rather than a recommendation.

Where I would not bother

Code that waits once, for a short fixed time, in a place where nobody will ever assert on the duration does not need a TimeProvider parameter. Threading one through is a change to a constructor signature and a dependency-injection registration, and it should buy something.

It buys a lot wherever the timing is the behaviour. Retry schedules, circuit breaker cool-off windows, token refresh a few minutes before expiry, a poller whose interval somebody will eventually tune in configuration. Those are the places where the test is the only thing standing between a schedule and a typo in it, and where a suite that runs the real schedule in milliseconds is worth the wrapper class.

The demo is at github.com/gdhami-net/timeprovider-testing-demo. dotnet test runs all fourteen; check.sh and check.ps1 run the same suite and print what each group covers. Both failure modes above are asserted rather than reproduced, so every test in the repo passes, including the two that describe what goes wrong.