Creating Background Services in ASP.NET Core and .NET

Created with: ASP.NET Core 8 and .NET 8
Level: Intermediate — Prerequisites: C#, ASP.NET Core, Dependency Injection

Table of Contents

1. Course Overview
2. Hosted Services in ASP.NET Core
   • 2.1 What is a Hosted Service?
   • 2.2 Use Cases
   • 2.3 IHostedService Interface — Full Detail
   • 2.4 BackgroundService Abstract Class
   • 2.5 WeatherCacheService — TennisBookings Demo
   • 2.6 PeriodicTimer vs Task.Delay
   • 2.7 System.Threading.Channels — Inter-service Communication
   • 2.8 FileProcessingChannel — TennisBookings Demo
   • 2.9 Scoped Services in a BackgroundService
3. Building .NET Worker Services
   • 3.1 What is a Worker Service?
   • 3.2 dotnet new worker Template
   • 3.3 .NET Generic Hosting
   • 3.4 ScoreProcessor Architecture — Demo
   • 3.5 QueueReadingService (producer)
   • 3.6 ScoreProcessingService (consumer)
   • 3.7 Web Application Refactoring
4. Advanced Hosted Service Concepts
   • 4.1 BackgroundService Implementation Details
   • 4.2 Exception Handling
   • 4.3 IHostApplicationLifetime — Application Lifecycle
   • 4.4 Service Registration Order
   • 4.5 Graceful Shutdown Timeout
   • 4.6 Overriding StartAsync and StopAsync
   • 4.7 Avoiding Blocking Code in StartAsync
   • 4.8 Unit Testing Worker Services
   • 4.9 Polly — Retry Policies
   • 4.10 Health Checks for Worker Services
   • 4.11 Structured Logging in Hosted Services
5. Running in Production
   • 5.1 Docker (recommended)
   • 5.2 Windows Service
   • 5.3 Linux Daemon (systemd)
   • 5.4 Azure App Service / WebJobs
   • 5.5 Azure Container Apps
   • 5.6 Deployment Options Comparison
6. Alternatives: Quartz.NET and Hangfire
7. Summary and Best Practices
8. Review Questions

1. Course Overview

Main Objectives

• Implement hosted services in ASP.NET Core applications
• Build .NET Worker Services for a microservices architecture
• Coordinate work between requests and hosted services via Channels
• Handle exceptions, lifecycle management, and graceful shutdown
• Deploy Worker Services to production (Docker, Windows Service, Linux daemon, Azure)

Demo Application: TennisBookings

A tennis court booking system with:

• An ASP.NET Core Razor Pages website for members
• An MVC admin area
• Several simulated external APIs (weather, player rankings, statistics)
• Processing of tennis result files (CSV) via AWS S3 + SQS

graph TD
    subgraph "TennisBookings Solution"
        Web["TennisBookings\n(ASP.NET Core Web App)"]
        SP["TennisBookings.ScoreProcessor\n(.NET Worker Service)"]
        RP["TennisBookings.ResultsProcessing\n(.NET Standard Library)"]
        Web -->|references| RP
        SP -->|references| RP
    end
    subgraph "Simulated External Services"
        WAPI["WeatherApi\n(:5001)"]
        TAPI["TennisPlayerApi\n(:5002)"]
        SAPI["StatisticsApi\n(:5003)"]
    end
    Web --> WAPI
    Web --> TAPI
    Web --> SAPI

Technical Prerequisites

2. Hosted Services in ASP.NET Core

2.1 What is a Hosted Service?

A hosted service is a background task that runs outside the HTTP request pipeline in an ASP.NET Core application. It is based on the IHostedService interface or the BackgroundService abstract class.

graph TD
    App["ASP.NET Core Application\n(Generic Host)"] --> Pipeline["HTTP Request Pipeline\n(Controllers, Middlewares, Razor Pages)"]
    App --> BG1["Hosted Service 1\n(WeatherCacheService)"]
    App --> BG2["Hosted Service 2\n(FileProcessingService)"]
    App --> BG3["Hosted Service N\n(...)"]
    Pipeline --> User["HTTP Response to User"]
    BG1 --> Cache["Weather cache refresh\n(continuous, outside requests)"]
    BG2 --> Files["Uploaded file processing\n(outside requests)"]
    style Pipeline fill:#4CAF50,color:#fff
    style BG1 fill:#2196F3,color:#fff
    style BG2 fill:#FF9800,color:#fff
    style BG3 fill:#9C27B0,color:#fff

Key point: Hosted services run concurrently with the HTTP pipeline within the same process. They start with the application and stop with it.

2.2 Hosted Service Use Cases

⚠️ Warning: If you have many hosted services unrelated to the main responsibility of the web application, consider moving them to separate Worker Services (independent microservices).

2.3 IHostedService Interface — Full Detail

// Defined in Microsoft.Extensions.Hosting since .NET Core 2.0
public interface IHostedService
{
    // Called at application startup
    // The host waits for StartAsync to complete before starting the next service
    // → MUST complete quickly (do not do long work here)
    Task StartAsync(CancellationToken cancellationToken);

    // Called during graceful application shutdown (SIGTERM, Ctrl+C, StopApplication())
    // cancellationToken = forced shutdown token (triggered after ShutdownTimeout)
    Task StopAsync(CancellationToken cancellationToken);
}

Direct IHostedService implementation (rare, useful for special cases):

// Example: service that runs once at startup
public class DatabaseMigrationService : IHostedService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<DatabaseMigrationService> _logger;

    public DatabaseMigrationService(
        IServiceProvider serviceProvider,
        ILogger<DatabaseMigrationService> logger)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Applying database migrations...");
        using var scope = _serviceProvider.CreateScope();
        var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();
        await dbContext.Database.MigrateAsync(cancellationToken);
        _logger.LogInformation("Migrations applied successfully.");
    }

    public Task StopAsync(CancellationToken cancellationToken)
        => Task.CompletedTask;
}

2.4 BackgroundService Abstract Class

Since .NET Core 2.1, BackgroundService encapsulates the repetitive IHostedService boilerplate:

// Simplified source of BackgroundService (GitHub: dotnet/runtime)
public abstract class BackgroundService : IHostedService, IDisposable
{
    private Task? _executeTask;
    private CancellationTokenSource? _stoppingCts;

    // Public property to observe the task status (useful for testing)
    public virtual Task? ExecuteTask => _executeTask;

    // ★ THE only method you must implement — your background logic
    protected abstract Task ExecuteAsync(CancellationToken stoppingToken);

    public virtual Task StartAsync(CancellationToken cancellationToken)
    {
        // Creates a CancellationTokenSource linked to the passed token
        _stoppingCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);

        // Launches ExecuteAsync WITHOUT awaiting it → does not block StartAsync
        _executeTask = ExecuteAsync(_stoppingCts.Token);

        // If ExecuteAsync completed immediately (e.g. exception), return its task
        if (_executeTask.IsCompleted)
            return _executeTask;

        return Task.CompletedTask;
    }

    public virtual async Task StopAsync(CancellationToken cancellationToken)
    {
        if (_executeTask is null) return;

        try
        {
            // Signal cancellation to ExecuteAsync
            _stoppingCts!.Cancel();
        }
        finally
        {
            // Wait for either ExecuteAsync to finish or the shutdown timeout
            await Task.WhenAny(
                _executeTask,
                Task.Delay(Timeout.Infinite, cancellationToken));
        }
    }

    public virtual void Dispose() => _stoppingCts?.Cancel();
}

Critical point: ExecuteAsync is not awaited in StartAsync. Startup does not wait for ExecuteAsync to finish. However: any synchronous code before the first await in ExecuteAsync still blocks startup.

2.5 Complete Example: WeatherCacheService

Problem Context

The TennisBookings home page displays the current weather. Without a hosted service, an HTTP call to the weather API is made on every cache expiration → poor UX.

Solution: A BackgroundService that refreshes the weather cache in the background.

// WeatherCacheService.cs
public class WeatherCacheService : BackgroundService
{
    private readonly IWeatherApiClient _weatherApiClient;
    private readonly IDistributedCache<WeatherResult> _cache;
    private readonly ILogger<WeatherCacheService> _logger;
    private readonly int _minutesToCache;
    private readonly int _refreshIntervalInSeconds;

    public WeatherCacheService(
        IWeatherApiClient weatherApiClient,
        IDistributedCache<WeatherResult> cache,
        IOptionsMonitor<ExternalServicesConfiguration> options,
        ILogger<WeatherCacheService> logger)
    {
        _weatherApiClient = weatherApiClient;
        _cache = cache;
        _logger = logger;
        var config = options.Get(ExternalServicesConfiguration.WeatherApi);
        _minutesToCache = config.MinsToCache;
        _refreshIntervalInSeconds = _minutesToCache > 1
            ? (_minutesToCache - 1) * 60
            : 30;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation(
            "WeatherCacheService started. Interval: {Seconds}s",
            _refreshIntervalInSeconds);

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await RefreshWeatherCacheAsync(stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error refreshing weather cache.");
            }

            await Task.Delay(
                TimeSpan.FromSeconds(_refreshIntervalInSeconds),
                stoppingToken);
        }
    }

    private async Task RefreshWeatherCacheAsync(CancellationToken ct)
    {
        var forecast = await _weatherApiClient
            .GetWeatherForecastAsync("Eastbourne", ct);

        if (forecast is not null)
        {
            var currentWeather = new WeatherResult
            {
                City = "Eastbourne",
                Weather = forecast.Weather
            };
            var cacheKey = $"current_weather_{DateTime.UtcNow:yyyy_MM_dd}";
            _logger.LogInformation("Weather cache updated for {City}.", currentWeather.City);
            await _cache.SetAsync(cacheKey, currentWeather, _minutesToCache);
        }
    }
}

Registration in Program.cs

// Program.cs (ASP.NET Core 8)
var builder = WebApplication.CreateBuilder(args);

builder.Services.Configure<ExternalServicesConfiguration>(
    ExternalServicesConfiguration.WeatherApi,
    builder.Configuration.GetSection("ExternalServices:WeatherApi"));

builder.Services.AddHttpClient<IWeatherApiClient, WeatherApiClient>(client =>
{
    client.BaseAddress = new Uri(
        builder.Configuration["ExternalServices:WeatherApi:Url"]!);
});

builder.Services.AddDistributedMemoryCache();
builder.Services.AddSingleton(typeof(IDistributedCache<>), typeof(DistributedCache<>));

// ★ Register the hosted service
builder.Services.AddHostedService<WeatherCacheService>();

var app = builder.Build();

2.6 PeriodicTimer vs Task.Delay

Since .NET 6, PeriodicTimer is the recommended way to implement periodic loops.

// ❌ Old approach with Task.Delay — temporal drift
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        await DoWorkAsync(stoppingToken); // e.g. takes 3s
        await Task.Delay(10000, stoppingToken); // + 10s → real period = 13s!
    }
}

// ✅ New approach with PeriodicTimer (.NET 6+)
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    // FIXED interval of 10 seconds — regardless of work duration
    using var timer = new PeriodicTimer(TimeSpan.FromSeconds(10));

    // WaitForNextTickAsync returns false if the timer is stopped/cancelled
    while (await timer.WaitForNextTickAsync(stoppingToken))
    {
        await DoWorkAsync(stoppingToken);
    }
}

PeriodicTimer vs Task.Delay Comparison

Complete Example with PeriodicTimer

// PlayerRankingCacheService.cs
public class PlayerRankingCacheService : BackgroundService
{
    private readonly IPlayerRankingApiClient _apiClient;
    private readonly IMemoryCache _cache;
    private readonly ILogger<PlayerRankingCacheService> _logger;

    public PlayerRankingCacheService(
        IPlayerRankingApiClient apiClient,
        IMemoryCache cache,
        ILogger<PlayerRankingCacheService> logger)
    {
        _apiClient = apiClient;
        _cache = cache;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Load immediately at startup
        await UpdateRankingsAsync(stoppingToken);

        // Then refresh every hour
        using var timer = new PeriodicTimer(TimeSpan.FromHours(1));

        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            await UpdateRankingsAsync(stoppingToken);
        }
    }

    private async Task UpdateRankingsAsync(CancellationToken ct)
    {
        try
        {
            var rankings = await _apiClient.GetTopRankingsAsync(ct);
            if (rankings?.Any() == true)
            {
                _cache.Set("player_rankings", rankings,
                    new MemoryCacheEntryOptions
                    {
                        AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2)
                    });
                _logger.LogInformation("{Count} rankings updated.", rankings.Count);
            }
        }
        catch (Exception ex) when (!ct.IsCancellationRequested)
        {
            _logger.LogError(ex, "Error updating player rankings.");
        }
    }
}

2.7 System.Threading.Channels — Inter-service Communication

Channels allow passing work from controllers to hosted services in a thread-safe manner, without manual locks.

graph LR
    subgraph "HTTP Thread (Producer)"
        C1["Controller A"] -->|TryWrite| CW["ChannelWriter<T>"]
        C2["Controller B"] -->|TryWrite| CW
    end
    subgraph "Channel (internal buffer)"
        CW --> BUF["Internal Buffer\n(bounded or unbounded)"]
        BUF --> CR["ChannelReader<T>"]
    end
    subgraph "Background Thread (Consumer)"
        CR -->|ReadAllAsync| SVC["BackgroundService"]
    end
    style CW fill:#4CAF50,color:#fff
    style BUF fill:#2196F3,color:#fff
    style CR fill:#FF9800,color:#fff

Bounded vs Unbounded Channel

// UNBOUNDED Channel — unlimited capacity
// RISK: infinite memory consumption if producer is faster
var unboundedChannel = Channel.CreateUnbounded<string>(new UnboundedChannelOptions
{
    SingleWriter = true,
    SingleReader = true
});

// BOUNDED Channel — limited capacity, implements backpressure
var boundedChannel = Channel.CreateBounded<string>(new BoundedChannelOptions(capacity: 100)
{
    SingleWriter = false,
    SingleReader = true,
    // Strategy when the channel is full:
    FullMode = BoundedChannelFullMode.Wait           // Wait (recommended)
    // FullMode = BoundedChannelFullMode.DropOldest  // Drop the oldest
    // FullMode = BoundedChannelFullMode.DropNewest  // Ignore the new item
});

Complete ChannelReader / ChannelWriter API

Channel<string> channel = Channel.CreateBounded<string>(10);
ChannelWriter<string> writer = channel.Writer;
ChannelReader<string> reader = channel.Reader;

// ─── PRODUCER SIDE ────────────────────────────────────────────────────────

// 1. TryWrite: non-blocking attempt
bool success = writer.TryWrite("report.csv");

// 2. WaitToWriteAsync + TryWrite: recommended pattern
if (await writer.WaitToWriteAsync(cancellationToken))
    writer.TryWrite("report.csv");

// 3. WriteAsync: combines WaitToWriteAsync + TryWrite
await writer.WriteAsync("report.csv", cancellationToken);

// 4. Signal end of production
writer.Complete();                         // Normal close
writer.Complete(new Exception("Error"));  // Close with exception

// ─── CONSUMER SIDE ──────────────────────────────────────────────────────

// 1. TryRead: non-blocking
if (reader.TryRead(out var item))
    Console.WriteLine(item);

// 2. ReadAsync: single item, waits if empty
var single = await reader.ReadAsync(cancellationToken);

// 3. WaitToReadAsync + TryRead
while (await reader.WaitToReadAsync(cancellationToken))
    while (reader.TryRead(out var msg))
        Console.WriteLine(msg);

// 4. ★ ReadAllAsync — async stream (C# 8) — the most elegant
// Terminates automatically when Writer.Complete() is called
await foreach (var item in reader.ReadAllAsync(cancellationToken))
    Console.WriteLine(item);

2.8 FileProcessingChannel — TennisBookings Demo

The Problem

Uploading a CSV file can take several seconds. Without a hosted service, the user waits for all processing to complete.

Solution: Offload processing to a BackgroundService via a Channel.

sequenceDiagram
    participant U as User
    participant C as ResultsController
    participant Ch as FileProcessingChannel
    participant S as FileProcessingService
    participant RP as ResultsProcessing

    U->>C: POST /admin/results (CSV file)
    C->>C: Save temporary file
    C->>Ch: AddFileAsync("temp/file.csv")
    Ch-->>C: true (added to channel)
    C-->>U: 202 Redirect (immediate!)

    Note over S: In the background...
    Ch->>S: ReadAllAsync() -> "temp/file.csv"
    S->>RP: ProcessAsync(stream)
    RP-->>S: Results processed
    S->>S: File.Delete("temp/file.csv")

FileProcessingChannel

// FileProcessingChannel.cs
public class FileProcessingChannel
{
    private const int MaxMessagesInChannel = 100;
    private readonly Channel<string> _channel;
    private readonly ILogger<FileProcessingChannel> _logger;

    public FileProcessingChannel(ILogger<FileProcessingChannel> logger)
    {
        _logger = logger;
        var options = new BoundedChannelOptions(MaxMessagesInChannel)
        {
            SingleWriter = false,
            SingleReader = true,
            FullMode = BoundedChannelFullMode.Wait
        };
        _channel = Channel.CreateBounded<string>(options);
    }

    // Producer method (called from the controller)
    public async Task<bool> AddFileAsync(string fileName,
        CancellationToken ct = default)
    {
        while (await _channel.Writer.WaitToWriteAsync(ct) &&
               !ct.IsCancellationRequested)
        {
            if (_channel.Writer.TryWrite(fileName))
            {
                _logger.LogTrace("File {FileName} added to channel.", fileName);
                return true;
            }
        }
        return false;
    }

    // Consumer method — IAsyncEnumerable for await foreach
    public IAsyncEnumerable<string> ReadAllAsync(CancellationToken ct = default)
        => _channel.Reader.ReadAllAsync(ct);

    // Number of files waiting (useful for health checks)
    public int FileCount => _channel.Reader.Count;

    public bool TryCompleteWriter(Exception? ex = null)
        => _channel.Writer.TryComplete(ex);
}

ResultsController (producer)

[Area("Admin")]
public class ResultsController : Controller
{
    private readonly FileProcessingChannel _fileProcessingChannel;
    private readonly ILogger<ResultsController> _logger;

    public ResultsController(
        FileProcessingChannel fileProcessingChannel,
        ILogger<ResultsController> logger)
    {
        _fileProcessingChannel = fileProcessingChannel;
        _logger = logger;
    }

    [HttpPost("admin/results/upload")]
    public async Task<IActionResult> UploadFile(IFormFile file, CancellationToken ct)
    {
        if (file is null || file.Length == 0)
            return BadRequest("Invalid file.");

        var tempFile = Path.Combine(Path.GetTempPath(), $"{Guid.NewGuid()}.csv");

        await using (var stream = System.IO.File.Create(tempFile))
        {
            await file.CopyToAsync(stream, ct);
        }

        var added = await _fileProcessingChannel.AddFileAsync(tempFile, ct);

        if (!added)
        {
            System.IO.File.Delete(tempFile);
            return StatusCode(503, "Service temporarily unavailable.");
        }

        _logger.LogInformation(
            "File {FileName} submitted. Pending: {Count}",
            file.FileName, _fileProcessingChannel.FileCount);

        return RedirectToAction("UploadConfirmation");
    }
}

FileProcessingService (consumer)

// FileProcessingService.cs
public class FileProcessingService : BackgroundService
{
    private readonly FileProcessingChannel _fileProcessingChannel;
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<FileProcessingService> _logger;

    public FileProcessingService(
        FileProcessingChannel fileProcessingChannel,
        IServiceProvider serviceProvider,
        ILogger<FileProcessingService> logger)
    {
        _fileProcessingChannel = fileProcessingChannel;
        _serviceProvider = serviceProvider;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("FileProcessingService started.");

        await foreach (var fileName in _fileProcessingChannel
            .ReadAllAsync()
            .WithCancellation(stoppingToken))
        {
            await ProcessFileAsync(fileName, stoppingToken);
        }
    }

    private async Task ProcessFileAsync(string fileName, CancellationToken ct)
    {
        _logger.LogInformation("Processing file: {FileName}", fileName);

        try
        {
            // ★ Create a DI scope to access Scoped services (e.g. DbContext)
            using var scope = _serviceProvider.CreateScope();
            var processor = scope.ServiceProvider
                .GetRequiredService<IResultProcessor>();

            await using var stream = File.OpenRead(fileName);
            // We do NOT pass ct to avoid partial processing
            await processor.ProcessAsync(stream, CancellationToken.None);

            _logger.LogInformation("File processed successfully: {FileName}", fileName);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing file: {FileName}", fileName);
        }
        finally
        {
            if (File.Exists(fileName))
                File.Delete(fileName);
        }
    }
}

Registration (Program.cs)

builder.Services.AddSingleton<FileProcessingChannel>();
builder.Services.AddScoped<IResultProcessor, TennisResultProcessor>();
builder.Services.AddHostedService<FileProcessingService>();

Important: BackgroundService instances are Singletons. To access Scoped services (e.g. DbContext), you must create a scope manually via IServiceProvider.CreateScope().

2.9 Scoped Services in a BackgroundService

The Problem

// ❌ BAD — Exception at startup: cannot consume scoped service from singleton
public class MyService : BackgroundService
{
    public MyService(AppDbContext dbContext) { } // ← InvalidOperationException!
}

Solution: IServiceScopeFactory (recommended)

// ✅ GOOD — Use IServiceScopeFactory
public class DataSyncService : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<DataSyncService> _logger;

    public DataSyncService(IServiceScopeFactory scopeFactory,
        ILogger<DataSyncService> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(5));

        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            // New scope for each iteration → DbContext.Dispose() correct
            await using var scope = _scopeFactory.CreateAsyncScope();
            var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();
            var repo = scope.ServiceProvider.GetRequiredService<IMatchRepository>();

            var matches = await dbContext.Matches
                .Where(m => !m.IsSynced)
                .ToListAsync(stoppingToken);

            foreach (var match in matches)
            {
                await repo.SyncToExternalSystemAsync(match, stoppingToken);
                match.IsSynced = true;
            }

            await dbContext.SaveChangesAsync(stoppingToken);
            _logger.LogInformation("{Count} matches synchronized.", matches.Count);
        }
    }
}

3. Building .NET Worker Services

3.1 What is a Worker Service?

A Worker Service is essentially a .NET console application that uses the generic hosting library.

graph TD
    subgraph "Worker Service = Console App + Generic Host"
        GH["Generic Host\n(.NET Hosting Library)"]
        GH --> DI["Dependency Injection\n(IServiceCollection)"]
        GH --> LOG["Logging\n(ILogger, Serilog, etc.)"]
        GH --> CFG["Configuration\n(appsettings.json, env vars)"]
        GH --> LT["Lifecycle Management\n(IHostApplicationLifetime)"]
        GH --> HS["Hosted Services\n(IHostedService)"]
    end
    subgraph "Use Cases"
        WQ["Message processing\n(SQS, RabbitMQ, Kafka)"]
        WF["File reaction\n(Blob Storage events)"]
        WD["Ingestion pipelines\n(ETL, data enrichment)"]
        WS["Scheduled jobs\n(reports, cleanup)"]
    end
    GH --> WQ
    GH --> WF
    GH --> WD
    GH --> WS

Differences from a Hosted Service in a Web App:

3.2 dotnet new worker Template

# Via CLI
dotnet new worker -n TennisBookings.ScoreProcessor
cd TennisBookings.ScoreProcessor

Generated Project Structure

TennisBookings.ScoreProcessor/
├── TennisBookings.ScoreProcessor.csproj
├── Program.cs              <- Host configuration
├── Worker.cs               <- Default BackgroundService
├── appsettings.json
└── appsettings.Development.json

Generated Program.cs (.NET 8)

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddHostedService<Worker>();

var host = builder.Build();
host.Run();

Generated Worker.cs (template)

public class Worker : BackgroundService
{
    private readonly ILogger<Worker> _logger;

    public Worker(ILogger<Worker> logger) => _logger = logger;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
            await Task.Delay(1000, stoppingToken);
        }
    }
}

Generated .csproj

<Project Sdk="Microsoft.NET.Sdk.Worker">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="8.0.0" />
  </ItemGroup>
</Project>

Note: Microsoft.NET.Sdk.Worker is different from Microsoft.NET.Sdk.Web. It does not include Kestrel or the HTTP pipeline.

3.3 .NET Generic Hosting

Historical Evolution

timeline
    title .NET Host Evolution
    .NET Core 2.0 : WebHost (coupled to ASP.NET Core)
    .NET Core 2.1 : Generic Host (separated from WebHost)
    .NET Core 3.0 : Unification - ASP.NET Core uses Generic Host
    .NET 6 : WebApplicationBuilder (syntactic sugar on Generic Host)
    .NET 7 : HostApplicationBuilder (equivalent for Worker Services)
    .NET 8 : Stabilization and minor improvements

Important HostApplicationBuilder Properties

var builder = Host.CreateApplicationBuilder(args);

// Services — dependency injection
builder.Services.AddSingleton<MyService>();
builder.Services.AddHostedService<MyBackgroundService>();

// Configuration — add sources
builder.Configuration.AddEnvironmentVariables(prefix: "MYAPP_");
builder.Configuration.AddJsonFile("custom.json", optional: true);

// Logging — customization
builder.Logging.ClearProviders();
builder.Logging.AddConsole();
builder.Logging.AddJsonConsole();

// Host Options
builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromSeconds(60);
    options.ServicesStartConcurrently = false;
    options.ServicesStopConcurrently = true; // .NET 8
});

var host = builder.Build();
host.Run();

Default Configuration Sources

1. appsettings.json
2. appsettings.{Environment}.json
3. Secret Manager (Development only)
4. Environment variables
5. Command-line arguments

3.4 ScoreProcessor Architecture — Demo

graph LR
    subgraph "TennisBookings Web App"
        Form["Upload form\n/admin/results"] -->|POST CSV| Ctrl["ResultsController"]
        Ctrl -->|"TransferUtilityUploadAsync()"| S3["Amazon S3\ntennisresults bucket"]
        S3 -->|S3 Event Notification| SNS["Amazon SNS"]
        SNS -->|Subscribe| SQS["Amazon SQS Queue"]
    end
    subgraph "TennisBookings.ScoreProcessor Worker"
        QRS["QueueReadingService\nPoll SQS"] -->|SQS message| CH["SqsMessageChannel\nChannel<Message>"]
        CH -->|ReadAllAsync| SPS["ScoreProcessingService\nProcesses from S3"]
        SPS -->|"GetObjectAsync()"| S32["Amazon S3"]
        SPS -->|"DeleteMessageAsync()"| SQS2["Amazon SQS"]
    end
    SQS -->|receive messages| QRS
    style CH fill:#2196F3,color:#fff
    style QRS fill:#4CAF50,color:#fff
    style SPS fill:#FF9800,color:#fff

3.5 SQS Reading Service — Producer

// SqsMessageChannel.cs
public class SqsMessageChannel : ISqsMessageChannel
{
    private readonly Channel<Message> _channel;
    private readonly ILogger<SqsMessageChannel> _logger;

    public SqsMessageChannel(ILogger<SqsMessageChannel> logger)
    {
        _logger = logger;
        _channel = Channel.CreateBounded<Message>(new BoundedChannelOptions(50)
        {
            SingleWriter = true,
            SingleReader = true,
            FullMode = BoundedChannelFullMode.Wait
        });
    }

    public async Task WriteMessagesAsync(IReadOnlyList<Message> messages, CancellationToken ct)
    {
        foreach (var message in messages)
        {
            await _channel.Writer.WriteAsync(message, ct);
            _logger.LogDebug("Message {Id} added to channel.", message.MessageId);
        }
    }

    public IAsyncEnumerable<Message> ReadAllAsync(CancellationToken ct = default)
        => _channel.Reader.ReadAllAsync(ct);

    public bool TryCompleteWriter(Exception? ex = null)
        => _channel.Writer.TryComplete(ex);
}

// QueueReadingService.cs — Producer
public class QueueReadingService : BackgroundService
{
    private readonly ILogger<QueueReadingService> _logger;
    private readonly ISqsMessageQueue _sqsMessageQueue;
    private readonly ISqsMessageChannel _sqsMessageChannel;
    private readonly IHostApplicationLifetime _appLifetime;
    private readonly string _queueUrl;

    public QueueReadingService(
        ILogger<QueueReadingService> logger,
        ISqsMessageQueue sqsMessageQueue,
        IOptions<AwsServicesConfiguration> options,
        ISqsMessageChannel sqsMessageChannel,
        IHostApplicationLifetime appLifetime)
    {
        _logger = logger;
        _sqsMessageQueue = sqsMessageQueue;
        _sqsMessageChannel = sqsMessageChannel;
        _appLifetime = appLifetime;
        _queueUrl = options.Value.ScoresQueueUrl;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            await Task.Yield();

            var request = new ReceiveMessageRequest
            {
                QueueUrl = _queueUrl,
                MaxNumberOfMessages = 10,
                WaitTimeSeconds = 5 // Long polling
            };

            while (!stoppingToken.IsCancellationRequested)
            {
                ReceiveMessageResponse response;
                try
                {
                    response = await _sqsMessageQueue
                        .ReceiveMessageAsync(request, stoppingToken);
                }
                catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
                {
                    break;
                }
                catch (AmazonSQSException ex)
                {
                    _logger.LogError(ex, "AWS SQS error.");
                    throw;
                }

                if (response.HttpStatusCode == HttpStatusCode.OK
                    && response.Messages.Count > 0)
                {
                    _logger.LogInformation("Received {Count} message(s).", response.Messages.Count);
                    await _sqsMessageChannel.WriteMessagesAsync(response.Messages, stoppingToken);
                }
                else
                {
                    await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
                }
            }
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            _logger.LogCritical(ex, "Fatal error in QueueReadingService.");
        }
        finally
        {
            _sqsMessageChannel.TryCompleteWriter();
            _appLifetime.StopApplication();
        }
    }
}

3.6 ScoreProcessingService — Consumer

// ScoreProcessingService.cs
public class ScoreProcessingService : BackgroundService
{
    private readonly ILogger<ScoreProcessingService> _logger;
    private readonly ISqsMessageChannel _sqsMessageChannel;
    private readonly IScoreProcessor _scoreProcessor;
    private readonly IHostApplicationLifetime _appLifetime;

    public ScoreProcessingService(
        ILogger<ScoreProcessingService> logger,
        ISqsMessageChannel sqsMessageChannel,
        IScoreProcessor scoreProcessor,
        IHostApplicationLifetime appLifetime)
    {
        _logger = logger;
        _sqsMessageChannel = sqsMessageChannel;
        _scoreProcessor = scoreProcessor;
        _appLifetime = appLifetime;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            await Task.Yield();

            await foreach (var message in _sqsMessageChannel.ReadAllAsync(stoppingToken))
            {
                await ProcessMessageAsync(message);
            }
        }
        catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation("ScoreProcessingService stopped normally.");
        }
        catch (Exception ex)
        {
            _logger.LogCritical(ex, "Fatal error in ScoreProcessingService.");
        }
        finally
        {
            _appLifetime.StopApplication();
        }
    }

    private async Task ProcessMessageAsync(Message message)
    {
        _logger.LogInformation("Processing message {Id}.", message.MessageId);
        try
        {
            var result = await _scoreProcessor.ProcessAsync(message.Body);
            if (result.IsSuccess)
                _logger.LogInformation(
                    "Message {Id} processed. {Count} scores recorded.",
                    message.MessageId, result.ScoreCount);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing message {Id}.", message.MessageId);
        }
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        var sw = Stopwatch.StartNew();
        await base.StopAsync(cancellationToken);
        sw.Stop();
        _logger.LogInformation("Stopped in {Ms}ms.", sw.ElapsedMilliseconds);
    }
}

Complete Worker Service Program.cs

var builder = Host.CreateApplicationBuilder(args);

builder.Services.Configure<AwsServicesConfiguration>(
    builder.Configuration.GetSection("AWS"));

builder.Services.AddAWSService<IAmazonSQS>();
builder.Services.AddAWSService<IAmazonS3>();

builder.Services.AddSingleton<ISqsMessageChannel, SqsMessageChannel>();
builder.Services.AddSingleton<ISqsMessageQueue, SqsMessageQueue>();
builder.Services.AddSingleton<IS3EventNotificationMessageParser, S3EventNotificationMessageParser>();
builder.Services.AddSingleton<IScoreProcessor, AwsScoreProcessor>();

builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromSeconds(60);
});

// ORDER MATTERS: producer before consumer
// At shutdown: ScoreProcessingService stops first, then QueueReadingService
builder.Services.AddHostedService<QueueReadingService>();
builder.Services.AddHostedService<ScoreProcessingService>();

var host = builder.Build();
await host.RunAsync();

3.7 Web Application Refactoring

After creating the Worker Service, the controller stores files directly in S3:

[Area("Admin")]
public class ResultsController : Controller
{
    private readonly IAmazonS3 _s3Client;
    private readonly string _bucketName;

    public ResultsController(IAmazonS3 s3Client, IOptions<AwsServicesConfiguration> options)
    {
        _s3Client = s3Client;
        _bucketName = options.Value.FileUploadBucketName;
    }

    [HttpPost("admin/results/upload-v3")]
    public async Task<IActionResult> UploadFileV3(IFormFile file, CancellationToken ct)
    {
        if (file is null || file.Length == 0) return BadRequest();

        var objectKey = $"{Guid.NewGuid()}.csv";

        try
        {
            var transferUtility = new TransferUtility(_s3Client);
            await transferUtility.UploadAsync(
                new TransferUtilityUploadRequest
                {
                    BucketName = _bucketName,
                    Key = objectKey,
                    InputStream = file.OpenReadStream(),
                    ContentType = "text/csv"
                }, ct);
            // S3 upload automatically triggers SNS -> SQS -> Worker Service
        }
        catch (AmazonS3Exception)
        {
            return RedirectToAction("UploadFailed");
        }

        return RedirectToAction("UploadSuccess");
    }
}

4. Advanced Hosted Service Concepts

4.1 BackgroundService Implementation Details

sequenceDiagram
    participant Host as Generic Host
    participant BS as BackgroundService
    participant EA as ExecuteAsync (your method)

    Host->>BS: StartAsync(cancellationToken)
    BS->>BS: Creates _stoppingCts linked to cancellationToken
    BS->>EA: Launches ExecuteAsync(_stoppingCts.Token)
    Note over BS,EA: ExecuteAsync is NOT awaited!
    BS-->>Host: Task.CompletedTask (immediate return)
    Note over Host: Host starts the next service...

    Note over EA: ExecuteAsync runs in the background...

    Host->>BS: StopAsync(shutdownToken)
    BS->>BS: _stoppingCts.Cancel()
    BS->>EA: stoppingToken is cancelled
    BS->>BS: await Task.WhenAny(_executeTask, infinite_delay)
    Note over BS: Waits for ExecuteAsync to end OR shutdown timeout
    EA-->>BS: Task completed
    BS-->>Host: StopAsync completes

4.2 Exception Handling in BackgroundService

Default Behavior

Complete Exception Handling Pattern

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    try
    {
        await Task.Yield();

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessNextBatchAsync(stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                _logger.LogInformation("Shutdown requested.");
                break;
            }
            catch (HttpRequestException ex) when (IsTransient(ex))
            {
                _logger.LogWarning(ex, "Transient error. Retrying in 30s.");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogCritical(ex, "Fatal unrecoverable error.");
                throw;
            }
        }
    }
    finally
    {
        _appLifetime.StopApplication();
    }
}

private static bool IsTransient(HttpRequestException ex)
    => ex.StatusCode is HttpStatusCode.TooManyRequests
        or HttpStatusCode.ServiceUnavailable
        or HttpStatusCode.GatewayTimeout;

4.3 IHostApplicationLifetime — Application Lifecycle

public interface IHostApplicationLifetime
{
    // Triggered when the application is fully started
    CancellationToken ApplicationStarted { get; }
    // Triggered when graceful shutdown begins
    CancellationToken ApplicationStopping { get; }
    // Triggered when graceful shutdown is complete
    CancellationToken ApplicationStopped { get; }
    // Trigger application shutdown programmatically
    void StopApplication();
}

Usage in a BackgroundService

public class MyService : BackgroundService
{
    private readonly IHostApplicationLifetime _lifetime;

    public MyService(IHostApplicationLifetime lifetime, ILogger<MyService> logger)
    {
        _lifetime = lifetime;
        // Subscribe to lifecycle events
        _lifetime.ApplicationStarted.Register(
            () => logger.LogInformation("Application started. Service ready."));
        _lifetime.ApplicationStopping.Register(
            () => logger.LogInformation("Graceful shutdown in progress..."));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            await DoWorkAsync(stoppingToken);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            // Fatal error → clean app shutdown
            _lifetime.StopApplication();
            throw;
        }
    }
}

The Three Lifecycle Tokens

timeline
    title Application Lifecycle
    ApplicationStarted : Host fully started
    : All StartAsync calls completed
    ApplicationStopping : Ctrl+C / SIGTERM / StopApplication() received
    : IHostedService.StopAsync called on each service
    ApplicationStopped : Graceful shutdown complete
    : Application about to exit

4.4 Service Registration Order

graph LR
    subgraph "Registration"
        R1["AddHostedService<ServiceA>()"]
        R2["AddHostedService<ServiceB>()"]
        R3["AddHostedService<ServiceC>()"]
        R1 --> R2 --> R3
    end
    subgraph "Startup (normal order)"
        S1["StartAsync ServiceA"] --> S2["StartAsync ServiceB"] --> S3["StartAsync ServiceC"]
    end
    subgraph "Shutdown (REVERSE order)"
        T3["StopAsync ServiceC"] --> T2["StopAsync ServiceB"] --> T1["StopAsync ServiceA"]
    end

Rule for ScoreProcessor: QueueReadingService before ScoreProcessingService. At shutdown, ScoreProcessingService stops first (finishes the current message), then QueueReadingService closes the channel.

// Correct order: producer before consumer
builder.Services.AddHostedService<QueueReadingService>();    // Registered 1st
builder.Services.AddHostedService<ScoreProcessingService>(); // Registered 2nd
// At shutdown: ScoreProcessingService stops first (reverse order)

4.5 Graceful Shutdown Timeout

builder.Services.Configure<HostOptions>(options =>
{
    // Max delay for all StopAsync calls to complete (default: 30s)
    options.ShutdownTimeout = TimeSpan.FromSeconds(60);

    // .NET 8: stop services in parallel (faster but watch for race conditions)
    options.ServicesStopConcurrently = true;
});

If ShutdownTimeout expires before all StopAsync calls complete, the host forces shutdown (data potentially lost).

4.6 Overriding StartAsync and StopAsync

public class ScoreProcessingService : BackgroundService
{
    private readonly ILogger<ScoreProcessingService> _logger;

    // Override StartAsync — initialize before ExecuteAsync starts
    public override async Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Initialization in progress...");
        await InitializeAsync(cancellationToken);
        // ALWAYS call base.StartAsync to launch ExecuteAsync!
        await base.StartAsync(cancellationToken);
        _logger.LogInformation("Service initialized.");
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await DoWorkAsync(stoppingToken);
    }

    // Override StopAsync — measure shutdown time
    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        var sw = Stopwatch.StartNew();
        _logger.LogInformation("Shutdown in progress...");
        await base.StopAsync(cancellationToken);
        sw.Stop();
        _logger.LogInformation("Stopped in {Elapsed}ms.", sw.ElapsedMilliseconds);
    }

    private Task InitializeAsync(CancellationToken ct) => Task.CompletedTask;
    private Task DoWorkAsync(CancellationToken ct) => Task.CompletedTask;
}

4.7 Avoiding Blocking Code in StartAsync

// ❌ PROBLEM: Synchronous code at the start of ExecuteAsync -> blocks startup
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    Thread.Sleep(5000);    // BLOCKS startup of other services for 5s!
    var data = LoadHeavyDataSync();

    await foreach (var item in _channel.ReadAllAsync(stoppingToken)) { }
}

// ✅ SOLUTION: await Task.Yield() at the very beginning
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    // Task.Yield() immediately returns an incomplete Task
    // -> Yields control to the calling thread (StartAsync returns immediately)
    // -> ExecuteAsync resumes on a ThreadPool thread
    await Task.Yield();

    Thread.Sleep(5000);    // No longer impacts startup
    var data = LoadHeavyDataSync();

    await foreach (var item in _channel.ReadAllAsync(stoppingToken)) { }
}

Rule: await Task.Yield() should be the first line of ExecuteAsync if your code has synchronous work before the first await.

4.8 Unit Testing Worker Services

// QueueReadingServiceTests.cs (xUnit + Moq + FluentAssertions)
public class QueueReadingServiceTests
{
    [Fact]
    public async Task ExecuteAsync_WhenSqsThrowsException_ShouldCallStopApplication()
    {
        // Arrange
        var mockSqsQueue = new Mock<ISqsMessageQueue>();
        mockSqsQueue
            .Setup(q => q.ReceiveMessageAsync(
                It.IsAny<ReceiveMessageRequest>(),
                It.IsAny<CancellationToken>()))
            .ThrowsAsync(new AmazonSQSException("Invalid credentials"));

        var mockChannel = new Mock<ISqsMessageChannel>();
        var mockLifetime = new Mock<IHostApplicationLifetime>();

        var service = new QueueReadingService(
            NullLogger<QueueReadingService>.Instance,
            mockSqsQueue.Object,
            Options.Create(new AwsServicesConfiguration
            {
                ScoresQueueUrl = "https://sqs.fake/queue"
            }),
            mockChannel.Object,
            mockLifetime.Object);

        // Act
        var exception = await Assert.ThrowsAsync<AmazonSQSException>(
            () => service.StartAsync(CancellationToken.None));

        // Assert
        exception.Message.Should().Contain("Invalid credentials");
        mockChannel.Verify(c => c.TryCompleteWriter(It.IsAny<Exception>()), Times.Once);
        mockLifetime.Verify(l => l.StopApplication(), Times.Once);
    }

    [Fact]
    public async Task ExecuteAsync_WhenCancelled_ShouldStopGracefully()
    {
        // Arrange
        var mockSqsQueue = new Mock<ISqsMessageQueue>();
        mockSqsQueue
            .Setup(q => q.ReceiveMessageAsync(It.IsAny<ReceiveMessageRequest>(),
                It.IsAny<CancellationToken>()))
            .ReturnsAsync(new ReceiveMessageResponse
            {
                HttpStatusCode = HttpStatusCode.OK,
                Messages = new List<Message>()
            });

        var mockChannel = new Mock<ISqsMessageChannel>();
        var mockLifetime = new Mock<IHostApplicationLifetime>();

        var service = new QueueReadingService(
            NullLogger<QueueReadingService>.Instance,
            mockSqsQueue.Object,
            Options.Create(new AwsServicesConfiguration
            {
                ScoresQueueUrl = "https://sqs.fake/queue"
            }),
            mockChannel.Object,
            mockLifetime.Object);

        // Act
        await service.StartAsync(CancellationToken.None);
        await Task.Delay(100);
        await service.StopAsync(CancellationToken.None);

        // Assert
        mockLifetime.Verify(l => l.StopApplication(), Times.AtLeastOnce);
    }
}

4.9 Polly — Retry Policies in Background Services

Polly is a resilience library for .NET.

dotnet add package Microsoft.Extensions.Http.Polly
dotnet add package Polly

Retry with Exponential Backoff on HttpClient

builder.Services
    .AddHttpClient<IWeatherApiClient, WeatherApiClient>()
    .AddTransientHttpErrorPolicy(policy =>
        policy.WaitAndRetryAsync(
            retryCount: 3,
            sleepDurationProvider: attempt =>
                TimeSpan.FromSeconds(Math.Pow(2, attempt)), // 2s, 4s, 8s
            onRetry: (exception, delay, attempt, _) =>
                Console.WriteLine($"Attempt {attempt} after {delay.TotalSeconds:F1}s.")))
    .AddTransientHttpErrorPolicy(policy =>
        policy.CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 5,
            durationOfBreak: TimeSpan.FromSeconds(30)));

Retry in the BackgroundService Itself

public class ResilientProcessingService : BackgroundService
{
    private readonly IAsyncPolicy _retryPolicy;
    private readonly ILogger<ResilientProcessingService> _logger;

    public ResilientProcessingService(ILogger<ResilientProcessingService> logger)
    {
        _logger = logger;
        _retryPolicy = Policy
            .Handle<HttpRequestException>()
            .Or<TimeoutException>()
            .WaitAndRetryAsync(
                retryCount: 5,
                sleepDurationProvider: attempt =>
                    TimeSpan.FromSeconds(Math.Min(30, Math.Pow(2, attempt))),
                onRetry: (ex, delay, attempt, _) =>
                    _logger.LogWarning(ex,
                        "Attempt {Attempt}. Next retry in {Delay:F1}s.",
                        attempt, delay.TotalSeconds));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(1));

        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            await _retryPolicy.ExecuteAsync(async () =>
            {
                await FetchAndProcessDataAsync(stoppingToken);
            });
        }
    }

    private Task FetchAndProcessDataAsync(CancellationToken ct) => Task.CompletedTask;
}

4.10 Health Checks for Worker Services

dotnet add package Microsoft.Extensions.Diagnostics.HealthChecks

// HealthChecks/FileProcessingChannelHealthCheck.cs
public class FileProcessingChannelHealthCheck : IHealthCheck
{
    private readonly FileProcessingChannel _channel;

    public FileProcessingChannelHealthCheck(FileProcessingChannel channel)
        => _channel = channel;

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken ct = default)
    {
        var count = _channel.FileCount;

        var result = count switch
        {
            < 50 => HealthCheckResult.Healthy($"Channel OK. {count} pending."),
            < 80 => HealthCheckResult.Degraded($"Channel loaded. {count} pending."),
            _    => HealthCheckResult.Unhealthy($"Channel saturated! {count} pending.")
        };

        return Task.FromResult(result);
    }
}

// HealthChecks/WorkerServiceHealthCheck.cs
public class WorkerServiceHealthCheck<T> : IHealthCheck
    where T : BackgroundService
{
    private readonly T _backgroundService;

    public WorkerServiceHealthCheck(T backgroundService)
        => _backgroundService = backgroundService;

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken ct = default)
    {
        var task = _backgroundService.ExecuteTask;

        var result = task switch
        {
            null => HealthCheckResult.Unhealthy("Service not started."),
            { IsCompletedSuccessfully: true } =>
                HealthCheckResult.Unhealthy("Service completed unexpectedly."),
            { IsFaulted: true, Exception: var ex } =>
                HealthCheckResult.Unhealthy($"Service faulted: {ex?.Message}"),
            { IsCanceled: true } =>
                HealthCheckResult.Unhealthy("Service cancelled."),
            _ => HealthCheckResult.Healthy("Service is running.")
        };

        return Task.FromResult(result);
    }
}

// Program.cs
builder.Services.AddHealthChecks()
    .AddCheck<FileProcessingChannelHealthCheck>(
        name: "file-processing-channel",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "channel", "background" })
    .AddCheck<WorkerServiceHealthCheck<FileProcessingService>>(
        name: "file-processing-service",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "background" });

app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});
app.MapHealthChecks("/health/live");

4.11 Structured Logging in Hosted Services

public class FileProcessingService : BackgroundService
{
    // LoggerMessage.Define for better performance (avoids allocations)
    private static readonly Action<ILogger, string, Exception?> _logProcessingStarted =
        LoggerMessage.Define<string>(
            LogLevel.Information,
            new EventId(1, "ProcessingStarted"),
            "Processing of file {FileName} started.");

    private static readonly Action<ILogger, string, long, Exception?> _logProcessingCompleted =
        LoggerMessage.Define<string, long>(
            LogLevel.Information,
            new EventId(2, "ProcessingCompleted"),
            "File {FileName} processed in {ElapsedMs}ms.");

    private readonly ILogger<FileProcessingService> _logger;
    private readonly FileProcessingChannel _channel;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var fileName in _channel.ReadAllAsync(stoppingToken))
        {
            var sw = Stopwatch.StartNew();
            _logProcessingStarted(_logger, fileName, null);

            // Logging scope: all log lines within this block
            // will have the FileName and CorrelationId properties
            using var _ = _logger.BeginScope(new Dictionary<string, object>
            {
                ["FileName"] = fileName,
                ["CorrelationId"] = Guid.NewGuid().ToString("N")
            });

            try
            {
                await ProcessFileAsync(fileName, stoppingToken);
                sw.Stop();
                _logProcessingCompleted(_logger, fileName, sw.ElapsedMilliseconds, null);
            }
            catch (Exception ex)
            {
                sw.Stop();
                _logger.LogError(ex,
                    "Failed to process {FileName} after {ElapsedMs}ms.",
                    fileName, sw.ElapsedMilliseconds);
            }
        }
    }

    private Task ProcessFileAsync(string f, CancellationToken ct) => Task.CompletedTask;
}

5. Running in Production

5.1 Docker — Recommended Approach for Microservices

Optimized Multi-Stage Dockerfile

# Dockerfile — TennisBookings.ScoreProcessor

# === Stage 1: Build =========================================================
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src

# Copy .csproj files first (Docker cache optimization)
COPY ["src/TennisBookings.ScoreProcessor/TennisBookings.ScoreProcessor.csproj",
      "TennisBookings.ScoreProcessor/"]
COPY ["src/TennisBookings.ResultsProcessing/TennisBookings.ResultsProcessing.csproj",
      "TennisBookings.ResultsProcessing/"]

RUN dotnet restore "TennisBookings.ScoreProcessor/TennisBookings.ScoreProcessor.csproj"

COPY src/ .

RUN dotnet publish "TennisBookings.ScoreProcessor/TennisBookings.ScoreProcessor.csproj" \
    --configuration Release \
    --output /app/publish \
    --no-restore

# === Stage 2: Runtime =======================================================
# Lightweight runtime image (without SDK)
FROM mcr.microsoft.com/dotnet/runtime:8.0 AS final

# Non-root user (security)
RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser
USER appuser

WORKDIR /app
COPY --from=build /app/publish .

ENV DOTNET_ENVIRONMENT=Production
ENTRYPOINT ["dotnet", "TennisBookings.ScoreProcessor.dll"]

.dockerignore

**/.vs
**/.git
**/bin
**/obj
src/**
!src/TennisBookings.ScoreProcessor/**
!src/TennisBookings.ResultsProcessing/**

Docker Commands

# Build
docker build -f src/TennisBookings.ScoreProcessor/Dockerfile \
    -t tennis-score-processor:latest .

# Run with environment variables
docker run \
    -e DOTNET_ENVIRONMENT=Development \
    -e AWS__ScoresQueueUrl="https://sqs.eu-west-1.amazonaws.com/123456/queue" \
    tennis-score-processor:latest

docker-compose.yml (LocalStack for dev)

version: '3.8'
services:
  score-processor:
    build:
      context: .
      dockerfile: src/TennisBookings.ScoreProcessor/Dockerfile
    environment:
      - DOTNET_ENVIRONMENT=Development
      - AWS__ServiceURL=http://localstack:4566
      - AWS__ScoresQueueUrl=http://localstack:4566/000000000000/tennis-scores
    depends_on:
      - localstack

  localstack:
    image: localstack/localstack:latest
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3,sqs,sns

5.2 Windows Service

<PackageReference Include="Microsoft.Extensions.Hosting.WindowsServices" Version="8.0.0" />

// Program.cs
var builder = Host.CreateApplicationBuilder(args);

// UseWindowsService():
// 1. Configures WindowsServiceLifetime (responds to SCM commands)
// 2. Sets ContentRootPath = AppContext.BaseDirectory
// 3. Enables logging to Windows Event Log
builder.Host.UseWindowsService(options =>
{
    options.ServiceName = "TennisBookings Score Processor";
});

builder.Services.AddHostedService<QueueReadingService>();
builder.Services.AddHostedService<ScoreProcessingService>();

var host = builder.Build();
await host.RunAsync();

# Publish as single-file executable
dotnet publish src/TennisBookings.ScoreProcessor `
    --configuration Release `
    --runtime win-x64 `
    --self-contained true `
    -p:PublishSingleFile=true `
    -p:PublishReadyToRun=true `
    --output C:\Services\ScoreProcessor

# Register as Windows Service (admin rights required)
sc.exe create "TennisScoreProcessor" `
    binPath="C:\Services\ScoreProcessor\TennisBookings.ScoreProcessor.exe" `
    start=auto `
    DisplayName="TennisBookings Score Processor"

sc.exe start "TennisScoreProcessor"
sc.exe query "TennisScoreProcessor"

# Event Viewer logs
Get-EventLog -LogName Application -Source "TennisBookings Score Processor" -Newest 20

# Removal
sc.exe stop "TennisScoreProcessor"
sc.exe delete "TennisScoreProcessor"

5.3 Linux Daemon (systemd)

<PackageReference Include="Microsoft.Extensions.Hosting.Systemd" Version="8.0.0" />

// Program.cs
var builder = Host.CreateApplicationBuilder(args);

// UseSystemd():
// 1. Configures SystemdLifetime (sd_notify protocol)
// 2. Notifies systemd that the service is ready
// 3. Integrates logging with journald
builder.Host.UseSystemd();

builder.Services.AddHostedService<QueueReadingService>();
builder.Services.AddHostedService<ScoreProcessingService>();

var host = builder.Build();
await host.RunAsync();

# /etc/systemd/system/tennisscore.service
[Unit]
Description=TennisBookings Score Processor Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=tennisscore
Group=tennisscore
WorkingDirectory=/srv/tennisscore
ExecStart=/srv/tennisscore/TennisBookings.ScoreProcessor
Restart=on-failure
RestartSec=5
Environment=DOTNET_ENVIRONMENT=Production
EnvironmentFile=-/etc/tennisscore/environment

[Install]
WantedBy=multi-user.target

# Publish Linux x64
dotnet publish src/TennisBookings.ScoreProcessor \
    --configuration Release \
    --runtime linux-x64 \
    --self-contained true \
    -p:PublishSingleFile=true \
    --output ./publish/linux-daemon

# Deploy
sudo cp tennisscore.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable tennisscore
sudo systemctl start tennisscore

# Management
sudo systemctl status tennisscore
sudo journalctl -fu tennisscore        # Live logs
sudo journalctl -u tennisscore --since "1 hour ago"

5.4 Azure App Service — WebJobs

// No specific code changes for Azure WebJobs
var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddHostedService<QueueReadingService>();
builder.Services.AddHostedService<ScoreProcessingService>();
var host = builder.Build();
await host.RunAsync();

Deploy from Visual Studio:
1. Right-click -> Publish
2. Target: Azure -> Azure WebJobs
3. Create/select an App Service Plan
4. Deployment mode: Self-contained
5. Target runtime: win-x64 (or linux-x64)
6. WebJob type: Continuous (always active)
7. App Service -> Configuration -> Always On: ON (required!)
8. Logs: App Service -> Log Stream

Important: Enable Always On in App Service for continuous WebJobs.

5.5 Azure Container Apps

# Prerequisites
az extension add --name containerapp

# Create the environment
az containerapp env create \
    --name tennisbookings-env \
    --resource-group TennisBookings \
    --location eastus

# Build and push to Azure Container Registry
az acr build \
    --registry tennisbookingsacr \
    --image score-processor:latest \
    --file src/TennisBookings.ScoreProcessor/Dockerfile .

# Deploy
az containerapp create \
    --name score-processor \
    --resource-group TennisBookings \
    --environment tennisbookings-env \
    --image tennisbookingsacr.azurecr.io/score-processor:latest \
    --min-replicas 0 \
    --max-replicas 10 \
    --scale-rule-name "queue-length" \
    --scale-rule-type "azure-servicebus" \
    --scale-rule-metadata "queueName=tennis-scores" "messageCount=5"

# KEDA scaling — Scale to zero when no messages
scale:
  minReplicas: 0
  maxReplicas: 20
  rules:
  - name: sqs-scale
    custom:
      type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/123/tennis-scores
        targetQueueLength: "5"

5.6 Deployment Options Comparison

graph TD
    D["Worker Service to deploy"]
    D --> DC["Docker Container\n(recommended)"]
    D --> WS["Windows Service"]
    D --> LS["Linux systemd"]
    D --> AZ["Azure WebJobs"]
    D --> ACA["Azure Container Apps"]
    DC --> DC1["Cross-platform, scalable, portable"]
    WS --> WS1["Simple on Windows, integrated Event Log"]
    LS --> LS1["Linux standard, journald logging"]
    AZ --> AZ1["Simple deployment, limited scaling"]
    ACA --> ACA1["Scale to zero, KEDA, managed infra"]

6. Alternatives: Quartz.NET and Hangfire

Quartz.NET

dotnet add package Quartz
dotnet add package Quartz.Extensions.Hosting

// Quartz.NET Job
public class DailyReportJob : IJob
{
    private readonly IReportService _reportService;

    public DailyReportJob(IReportService reportService)
        => _reportService = reportService;

    public async Task Execute(IJobExecutionContext context)
    {
        await _reportService.GenerateDailyReportAsync(context.CancellationToken);
    }
}

// Program.cs
builder.Services.AddQuartz(q =>
{
    q.UseMicrosoftDependencyInjectionJobFactory();

    var jobKey = new JobKey("DailyReport");
    q.AddJob<DailyReportJob>(opts => opts.WithIdentity(jobKey));

    q.AddTrigger(opts => opts
        .ForJob(jobKey)
        .WithIdentity("DailyReport-trigger")
        .WithCronSchedule("0 0 2 * * ?") // Every day at 2:00 AM
    );
});

builder.Services.AddQuartzHostedService(q => { q.WaitForJobsToComplete = true; });

Hangfire

dotnet add package Hangfire.AspNetCore
dotnet add package Hangfire.InMemory

builder.Services.AddHangfire(config => config.UseInMemoryStorage());
builder.Services.AddHangfireServer();

// Schedule jobs
RecurringJob.AddOrUpdate<IEmailService>(
    "send-weekly-summary",
    service => service.SendWeeklySummaryAsync(),
    Cron.Weekly);

BackgroundJob.Enqueue<IReportService>(
    service => service.GenerateReportAsync("2024-Q1"));

app.UseHangfireDashboard("/hangfire");

When to Use What?

7. Summary and Best Practices

Overall Architecture

graph TD
    subgraph "ASP.NET Core Web App"
        HTTP["HTTP Requests\n(Razor Pages, MVC)"]
        WCS["WeatherCacheService\n(PeriodicTimer)"]
        FPC["FileProcessingChannel\n(Channel<string>)"]
        FPS["FileProcessingService\n(consumes the channel)"]
        HTTP -->|upload CSV| FPC
        FPC -->|ReadAllAsync| FPS
    end
    subgraph "Worker Service"
        QRS["QueueReadingService\n(SQS polling)"]
        SMC["SqsMessageChannel\n(Channel<Message>)"]
        SPS["ScoreProcessingService\n(S3 processing)"]
        QRS -->|WriteMessagesAsync| SMC
        SMC -->|ReadAllAsync| SPS
    end
    subgraph "Infrastructure"
        CACHE["Distributed Cache"]
        S3["Amazon S3"]
        SQS["Amazon SQS"]
    end
    WCS --> CACHE
    HTTP --> CACHE
    SPS --> S3
    QRS --> SQS

Hosted Service vs Worker Service Decision

graph TD
    Q{Is the background task\ndirectly related to\nthe web application?}
    Q -->|Yes| HS["Hosted Service\nin ASP.NET Core"]
    Q -->|No| WS["Worker Service\nseparate project"]
    HS --> HS1["Ex: WeatherCacheService\nWeather cache refresh"]
    HS --> HS2["Ex: FileProcessingService\nProcessing web uploads"]
    WS --> WS1["Ex: ScoreProcessor\nSQS messages, independent of web"]
    WS --> WS2["Ex: DataEnrichmentService\nAutonomous ETL pipeline"]

Architectural Decisions

Anti-patterns to Avoid

// ❌ 1. Infinite loop without CancellationToken
while (true)  // Never do this!
    await DoWork();

// ✅ Correct
while (!stoppingToken.IsCancellationRequested)
    await DoWork(stoppingToken);

// ❌ 2. Scoped service injected directly (InvalidOperationException)
public BadService(AppDbContext dbContext) { }

// ✅ Correct
public GoodService(IServiceScopeFactory scopeFactory) { }

// ❌ 3. Unbounded channel → memory leak
var ch = Channel.CreateUnbounded<string>();

// ✅ Correct
var ch = Channel.CreateBounded<string>(new BoundedChannelOptions(100)
    { FullMode = BoundedChannelFullMode.Wait });

// ❌ 4. Unhandled exception (undefined behavior < .NET 6)
protected override async Task ExecuteAsync(CancellationToken ct)
{
    await foreach (var msg in _channel.ReadAllAsync(ct))
        await ProcessAsync(msg); // Exception -> service silently dies!
}

// ✅ Correct
protected override async Task ExecuteAsync(CancellationToken ct)
{
    try
    {
        await foreach (var msg in _channel.ReadAllAsync(ct))
        {
            try { await ProcessAsync(msg, ct); }
            catch (Exception ex) { _logger.LogError(ex, "Error."); }
        }
    }
    finally { _lifetime.StopApplication(); }
}

// ❌ 5. Task.Delay with temporal drift
while (!ct.IsCancellationRequested)
{
    await DoWork();        // 3s work
    await Task.Delay(10000, ct); // + 10s -> real period = 13s!
}

// ✅ Correct
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(10));
while (await timer.WaitForNextTickAsync(ct))
    await DoWork(); // Period always 10s

// ❌ 6. Not calling base.StartAsync in an override
public override Task StartAsync(CancellationToken ct)
{
    return Task.CompletedTask; // ExecuteAsync will NEVER be called!
}

// ✅ Correct
public override async Task StartAsync(CancellationToken ct)
{
    await InitAsync(ct);
    await base.StartAsync(ct); // Launches ExecuteAsync
}

Production Checklist

Before going to production, verify:
[ ] ExecuteAsync starts with await Task.Yield() if synchronous code follows
[ ] All async methods receive the CancellationToken
[ ] Exception handling with try/catch in ExecuteAsync
[ ] IHostApplicationLifetime.StopApplication() called in the finally
[ ] Bounded channel with BoundedChannelFullMode.Wait
[ ] Scoped services created via IServiceScopeFactory
[ ] Health checks configured to monitor the service
[ ] Structured logging with named properties
[ ] ShutdownTimeout configured based on maximum processing time
[ ] Unit tests with NullLogger and mocked dependencies
[ ] Multi-stage Dockerfile with runtime image (not SDK)
[ ] Non-root user in Docker container
[ ] Retry policy (Polly) on external HTTP calls

8. Review Questions

Q1: What is the difference between IHostedService and BackgroundService?

Answer: IHostedService is the base interface with StartAsync() and StopAsync(). BackgroundService is an abstract class that implements IHostedService and handles the boilerplate (creating the CancellationTokenSource, managing graceful shutdown). It exposes the abstract ExecuteAsync() method that you must implement. In almost all cases, inherit from BackgroundService rather than implementing IHostedService directly.

Q2: Why doesn’t ExecuteAsync block application startup?

Answer: In BackgroundService.StartAsync(), ExecuteAsync() is launched but not awaited (_executeTask = ExecuteAsync(...) without await). The method returns Task.CompletedTask immediately. However, any synchronous code before the first await in ExecuteAsync still runs synchronously. Solution: start with await Task.Yield().

Q3: What is a bounded channel and why is it preferred over an unbounded channel?

Answer: A bounded channel has a maximum capacity. When it is full, the producer waits (with BoundedChannelFullMode.Wait). This implements backpressure: if the consumer is slower than the producer, the producer automatically slows down. An unbounded channel can grow indefinitely and cause memory leaks if the consumer can’t keep up.

Q4: How do you access a Scoped service (e.g. DbContext) from a BackgroundService?

Answer: BackgroundService instances are Singletons. Injecting a Scoped service directly causes an InvalidOperationException. Use IServiceScopeFactory:

await using var scope = _scopeFactory.CreateAsyncScope();
var dbContext = scope.ServiceProvider.GetRequiredService<AppDbContext>();
// The scope is disposed automatically -> DbContext.Dispose() called

Q5: What is the shutdown order of hosted services?

Answer: At shutdown, hosted services are stopped in reverse order of their registration. This is important for producer/consumer pairs: register the producer before the consumer. At shutdown, the consumer stops first (finishes current processing), then the producer closes the channel.

Q6: What is the difference between PeriodicTimer and Task.Delay?

Answer:

• Task.Delay: the delay accumulates on top of execution time → temporal drift. If work takes 2s and the delay is 10s, the real period is 12s.
• PeriodicTimer (.NET 6+): ticks at fixed intervals regardless of work duration. WaitForNextTickAsync returns false when the timer is disposed/cancelled. Simpler as it integrates CancellationToken handling.

Q7: What happens if an unhandled exception is raised in ExecuteAsync in .NET 8?

Answer: Since .NET 6, an unhandled exception in ExecuteAsync causes application shutdown. Before .NET 6, the application continued but the service silently stopped working (“zombie” behavior). Always wrap ExecuteAsync code in a try/catch and decide if an exception is recoverable (retry) or fatal (_lifetime.StopApplication()).

Q8: How does UseWindowsService() modify the host behavior?

Answer: UseWindowsService() makes three changes:

1. Replaces ConsoleLifetime with WindowsServiceLifetime that responds to SCM commands
2. Sets ContentRootPath = AppContext.BaseDirectory
3. Enables logging to the Windows Event Log

Q9: Why use await Task.Yield() at the start of ExecuteAsync?

Answer: BackgroundService.StartAsync() launches ExecuteAsync() synchronously up to the first await. If your code before the first await is long, it blocks the startup of other services. await Task.Yield() forces an immediate async transition, yielding control to the calling thread, while ExecuteAsync resumes on a ThreadPool thread.

Q10: What are the advantages of System.Threading.Channels over ConcurrentQueue<T>?

Answer:

Channels avoid CPU polling and support native backpressure.

Q11: What is the difference between a Worker Service and an ASP.NET Core app with Hosted Services?

Answer:

Use a separate Worker Service when the task can evolve independently, needs different scaling, or doesn’t share the web app’s responsibilities.

Q12: How do you unit test a BackgroundService?

Answer: Instantiate the service directly by mocking its dependencies with NullLogger, Moq, and FluentAssertions:

var service = new QueueReadingService(
    NullLogger<QueueReadingService>.Instance,
    mockSqsQueue.Object,
    Options.Create(new AwsServicesConfiguration { ScoresQueueUrl = "fake" }),
    mockChannel.Object,
    mockLifetime.Object);

await service.StartAsync(CancellationToken.None);
await Task.Delay(100);
await service.StopAsync(CancellationToken.None);

mockLifetime.Verify(l => l.StopApplication(), Times.AtLeastOnce);

Q13: What is BoundedChannelFullMode and when should each value be used?

Answer:

For TennisBookings, BoundedChannelFullMode.Wait is correct: we don’t want to lose any files.

Search Terms

.net · asp.net · background · services · core · web · apis · c# · development · service · backgroundservice · hosted · worker · application · generated · periodictimer · program.cs · between · difference · exception · executeasync · registration · retry · task.delay