web項目在生產環境要求:

1. 停機時需要確保 running 的請求能被安全處理完畢
2. 停機時確保不接收新的請求
3. 需要有 healthCheck 接口
4. Load balancer 能對接 healthCheck 接口, 確保業務能達到 zero downtime update

實現機制:

1. 微軟官方關于dotnet-docker優雅關閉的文檔 https://github.com/dotnet/dotnet-docker/blob/main/samples/kubernetes/graceful-shutdown/graceful-shutdown.md
2. 默認的 health check

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks();

var app = builder.Build();
app.MapHealthChecks("/healthz");
app.Run();

默認Asp.net Core 在收到 Terminiate 信號后的處理過程

1. Stop accepting new requests
2. 如果有running request, 自動等待 30 秒后關閉running threads; 如果沒有running request, 立即退出. 默認的30秒timeout可以調整.
   當然我們可以自定義timeout的時間, 但我們很難設定合適的 timeout 值, 既要確保running事務被正常處理完畢, 又能及時響應shutdown 請求.

我推薦的方案

1. 在controller 中使用 Interlocked 類動態記錄 running request 數量
2. 實現一個自定義的 HostLifetime, 重寫 ApplicationStopping 事件, 在事件中實時檢查 running request數量, 如果大于零, 則sleep, 直到數量為0, 退出事件.
3. health check 自己實現一個個的 controller API, 擴展性更好.

示例代碼

MyRestHostLifetime 類

   /// <summary>
    ///
    /// https://github.com/dotnet/dotnet-docker/blob/main/samples/kubernetes/graceful-shutdown/graceful-shutdown.md
    /// </summary>
    public class MyRestHostLifetime : IHostLifetime, IDisposable
    {
        private IHostApplicationLifetime _applicationLifetime;
        private TimeSpan _delay;
        private IEnumerable<IDisposable>? _disposables;
        private AppState _appState;
        private ILogger<MyRestHostLifetime> _logger;

        public MyRestHostLifetime(IHostApplicationLifetime applicationLifetime, TimeSpan delay,
            AppState appState, ILogger<MyRestHostLifetime> logger)

        {
            _applicationLifetime = applicationLifetime;
            _delay = delay;
            _appState = appState;
            _logger = logger;
            _applicationLifetime.ApplicationStopping.Register(OnShutdown);
            _applicationLifetime.ApplicationStopped.Register(AfterShutdown);
        }

        private void OnShutdown()
        {
            while (_appState.getRunningRequests() >= 1)
            {
                _logger.LogWarning($"SIGTERM signal received, but there are {_appState.getRunningRequests()} running requests. Sleep one moment to ensure them handled. ");
                System.Threading.Thread.Sleep(1000);
            }
        }

        private void AfterShutdown()
        {
            string message = $"There are {_appState.getRunningRequests()} running requests. Application stopped. ";
            if (_appState.getRunningRequests() > 0)
            {
                _logger.LogWarning(message);
            }
            else
            {
                _logger.LogInformation(message);
            }
        }

        public Task StopAsync(CancellationToken cancellationToken)
        {
            return Task.CompletedTask;
        }

        public Task WaitForStartAsync(CancellationToken cancellationToken)
        {
            _disposables = new IDisposable[]
            {
            PosixSignalRegistration.Create(PosixSignal.SIGINT, HandleSignal),
            PosixSignalRegistration.Create(PosixSignal.SIGQUIT, HandleSignal),
            PosixSignalRegistration.Create(PosixSignal.SIGTERM, HandleSignal)
            };
            return Task.CompletedTask;
        }

        protected void HandleSignal(PosixSignalContext ctx)
        {
            ctx.Cancel = true;
            Task.Delay(_delay).ContinueWith(t => _applicationLifetime.StopApplication());
        }

        public void Dispose()
        {
            foreach (var disposable in _disposables ?? Enumerable.Empty<IDisposable>())
            {
                disposable.Dispose();
            }
        }
    }

AppState 類

記錄 running request的數量

 public class AppState
    {
        private long _runningRequests = 0;
        private long _completedRequests = 0;
        private long _failedRequests = 0;
        private long _succeededRequests = 0;

        public long runningRequests
        { get { return _runningRequests; } }

        public long completedRequests
        { get { return _completedRequests; } }

        public long failedRequests
        { get { return _failedRequests; } }

        public long succeededRequests
        { get { return _succeededRequests; } }

        public DateTime startTime { get; set; }

        public void markNewRequest()
        {
            Interlocked.Increment(ref _runningRequests);
        }

        public void markRequestCompleted(bool isFailed)
        {
            Interlocked.Decrement(ref _runningRequests);
            Interlocked.Increment(ref _completedRequests);
            if (isFailed)
            {
                Interlocked.Increment(ref _failedRequests);
            }
            else
            {
                Interlocked.Increment(ref _succeededRequests);
            }
        }

        public long getCompletedRequests()
        {
            return this._completedRequests;
        }

        public long getRunningRequests()
        {
            return this._runningRequests;
        }

        public long getFailedRequests()
        {
            return this._failedRequests;
        }

        public long getSucceededRequests()
        {
            return this._succeededRequests;
        }
    }

AppState 和 MyRestHostLifetime 注入DI的代碼片段

        //register AppState object
        var appState = new AppState() { startTime = DateTime.Now };
        services.AddSingleton<AppState>(appState);


        //register MyRestHostLifetime to ensure running requrest handled
        var provider = services.BuildServiceProvider();
        //register MyRestHostLifetime to ensure running requrest handled
        var lifetimeLogger = provider.GetRequiredService<ILogger<MyRestHostLifetime>>();
        services.AddSingleton<IHostLifetime>(sp => new MyRestHostLifetime(
            sp.GetRequiredService<IHostApplicationLifetime>(),
            TimeSpan.FromSeconds(0.1),
            appState, lifetimeLogger));