Class ProcessTerminator

Gracefully handles SIGTERM signals sent to a process to terminate itself.

Inheritance

object

ProcessTerminator

Namespace: Neon.Service

Assembly: Neon.Service.dll

Syntax

public sealed class ProcessTerminator

Remarks

This class listens for a termination signal and then gives the process some time to gracefully save state. The termination timeout defaults to 30 seconds but a custom value may be passed to the constructor.

note

The parent process or operating system typically enforces its own maximum timeout, so your process may still be killed before your timeout is reached.

This class provides two ways for the application to reach to a termination signal. Programs using the async/await pattern can monitor the CancellationToken returned by the CancellationToken property.

Applications may also use AddHandler(Action) to add one more more methods that will be called when a termination signal is received. Each handler will be called in parallel on its own thread.

Finally, you map pass one or more IDisposable instances to AddDisposable(IDisposable). Dispose() will be called for each of these in parallel on its own thread. This can be a handy way of hooking AsyncPeriodicTask instances and other structures into a ProcessTerminator.

Applications should call ReadyToExit() when they have gracefully stopped any activities and saved state so that the process will be terminated immediately. Otherwise, the process will be terminated when the parent process' timeout is finally exceeded.

Applications can also call Exit(int) to proactively signal that the process should terminate gracefully.

A ProcessTerminator is created automatically by NeonService for your service, with the constructor being passed optional parameters passed by your service to the NeonService constructor. You can use these to specify the minimum of amount of time the service will wait during graceful termination (i.e. for requests to be drained) as well as the maximum time allowed for graceful termination.

Minimum graceful termination time defaults to 11 seconds and the maximum graceful termination timeout defaults to 30 seconds. These are reasonable defaults for Kubernetes as discussed here:

https://blog.markvincze.com/graceful-termination-in-kubernetes-with-asp-net-core/

You can adjust these as required. For serviced deployed to Kubernetes, you should try to have the gracefulShutdownTimeout parameter match the pod's terminationGracePeriodSeconds (which typically defaults to 30 seconds).

Also, for ASP.NET applications, you should configure the shutdown timeout to GracefulShutdownTimeout seconds or what you specified in the gracefulShutdownTimeout constructor parameter. This will look something like in your NeonService derived implementation:

protected async override Task<int> OnRunAsync()
{
    _ = Host.CreateDefaultBuilder()
            .ConfigureHostOptions(
                options =>
                {
                    options.ShutdownTimeout = ProcessTerminator.DefaultMinShutdownTime; // <--- set the ASP.NET shutdown timeout here
                })
            .ConfigureWebHostDefaults(builder => builder.UseStartup<Startup>())
            .Build()
            .RunOperatorAsync(Array.Empty<;string>());
await StartedAsync();

// Handle termination gracefully.

await Terminator.StopEvent.WaitAsync();
Terminator.ReadyToExit();

return 0;

}

Constructors

ProcessTerminator(ILogger, TimeSpan, TimeSpan)

Constructor.

Declaration

public ProcessTerminator(ILogger logger = null, TimeSpan gracefulShutdownTimeout = default, TimeSpan minShutdownTime = default)

Parameters

Type	Name	Description
ILogger	logger	Optionally specifies a ILogger used for logging.
TimeSpan	gracefulShutdownTimeout	Optionally specifies the termination timeout (defaults to GracefulShutdownTimeout).
TimeSpan	minShutdownTime	Optionally specifies the minimum time to wait before allowing termination to proceed. This defaults to the minimum of gracefulShutdownTimeout and DefaultMinShutdownTime. See the remarks for more details.

Remarks

gracefulShutdownTimeout defaults to 30 seconds and for environments like Kubernetes, this should be set to the same value as the host pod's terminationGracePeriodSeconds when that's different from its default value of 30 seconds.

minShutdownTime can be used to control the minimum period the service will continue to run after receiving a TERM signal. This can be important for ASPNET based services because Kubernetes can take something like 10 seconds to remove the pod from the service load balancer after sending a TERM to the pod. Having a pod terminate before the load balancer is updated means that other pods may see request errors during this time. This blog post goes into this in some detail:

https://blog.markvincze.com/graceful-termination-in-kubernetes-with-asp-net-core/

minShutdownTime defaults to the minimum of gracefulShutdownTimeout and GracefulShutdownTimeout to wait for the service load balancer to update. This applies to both ASP.NET and headless services so you may wish to reduce minShutdownTime so that headless services will terminate quicker. Pass a negative timespan to disable this behavior.

note

The Signal() method ignores minShutdownTime to improve unit test performance.

Fields

DefaultGracefulTimeout

The default the maximum time a process terminator will wait for a service to terminate gracefully. (30 seconds)

Declaration

public static readonly TimeSpan DefaultGracefulTimeout

Field Value

Type	Description
TimeSpan

DefaultMinShutdownTime

The default the minimum time to wait before allowing termination to proceed. This allows pending requests to be drained. (11 seconds)

Declaration

public static readonly TimeSpan DefaultMinShutdownTime

Field Value

Type	Description
TimeSpan

Properties

CancellationToken

Returns the CancellationToken that will be cancelled when a termination signal is received or Exit(int) is called explicitly.

Declaration

public CancellationToken CancellationToken { get; }

Property Value

Type	Description
CancellationToken

CancellationTokenSource

Returns the CancellationTokenSource that can be used to cancel any outstanding operations before terminating a process.

Declaration

public CancellationTokenSource CancellationTokenSource { get; }

Property Value

Type	Description
CancellationTokenSource

DisableProcessExit

Optionally indicates that the terminator should not actually terminate the hosting process. This is typically enabled for testing or debugging.

Declaration

public bool DisableProcessExit { get; set; }

Property Value

Type	Description
bool

GracefulShutdownTimeout

Returns the termination timeout. This is specified in the constructor and is the maximum time the service will be allowed to shutdown gracefully before being terminated.

Declaration

public TimeSpan GracefulShutdownTimeout { get; }

Property Value

Type	Description
TimeSpan

MinShutdownTime

Returns the minimum amount of time the service should wait to quit after receiving a TERM signal. This is specified in the constructor and is used to allow the service to continue processing straggler requests while Kubernetes service or other load balancers are updating their routes.

Declaration

public TimeSpan MinShutdownTime { get; }

Property Value

Type	Description
TimeSpan

StopEvent

Returns the AsyncManualResetEvent that will be raised when the service is being stopped.

Declaration

public AsyncManualResetEvent StopEvent { get; }

Property Value

Type	Description
AsyncManualResetEvent

TerminateNow

Returns true when the application has been signalled to terminate.

Declaration

public bool TerminateNow { get; }

Property Value

Type	Description
bool

Methods

AddDisposable(IDisposable)

Adds a IDisposable instance that will be disposed when the process is being terminated. This can be a handy way to hook AsyncPeriodicTask and other components into a ProcessTerminator.

Declaration

public void AddDisposable(IDisposable disposable)

Parameters

Type	Name	Description
IDisposable	disposable

AddHandler(Action)

Adds a termination handler.

Declaration

public void AddHandler(Action handler)

Parameters

Type	Name	Description
Action	handler	The handler callback.

Exit(int)

Cleanly terminates the current process.

Declaration

public void Exit(int exitCode = 0)

Parameters

Type	Name	Description
int	exitCode	Optional process exit code (defaults to 0).

ReadyToExit()

Indicates that the application has gracefully stopped and is ready to be terminated.

Declaration

public void ReadyToExit()

Signal()

Emulates a signal instructing the service to close. This will typically be used for unit testing services.

Declaration

public void Signal()

Remarks

note

Signal() method ignores MinShutdownTime to improve unit test performance.

Exceptions

Type	Condition
TimeoutException	Thrown if the service did not exit gracefully in time before it would have been killed (e.g. by Kubernetes or Docker).