Class ClusterFixture
Fixture for testing against NEONKUBE clusters. This can execute against an existing cluster or it can manage the lifecycle of a new cluster during test runs.
note
The NEON_CLUSTER_TESTING environment variable must be defined on the current
machine to enable this feature.
Implements
Inherited Members
Namespace: Neon.Kube.Xunit
Assembly: Neon.Kube.Xunit.dll
Syntax
public class ClusterFixture : TestFixture, ITestFixtureRemarks
note
IMPORTANT: The base Neon TestFixture implementation DOES NOT
support parallel test execution. You need to explicitly disable parallel execution in
all test assemblies that rely on these test fixtures by adding a C# file named
AssemblyInfo.cs with:
[assembly: CollectionBehavior(DisableTestParallelization = true, MaxParallelThreads = 1)]and then define your test classes like:
public class MyTests : IClassFixture<ClusterFixture>
{
private const string clusterDefinitionYaml =
@"name: test
datacenter: test
purpose: test
isLocked: false # $lt;-- test clusters need to be unlocked
timeSources:
- pool.ntp.org
kubernetes:
allowPodsOnControlPlane: true
hosting:
environment: hyperv
hyperv:
useInternalSwitch: true
hypervisor:
namePrefix: "test"
vcpus: 4
memory: 8 GiB
osDisk: 64 GiB
network:
premiseSubnet: 100.64.0.0/24
gateway: 100.64.0.1
nodes:
master:
role: control-plane
address: 100.64.0.2
";
private ClusterFixture foxture;
public MyTests(ClusterFixture fixture)
{
this.fixture = foxture;
var status = fixture.StartAsync(clusterDefinitionYaml);
switch (status)
{
case TestFixtureStatus.Disabled:
return;
case TestFixtureStatus.Started:
// The fixture ensures that the cluster is reset when
// [Start()] is called the first time for a
// fixture instance.
break;
case TestFixtureStatus.AlreadyRunning:
// Reset the cluster between test method calls.
fixture.ResetCluster();
break;
}
}
[Collection(TestCollection.NonParallel)]
[CollectionDefinition(TestCollection.NonParallel, DisableParallelization = true)]
[ClusterFact]
public void Test()
{
// Implement your test here. Note that [fixture.Cluster] returns a [clusterProxy]
// that can be used to manage the cluster and [fixture.K8s] returns an
// [IKubernetes] client connected to the cluster with root privileges.
}
}
This fixture can be used to run tests against an existing NEONKUBE cluster as well as a new clusters deployed by the fixture. The idea here is that you'll have your unit test class inherit from Xunit.IClassFixture<TFixture>, passing ClusterFixture as the type parameter and then implementing a test class constructor that has a ClusterFixture parameter that will receive an instance of the fixture and use that to initialize the test cluster using StartWithClusterDefinition(ClusterDefinition, ClusterFixtureOptions) or one it its overrides.
StartWithClusterDefinition(ClusterDefinition, ClusterFixtureOptions) handles the deployment of the test cluster when it doesn't already exist as well as the removal of any previous cluster, depending on the parameters passed. You'll be calling this in your test class constructor. This method accepts a cluster definition in various forms and returns Disabled when cluster unit testing is disabled on the current machine, Started the first time one of these methods have been called on the fixture instance or AlreadyRunning when StartedAsync() has already been called on the fixture. Your test class typically use this value to decide whether to reset the cluster and or whether additional cluster configuration is required (e.g. deploying test applications).
Alternatively, you can use the StartWithCurrentCluster(ClusterFixtureOptions) method to run tests against the current cluster.
note
The current cluster must be unlocked and running.
It's up to you to call ResetCluster() within your test class constructor when you wish to reset the cluster state between test method executions. Alternatively, you could design your tests such that each method runs in its own namespace to improve test performance while still providing some isolation across test cases.
MANAGING YOUR TEST CLUSTER
You're tests will need to be able to deploy applications and otherwise to the test cluster and otherwise manage your test cluster. The K8s property returns a IKubernetes client for the cluster and the Cluster property returns a ClusterProxy that provides some higher level functionality. Most developers should probably stick with using K8s.
The fixture also provides the NeonExecuteCaptureAsync(params string[]) method which can be used for executing kubectl, helm, and other commands using the neon-cli. Commands will be executed against the test cluster (as the current config) and a ExecuteResponse will be returned holding the command exit code as well as the output text.
CLUSTER TEST METHOD ATTRIBUTES
Tests that require a NEONKUBE cluster will generally be quite slow and will require additional resources on the machine where the test is executing and potentially external resources including XenServer hosts, cloud accounts, specific network configuration, etc. This means that cluster based unit tests can generally run only on specifically configured enviroments.
We provide the ClusterFactAttribute and ClusterTheoryAttribute attributes
to manage this. These derive from Xunit.FactAttribute and Xunit.TheoryAttribute
respectively and set the base class Skip property when the NEON_CLUSTER_TESTING environment
variable does not exist.
Test methods that require NEONKUBE clusters should be tagged with ClusterFactAttribute or
ClusterTheoryAttribute instead of Xunit.FactAttribute or Xunit.TheoryAttribute.
Then by default, these methods won't be executed unless the user has explicitly enabled this on the test
machine by defining the NEON_CLUSTER_TESTING environment variable.
In addition to tagging test methods like this, you'll need to modify your test class constructors to do
nothing when the fixture's Start() methods return Disabled.
You can also use IsClusterTestingEnabled determine when cluster testing is
disabled.
TESTING SCENARIOS
ClusterFixture is designed to support some common testing scenarios, controlled by ClusterFixtureOptions.
| Fresh cluster |
The fixture will remove any existing cluster and deploy a fresh cluster for the tests. Configure
this by setting RemoveClusterOnStart to true. This is
the slowest option because deploying clusters can take 10-20 minutes.
|
| Reuse cluster |
The fixture will reuse an existing cluster if its reachable, healthy, and the the existing
cluster definition matches the test cluster definition. Configure this by setting
RemoveClusterOnStart to false. This is the default and
fastest option when the the required conditions are met. Otherwise, the existing cluster will
be removed and a new cluster will be deployed.
|
| Remove cluster |
Your test class can indicate that the test cluster will be removed after your test class finishes
running test methods. Configure this by setting RemoveClusterOnDispose
to noteClusters will continue running when the ClusterFixture is never disposed. This happens when the test runner fails or is stopped while debugging etc. |
The default ClusterFixtureOptions settings are configured to reuse clusters for better performance, leaving clusters running after running test cases. This is recommended for most user scenarios when you have enough resources to keep a test cluster running.
CLUSTER CONFLICTS
One thing you'll need to worry about is the possibility that a cluster created by one of the Start() methods may conflict with an existing production or NEONDESKTOP cluster. This fixture helps somewhat by persisting cluster state such as kubconfigs, logins, logs, etc. for each deployed cluster within separate directories named like ~/.neonkube/spaces/$fixture. This effectively isolates clusters deployed by the fixture from the user clusters.
IMPORTANT: You'll need to ensure that your cluster name does not conflict with any existing clusters deployed to the same environment and also that the node IP addresses don't conflict with existing clusters deployed on shared infrastructure such as local machines, Hyper-V or XenServer instances. You don't need to worry about IP address conflicts for cloud environments because nodes run on private networks there.
We recommend that you prefix your cluster name with something identifying the machine deploying the cluster. This could be the machine name, user or a combination of the machine and the current username, like runner0- or jeff-, or runner0-jeff-...
note
NEONKUBE maintainers can also use IProfileClient combined with the neon-assistant
tool to reference per-user and/or per-machine profile settings including things like cluster name prefixes,
reserved node IP addresses, etc. These can be referenced by cluster definitions using special macros like
$<$<$<NAME>>> as described here: PreprocessReader.
The goal here is prevent cluster and/or VM naming conflicts for test clusters deployed in parallel by different runners or developers on their own workstations as well as specifying environment specific settings such as host hypervisors, LAN configuration, and node IP addresses.
LIMITATIONS
ClusterFixture assumes that published cluster node images are invariant for a cluster version. The fixture will not automatically redeploy a cluster when a new node template is published without also incrementing the cluster version. This won't impact normal users but maintainers will need to manually remove test clusters for this situation.
note
In the past, we've been somewhat lazy and have node been incrementing cluster versions as we publish new node images. As of 3-30-2022, we're going to start incrementing versions properly so this should no longer be an issue.
ClusterFixture attempts to detect significant differences between an already deployed cluster and a new cluster definition and redeploy the cluster in this case. Unfortunately, the detection mechanism isn't perfect at this time and sometimes clusters that should be redeployed won't be.
Specifically, node labels won't be considered when detecting changes: https://github.com/nforgeio/neonKUBE/issues/1505
Constructors
ClusterFixture()
Constructor.
Declaration
public ClusterFixture()Properties
Cluster
Returns a ClusterProxy instance that can be used to manage the attached cluster after it has been started.
Declaration
public ClusterProxy Cluster { get; }Property Value
| Type | Description |
|---|---|
| ClusterProxy |
ClusterDefinition
Returns the cluster definition for cluster deployed by this fixture via one of the
Start() methods or null when the fixture was connected to the cluster
via one of the ConnectAsync() methods.
Declaration
public ClusterDefinition ClusterDefinition { get; }Property Value
| Type | Description |
|---|---|
| ClusterDefinition |
K8s
Returns a IKubernetes client instance with root privileges that can be used to manage the test cluster after it has been started.
Declaration
public IKubernetes K8s { get; }Property Value
| Type | Description |
|---|---|
| IKubernetes |
Methods
Dispose(bool)
Releases all associated resources.
Declaration
protected override void Dispose(bool disposing)Parameters
| Type | Name | Description |
|---|---|---|
| bool | disposing | Pass |
Overrides
NeonExecuteCaptureAsync(params string[])
Executes a neon-cli command against the current test cluster.
Declaration
public Task<ExecuteResponse> NeonExecuteCaptureAsync(params string[] args)Parameters
| Type | Name | Description |
|---|---|---|
| string[] | args | The command arguments. |
Returns
| Type | Description |
|---|---|
| Task<ExecuteResponse> | An ExecuteResponse with the exit code and output text. |
Remarks
neon-cli is a wrapper around the kubectl and helm tools.
KUBECTL COMMANDS:
neon-cli implements kubectl commands directly like:
neon get pods
neon apply -f myapp.yamlHELM COMMANDS:
neon-cli implements helm commands like neon helm...:
neon helm install -f values.yaml myapp .
neon helm uninstall myappResetCluster()
Resets the cluster.
Declaration
public void ResetCluster()Start(FileInfo, ClusterFixtureOptions)
Deploys a new cluster as specified by a cluster definition YAML file.
note
This method removes any existing NEONKUBE cluster before deploying a fresh one.
Declaration
public TestFixtureStatus Start(FileInfo clusterDefinitionFile, ClusterFixtureOptions options = null)Parameters
| Type | Name | Description |
|---|---|---|
| FileInfo | clusterDefinitionFile | FileInfo for the cluster definition YAML file. |
| ClusterFixtureOptions | options | Optionally specifies the options that ClusterFixture will use to manage the test cluster. |
Returns
| Type | Description | ||||||
|---|---|---|---|---|---|---|---|
| TestFixtureStatus | The TestFixtureStatus:
|
Remarks
IMPORTANT: Only one ClusterFixture can be run at a time on any one computer. This is due to the fact that cluster state like the kubeconfig, NEONKUBE logins, logs and other files will be written to ~/.neonkube/spaces/$fixture/* so multiple fixture instances will be confused when trying to manage these same files.
This means that not only will running ClusterFixture based tests in parallel within the same instance of Visual Studio fail, but but running these tests in different Visual Studio instances will also fail.
StartCluster(string, ClusterFixtureOptions)
Deploys a new cluster as specified by the cluster definition YAML definition.
note
This method removes any existing NEONKUBE cluster before deploying a fresh one.
Declaration
public TestFixtureStatus StartCluster(string clusterDefinitionYaml, ClusterFixtureOptions options = null)Parameters
| Type | Name | Description |
|---|---|---|
| string | clusterDefinitionYaml | The cluster definition YAML. |
| ClusterFixtureOptions | options | Optionally specifies the options that ClusterFixture will use to manage the test cluster. |
Returns
| Type | Description | ||||||
|---|---|---|---|---|---|---|---|
| TestFixtureStatus | The TestFixtureStatus:
|
Remarks
IMPORTANT: Only one ClusterFixture can be run at a time on any one computer. This is due to the fact that cluster state like the kubeconfig, NEONKUBE logins, logs and other files will be written to ~/.neonkube/spaces/$fixture/* so multiple fixture instances will be confused when trying to manage these same files.
This means that not only will running ClusterFixture based tests in parallel within the same instance of Visual Studio fail, but but running these tests in different Visual Studio instances will also fail.
StartWithClusterDefinition(ClusterDefinition, ClusterFixtureOptions)
Deploys a new test cluster as specified by the cluster definition passed or connects to a cluster previously deployed by this method when the cluster definition of the existing cluster and the definition passed here are the same.
Declaration
public TestFixtureStatus StartWithClusterDefinition(ClusterDefinition clusterDefinition, ClusterFixtureOptions options = null)Parameters
| Type | Name | Description |
|---|---|---|
| ClusterDefinition | clusterDefinition | The cluster definition model. |
| ClusterFixtureOptions | options | Optionally specifies the options that ClusterFixture will use to manage the test cluster. |
Returns
| Type | Description | ||||||
|---|---|---|---|---|---|---|---|
| TestFixtureStatus | The TestFixtureStatus:
|
Remarks
IMPORTANT: Only one ClusterFixture can be run at a time on any one computer. This is due to the fact that cluster state like the kubeconfig, NEONKUBE logins, logs and other files will be written to ~/.neonkube/spaces/$fixture/* so multiple fixture instances will be confused when trying to manage these same files.
This means that not only will running ClusterFixture based tests in parallel within the same instance of Visual Studio fail, but running these tests in different Visual Studio instances will also fail.
Exceptions
| Type | Condition |
|---|---|
| NeonKubeException | Thrown when the test cluster could not be deployed. |
StartWithCurrentCluster(ClusterFixtureOptions)
Initializes the test fixture to run tests against the current cluster. This is useful when developing unit tests against a developer managed cluster.
Declaration
public TestFixtureStatus StartWithCurrentCluster(ClusterFixtureOptions options = null)Parameters
| Type | Name | Description |
|---|---|---|
| ClusterFixtureOptions | options | Optionally specifies the options that ClusterFixture will use to manage the test cluster. |
Returns
| Type | Description |
|---|---|
| TestFixtureStatus | This always returns AlreadyRunning. |
Exceptions
| Type | Condition |
|---|---|
| NeonKubeException | Thrown when there isn't a current cluster or when it's locked. |