Class PreprocessReader

Preprocesses text returned by a TextReader by removing comments, expanding variables, and implementing simple conditionals.

Inheritance

object

MarshalByRefObject

TextReader

PreprocessReader

Namespace: Neon.IO

Assembly: Neon.Common.dll

Syntax

public class PreprocessReader : TextReader

Remarks

note

This class only implements ReadLine(), ReadLineAsync(), ReadToEnd(), and ReadToEndAsync(). The other methods will throw a NotImplementedException.

The processor removes comment lines from the text returned. A comment line starts with zero or more whitespace characters followed by "//".

note

Comment lines are indicated by a "//" prefix by default but this can be modified via ClearCommentMarkers() and AddCommentMarker(string).

The processor implements simple macro definition and conditional statements. These statements are identifying by a line with the pound sign (#) as the first non-whitespace character.

note

The processor statement character defaults to the pound sign (#) but can be changed by setting StatementMarker.

The following processing statements are supported:

#define NAME [=VALUE]	Defines a normal variable and setting an optional value. The empty string will be set by default. These variables can be referenced in processing statements or normal text lines as $<name>. Variable names are case sensitive and may include letter, number, dash, period, and underscore characters. By default, defined variables may be referenced like $<name> and environment variables like $<name>.
#if EXPRESSION	Conditionally includes text up to the next #else or #endif statement. The following expressions are supported: VALUE1 == VALUE2 VALUE1 != VALUE2 defined(NAME) undefined(NAME) The comparisions are performed after any variables are expanded. The values are trimmed on bothe ends and the string comparision is case sensitive.
#else	This can optionally be used within an #if statement to include lines when the condition is false.
#endif	This terminates an #if statement.
#switch VALUE	Provides an easy to conditionally include statements for multiple conditions. The subsequent #case and #default statements up to the next #endswitch statement will be processed.
#case VALUE	This command causes the lines up to the next #case, #default, or #endswitch to be outputed if the value matches that specified for the parent #switch statement.
#default	This command causes the lines up to the next #endswitch to be outputed if the value wasn't matched by any of the previous case statements. note #default must appear after all #case statements.
#endswitch	This command terminates a #switch statement.

Normal variables can be defined within the source text itself using the #define command described above and variables may also be added in code using Set(string, string). These variables can be referenced as $<NAME>. Environment variables can be referenced like $<env:NAME>, secrets like $<secret:NAME, and profile values like: $<profile:NAME.

note

You may encounter situations where the default ways of referencing variables conflicts with the syntax of the underlying source text being processed. In these cases, you can set VariableExpansionRegex to CurlyVariableExpansionRegex or ParenVariableExpansionRegex to change the format.

Variables are always expanded in #if and switch statements and are expanded by default in the other source lines. Variables are expanded using the $<NAME> syntax by default. The syntax can be modified by setting VariableExpansionRegex and variable expansion can be disabled by setting ExpandVariables=false.

note

By default, the reader will throw a KeyNotFoundException if an undefined normal variable is encountered. This behavior can be modified by setting DefaultVariable to a non-null string. In this case, undefined variable references will always be replaced by the value set. DefaultVariable defaults to null.

note

By default, the reader will throw a KeyNotFoundException if an undefined environment variable is encountered. This behavior can be modified by setting DefaultEnvironmentVariable to a non-null string. In this case, undefined environment variable references will always be replaced by the value set. DefaultEnvironmentVariable defaults to null.

Processing can also be customized via the StripComments, RemoveComments, RemoveBlank, ProcessStatements, Indent, TabStop, and StatementMarker properties.

Secret and profile Values

This class can integrate with a IProfileClient implementation added to ServiceContainer. This provides a way to abstract access to secrets and profile values from an external source. Three item types are supported:

secret

Secret passwords are often protected by a password manager. NEONFORGE has standardized internally on 1Password for example. Passwords are often required to satisfy complexity and other rules. Passwords are named by a string and are often persisted to a named location. 1Password stores to secrets in vaults. You'll need the password name and optionally, its location when referencing a password value. You may also request a specific secret property using an array syntax like: SECRETNAME[PROPERTY] Doing this overrides the default password property.

profile-value

Profile values are string name/value pairs that include non-secret definitions for the user, workstation, or overall environment such as the LAN. These can come in handy when implementing CI/CD where each server/user can be assigned unique profile values that reference specific test endpoints, etc. This is quite powerful. Profile values are simply named by a string. There is currently no concept of a source, location like secrets may have.

Secrets and profile values can be referenced via <TYPE:NAME[:SOURCE]> where TYPE is one of env, secret (value), or profile and NAME identifies the secret or profile value and source optionally specifies the secret source (this is ignored for profile values).

By default, a secret reference is replaced by the password property within the named secret. Use can use the square bracket syntax to select a different secret property.

Examples:

$<secret:my-secret;my-vault>      # secret from specific source
$<secret:my-secret>               # secret password from default source
$<secret:my-secret[username]>     # retrieve [username] from secret
$<secret:my-secret[password]>     # retrieve [password] from secret
$<profile:my-profile>             # profile value

This class will throw ProfileException when it encounters a secret/profile reference and there no injected IProfileClient implementation or if the implementation has trouble communicating with the profile server. This class also also throws a KeyNotFoundException when a named secret or profile value doesn't exist.

Constructors

PreprocessReader(byte[])

Constructs an instance from UTF-8 encoded bytes.

Declaration

public PreprocessReader(byte[] bytes)

Parameters

Type	Name	Description
byte[]	bytes	The input data.

PreprocessReader(byte[], Dictionary<string, string>)

Constructs an instance from UTF-8 encoded bytes, initializing some variables.

Declaration

public PreprocessReader(byte[] bytes, Dictionary<string, string> variables)

Parameters

Type	Name	Description
byte[]	bytes	The input data.
Dictionary<string, string>	variables	The variables.

PreprocessReader(TextReader)

Constructs an over another TextReader.

Declaration

public PreprocessReader(TextReader reader)

Parameters

Type	Name	Description
TextReader	reader	The source TextReader.

PreprocessReader(TextReader, Dictionary<string, string>)

Constructs an instance over another TextReader, initializing some variables.

Declaration

public PreprocessReader(TextReader reader, Dictionary<string, string> variables)

Parameters

Type	Name	Description
TextReader	reader	The source TextReader.
Dictionary<string, string>	variables	The variables.

PreprocessReader(string)

Constructs an instance from a string.

Declaration

public PreprocessReader(string input)

Parameters

Type	Name	Description
string	input	The input string.

PreprocessReader(string, Dictionary<string, string>)

Constructs an instance from a string, initializing some variables.

Declaration

public PreprocessReader(string input, Dictionary<string, string> variables)

Parameters

Type	Name	Description
string	input	The input string.
Dictionary<string, string>	variables	The variables.

Properties

AngleVariableExpansionRegex

A variable expansion Regex that matches normal variables like $<NAME>, environment variables like $<env:NAME>, profile value references like $<profile:NAME> and secret references like $<secret:NAME>, $<secret:NAME:SOURCE>, $<secret:NAME[PROPERTY]> or $<secret:NAME[PROPERTY]:SOURCE>

You can set the VariableExpansionRegex property to this value to change the PreprocessReader behavior.

Declaration

public static Regex AngleVariableExpansionRegex { get; }

Property Value

Type	Description
Regex

CurlyVariableExpansionRegex

A variable expansion Regex that matches normal variables like ${NAME}, environment variables like ${env:NAME}, profile value references like ${profile:NAME} and secret references like ${secret:NAME}, ${secret:NAME:SOURCE}, ${secret:NAME[PROPERTY]} or ${secret:NAME[PROPERTY]:SOURCE}

You can set the VariableExpansionRegex property to this value to change the PreprocessReader behavior.

Declaration

public static Regex CurlyVariableExpansionRegex { get; }

Property Value

Type	Description
Regex

DefaultEnvironmentVariable

The default value to use for an undefined environment variable or null if a KeyNotFoundException is to be thrown when a undefined environment variable is referenced. This defaults to null.

Declaration

public string DefaultEnvironmentVariable { get; set; }

Property Value

Type	Description
string

DefaultVariable

The default value to use for an undefined normal variable or null if a KeyNotFoundException is to be thrown when a undefined non-environment variable is referenced. This defaults to null.

Declaration

public string DefaultVariable { get; set; }

Property Value

Type	Description
string

DefaultVariableExpansionRegex

The default variable expansion Regex that matches normal variables like $<test>, environment variables like <env:test>, profile references like <profile:my-profile>, and secret references like <secret:NAME:SOURCE>. You can set the VariableExpansionRegex property to this value to change the PreprocessReader behavior.

Declaration

public static Regex DefaultVariableExpansionRegex { get; }

Property Value

Type	Description
Regex

ExpandVariables

Controls whether variables in the source are expanded. This defaults to true.

Declaration

public bool ExpandVariables { get; set; }

Property Value

Type	Description
bool

Indent

The number of spaces to indent the output. This defaults to 0.

Declaration

public int Indent { get; set; }

Property Value

Type	Description
int

LineEnding

Determines the line ending ReadToEnd() and ReadToEndAsync() will append to the lines they read. This defaults to Platform but may be changed to CRLF or LF.

Declaration

public LineEnding LineEnding { get; set; }

Property Value

Type	Description
LineEnding

ParenVariableExpansionRegex

A variable expansion Regex that matches normal variables like $(NAME), environment variables like $(env:NAME), profile value references like $(profile:NAME) and secret references like $(secret:NAME), $(secret:NAME:SOURCE), $(secret:NAME[PROPERTY]) or $(secret:NAME[PROPERTY]:SOURCE)

You can set the VariableExpansionRegex property to this value to change the PreprocessReader behavior.

Declaration

public static Regex ParenVariableExpansionRegex { get; }

Property Value

Type	Description
Regex

ProcessStatements

Controls whether preprocessor statements are processed. This defaults to true.

Declaration

public bool ProcessStatements { get; set; }

Property Value

Type	Description
bool

RemoveBlank

Controls whether blank lines or lines with only whitespace are to be removed while reading. This defaults to false.

Declaration

public bool RemoveBlank { get; set; }

Property Value

Type	Description
bool

RemoveComments

Controls whether comments are removed while reading. This defaults to false.

Declaration

public bool RemoveComments { get; set; }

Property Value

Type	Description
bool

Remarks

note

StripComments returns a blank line for comments and RemoveComments doesn't return a comment line at all.

StatementMarker

The leading character used to identify a preprocessing statement. This defaults to the pound sign (#).

Declaration

public char StatementMarker { get; set; }

Property Value

Type	Description
char

StripComments

Controls whether comments are stripped out while reading. This defaults to true.

Declaration

public bool StripComments { get; set; }

Property Value

Type	Description
bool

Remarks

note

StripComments returns a blank line for comments and RemoveComments doesn't return a comment line at all.

TabStop

Controls whether embedded TAB (\t) characters will be converted into spaces to format tab stops correctly. This defaults to zero which will not process any tabs.

Declaration

public int TabStop { get; set; }

Property Value

Type	Description
int

VariableExpansionRegex

The Regex used to match variable expansions. This defaults to matching variables of the form: ${NAME}.

Declaration

public Regex VariableExpansionRegex { get; set; }

Property Value

Type	Description
Regex

Remarks

note

You may use set encounter situations where the default syntax would conflict with the source text being processed. You map use the CurlyVariableExpansionRegex or ParenVariableExpansionRegex patterns as an alternative.

VariableValidationRegex

INTERNAL USE ONLY: The Regex used to validate variable names.

Declaration

public static Regex VariableValidationRegex { get; }

Property Value

Type	Description
Regex

Methods

AddCommentMarker(string)

Appends a comment marker prefix. This must be a non-empty string including only non-whitespace punctuation characters.

Declaration

public void AddCommentMarker(string marker)

Parameters

Type	Name	Description
string	marker	The comment prefix.

ClearCommentMarkers()

Clears any comment markers, effectively disabling comment stripping.

Declaration

public void ClearCommentMarkers()

Peek()

Reads the next character without changing the state of the reader or the character source. Returns the next available character without actually reading it from the reader.

Declaration

public override int Peek()

Returns

Type	Description
int	An integer representing the next character to be read, or -1 if no more characters are available or the reader does not support seeking.

Overrides

TextReader.Peek()

Exceptions

Type	Condition
ObjectDisposedException	The TextReader is closed.
IOException	An I/O error occurs.

Read()

Reads the next character from the text reader and advances the character position by one character.

Declaration

public override int Read()

Returns

Type	Description
int	The next character from the text reader, or -1 if no more characters are available. The default implementation returns -1.

Overrides

TextReader.Read()

Exceptions

Type	Condition
ObjectDisposedException	The TextReader is closed.
IOException	An I/O error occurs.

Read(char[], int, int)

Reads a specified maximum number of characters from the current reader and writes the data to a buffer, beginning at the specified index.

Declaration

public override int Read(char[] buffer, int index, int count)

Parameters

Type	Name	Description
char[]	buffer	When this method returns, contains the specified character array with the values between index and (index + count - 1) replaced by the characters read from the current source.
int	index	The position in buffer at which to begin writing.
int	count	The maximum number of characters to read. If the end of the reader is reached before the specified number of characters is read into the buffer, the method returns.

Returns

Type	Description
int	The number of characters that have been read. The number will be less than or equal to count, depending on whether the data is available within the reader. This method returns 0 (zero) if it is called when no more characters are left to read.

Overrides

TextReader.Read(char[], int, int)

Exceptions

Type	Condition
ArgumentNullException	buffer is null.
ArgumentException	The buffer length minus index is less than count.
ArgumentOutOfRangeException	index or count is negative.
ObjectDisposedException	The TextReader is closed.
IOException	An I/O error occurs.

ReadAsync(char[], int, int)

Reads a specified maximum number of characters from the current text reader asynchronously and writes the data to a buffer, beginning at the specified index.

Declaration

public override Task<int> ReadAsync(char[] buffer, int index, int count)

Parameters

Type	Name	Description
char[]	buffer	When this method returns, contains the specified character array with the values between index and (index + count - 1) replaced by the characters read from the current source.
int	index	The position in buffer at which to begin writing.
int	count	The maximum number of characters to read. If the end of the text is reached before the specified number of characters is read into the buffer, the current method returns.

Returns

Type	Description
Task<int>	A task that represents the asynchronous read operation. The value of the TResult parameter contains the total number of bytes read into the buffer. The result value can be less than the number of bytes requested if the number of bytes currently available is less than the requested number, or it can be 0 (zero) if the end of the text has been reached.

Overrides

TextReader.ReadAsync(char[], int, int)

Exceptions

Type	Condition
ArgumentNullException	buffer is null.
ArgumentOutOfRangeException	index or count is negative.
ArgumentException	The sum of index and count is larger than the buffer length.
ObjectDisposedException	The text reader has been disposed.
InvalidOperationException	The reader is currently in use by a previous read operation.

ReadBlock(char[], int, int)

Reads a specified maximum number of characters from the current text reader and writes the data to a buffer, beginning at the specified index.

Declaration

public override int ReadBlock(char[] buffer, int index, int count)

Parameters

Type	Name	Description
char[]	buffer	When this method returns, this parameter contains the specified character array with the values between index and (index + count -1) replaced by the characters read from the current source.
int	index	The position in buffer at which to begin writing.
int	count	The maximum number of characters to read.

Returns

Type	Description
int	The number of characters that have been read. The number will be less than or equal to count, depending on whether all input characters have been read.

Overrides

TextReader.ReadBlock(char[], int, int)

Exceptions

Type	Condition
ArgumentNullException	buffer is null.
ArgumentException	The buffer length minus index is less than count.
ArgumentOutOfRangeException	index or count is negative.
ObjectDisposedException	The TextReader is closed.
IOException	An I/O error occurs.

ReadBlockAsync(char[], int, int)

Reads a specified maximum number of characters from the current text reader asynchronously and writes the data to a buffer, beginning at the specified index.

Declaration

public override Task<int> ReadBlockAsync(char[] buffer, int index, int count)

Parameters

Type	Name	Description
char[]	buffer	When this method returns, contains the specified character array with the values between index and (index + count - 1) replaced by the characters read from the current source.
int	index	The position in buffer at which to begin writing.
int	count	The maximum number of characters to read. If the end of the text is reached before the specified number of characters is read into the buffer, the current method returns.

Returns

Type	Description
Task<int>	A task that represents the asynchronous read operation. The value of the TResult parameter contains the total number of bytes read into the buffer. The result value can be less than the number of bytes requested if the number of bytes currently available is less than the requested number, or it can be 0 (zero) if the end of the text has been reached.

Overrides

TextReader.ReadBlockAsync(char[], int, int)

Exceptions

Type	Condition
ArgumentNullException	buffer is null.
ArgumentOutOfRangeException	index or count is negative.
ArgumentException	The sum of index and count is larger than the buffer length.
ObjectDisposedException	The text reader has been disposed.
InvalidOperationException	The reader is currently in use by a previous read operation.

ReadLine()

Reads a line of characters from the text reader and returns the data as a string.

Declaration

public override string ReadLine()

Returns

Type	Description
string	The next line from the reader, or null if all characters have been read.

Overrides

TextReader.ReadLine()

Exceptions

Type	Condition
IOException	An I/O error occurs.
OutOfMemoryException	There is insufficient memory to allocate a buffer for the returned string.
ObjectDisposedException	The TextReader is closed.
ArgumentOutOfRangeException	The number of characters in the next line is larger than Int32.MaxValue

ReadLineAsync()

Reads a line of characters asynchronously and returns the data as a string.

Declaration

public override Task<string> ReadLineAsync()

Returns

Type	Description
Task<string>	A task that represents the asynchronous read operation. The value of the TResult parameter contains the next line from the text reader, or is null if all of the characters have been read.

Overrides

TextReader.ReadLineAsync()

Exceptions

Type	Condition
ArgumentOutOfRangeException	The number of characters in the next line is larger than Int32.MaxValue.
ObjectDisposedException	The text reader has been disposed.
InvalidOperationException	The reader is currently in use by a previous read operation.

ReadToEnd()

Reads all characters from the current position to the end of the text reader and returns them as one string.

Declaration

public override string ReadToEnd()

Returns

Type	Description
string	A string that contains all characters from the current position to the end of the text reader.

Overrides

TextReader.ReadToEnd()

Exceptions

Type	Condition
IOException	An I/O error occurs.
ObjectDisposedException	The TextReader is closed.
OutOfMemoryException	There is insufficient memory to allocate a buffer for the returned string.
ArgumentOutOfRangeException	The number of characters in the next line is larger than Int32.MaxValue

ReadToEndAsync()

Reads all characters from the current position to the end of the text reader asynchronously and returns them as one string.

Declaration

public override Task<string> ReadToEndAsync()

Returns

Type	Description
Task<string>	A task that represents the asynchronous read operation. The value of the TResult parameter contains a string with the characters from the current position to the end of the text reader.

Overrides

TextReader.ReadToEndAsync()

Exceptions

Type	Condition
ArgumentOutOfRangeException	The number of characters is larger than Int32.MaxValue.
ObjectDisposedException	The text reader has been disposed.
InvalidOperationException	The reader is currently in use by a previous read operation.

Set(string, bool)

Sets a variable to an boolean value.

Declaration

public void Set(string name, bool value)

Parameters

Type	Name	Description
string	name	The case sensitive variable name.
bool	value	The option value.

Remarks

note

The value set will be lowercase true or false.

Set(string, object)

Sets a variable to an object value.

Declaration

public void Set(string name, object value = null)

Parameters

Type	Name	Description
string	name	The case sensitive variable name.
object	value	The option value (defaults to the null).

Set(string, string)

Sets a variable to a string value.

Declaration

public void Set(string name, string value = "")

Parameters

Type	Name	Description
string	name	The case sensitive variable name.
string	value	The option value (defaults to the empty string).

SetYamlMode()

Configures the reader for parsing YAML by setting the StatementMarker to "@" and the comment marker to "#".

Declaration

public void SetYamlMode()

Extension Methods

IOExtensions.Lines(TextReader, bool)