Class AwsCli

Wraps the AWS-CLI with methods for common operations.

note

The class methods require that the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables be already set with the required AWS credentials.

Inheritance

object

AwsCli

Namespace: Neon.Deployment

Assembly: Neon.Deployment.dll

Syntax

public static class AwsCli

Methods

ExecuteSafe(params string[])

Executes an AWS-CLI command, ensuring that it completed without error.

Declaration

public static void ExecuteSafe(params string[] args)

Parameters

Type	Name	Description
string[]	args	The command and arguments.

Exceptions

Type	Condition
ExecuteException	Thrown for command errors.

RemoveCredentials()

Removes the AWS credential environment variables.

Declaration

public static void RemoveCredentials()

S3Download(string, string)

Downloads a file from S3.

Declaration

public static void S3Download(string sourceUri, string targetPath)

Parameters

Type	Name	Description
string	sourceUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.
string	targetPath	The target file path.

S3DownloadBytes(string)

Downloads a file from S3 as a byte array.

Declaration

public static byte[] S3DownloadBytes(string sourceUri)

Parameters

Type	Name	Description
string	sourceUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.

Returns

Type	Description
byte[]

S3DownloadText(string, Encoding)

Downloads a file from S3 as text.

Declaration

public static string S3DownloadText(string sourceUri, Encoding encoding = null)

Parameters

Type	Name	Description
string	sourceUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.
Encoding	encoding	Optionally specifies the character encoding. This defaults to UTF8.

Returns

Type	Description
string

S3Remove(string, bool, string, string)

Removes one S3 objects.

Declaration

public static void S3Remove(string targetUri, bool recursive = false, string include = null, string exclude = null)

Parameters

Type	Name	Description
string	targetUri	The target S3 URI or prefix for the object(s) to be removed. This may be either an s3://BUCKET[/KEY] or a https://s3.REGION.amazonaws.com/BUCKET[/KEY] URI referencing an S3 bucket and key. Note that the key is optional which means that all objects in the bucket are eligible for removal.
bool	recursive	Optionally indicates targetUri specifies a folder prefix and that all objects within the folder are eligble for removal.
string	include	Optionally specifies a pattern specifying the objects to be removed.
string	exclude	Optionally specifies a pattern specifying objects to be excluded from removal.

S3Upload(Stream, string, bool, string, bool)

Uploads the contents of a stream to an S3 bucket.

Declaration

public static void S3Upload(Stream input, string targetUri, bool gzip = false, string metadata = null, bool publicReadAccess = false)

Parameters

Type	Name	Description
Stream	input	The input stream.
string	targetUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.
bool	gzip	Optionally indicates that the target content encoding should be set to gzip.
string	metadata	Optionally specifies HTTP metadata headers to be returned when the object is downloaded from S3. This formatted as as comma separated a list of key/value pairs like: Content-Type=text,app-version=1.0.0 note AWS supports system as well as custom headers. System headers include standard HTTP headers such as Content-Type and Content-Encoding. Custom headers are required to include the x-amz-meta- prefix. You don't need to specify the x-amz-meta- prefix for setting custom headers; the AWS-CLI detects custom header names and adds the prefix automatically. This method will strip the prefix if present before calling the AWS-CLI to ensure the prefix doesn't end up being duplicated.
bool	publicReadAccess	Optionally grant the upload public read access.

S3Upload(string, string, bool, string, bool)

Uploads a file from the local workstation to S3.

Declaration

public static void S3Upload(string sourcePath, string targetUri, bool gzip = false, string metadata = null, bool publicReadAccess = false)

Parameters

Type	Name	Description
string	sourcePath	The source file path.
string	targetUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.
bool	gzip	Optionally indicates that the target content encoding should be set to gzip.
string	metadata	Optionally specifies HTTP metadata headers to be returned when the object is downloaded from S3. This formatted as as comma separated a list of key/value pairs like: Content-Type=text,app-version=1.0.0 note AWS supports system as well as custom headers. System headers include standard HTTP headers such as Content-Type and Content-Encoding. Custom headers are required to include the x-amz-meta- prefix. You don't need to specify the x-amz-meta- prefix for setting custom headers; the AWS-CLI detects custom header names and adds the prefix automatically. This method will strip the prefix if present before calling the AWS-CLI to ensure the prefix doesn't end up being duplicated.
bool	publicReadAccess	Optionally grant the upload public read access.

S3UploadBytes(byte[], string, bool, string, bool)

Uploads a byte array to an S3 bucket.

Declaration

public static void S3UploadBytes(byte[] bytes, string targetUri, bool gzip = false, string metadata = null, bool publicReadAccess = false)

Parameters

Type	Name	Description
byte[]	bytes	The byte array being uploaded.
string	targetUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.
bool	gzip	Optionally indicates that the target content encoding should be set to gzip.
string	metadata	Optionally specifies HTTP metadata headers to be returned when the object is downloaded from S3. This formatted as as comma separated a list of key/value pairs like: Content-Type=text,app-version=1.0.0 note AWS supports system as well as custom headers. System headers include standard HTTP headers such as Content-Type and Content-Encoding. Custom headers are required to include the x-amz-meta- prefix. You don't need to specify the x-amz-meta- prefix for setting custom headers; the AWS-CLI detects custom header names and adds the prefix automatically. This method will strip the prefix if present before calling the AWS-CLI to ensure the prefix doesn't end up being duplicated.
bool	publicReadAccess	Optionally grant the upload public read access.

S3UploadMultiPart(string, string, string, string, string, bool, long, bool, Action<long>)

Uploads a file in multiple parts from the local workstation to S3, returning the DownloadManifest details. required by DownloadMultiPart(DownloadManifest, string, DownloadProgressDelegate, IRetryPolicy, TimeSpan, bool) and DownloadMultiPartAsync(DownloadManifest, string, DownloadProgressDelegate, TimeSpan, IRetryPolicy, bool, CancellationToken) to actually download the entire file. The URI to the uploaded DownloadManifest details is also returned.

See the remarks for details about how this works.

Declaration

public static (DownloadManifest manifest, string manifestUri) S3UploadMultiPart(string sourcePath, string targetFolderUri, string version = null, string name = null, string filename = null, bool noMd5File = false, long maxPartSize = 78643200, bool publicReadAccess = false, Action<long> progressAction = null)

Parameters

Type	Name	Description
string	sourcePath	Path to the file being uploaded.
string	targetFolderUri	The target S3 URI structured like https://s3.REGION.amazonaws.com/BUCKET/... URI referencing an S3 bucket and the optional folder where the file's download information and parts will be uploaded. note The s3:// URI scheme is not supported.
string	version	Optionally specifies the download file version.
string	name	Optionally overrides the download file name specified by sourcePath to initialize Name.
string	filename	Optionally overrides the download file name specified by sourcePath to initialize Filename.
bool	noMd5File	This method creates a file named [sourcePath.md5] with the MD5 hash for the entire uploaded file by default. You may override this behavior by passing noMd5File=true.
long	maxPartSize	Optionally overrides the maximum part size (defaults to 75 MiB).
bool	publicReadAccess	Optionally grant the upload public read access.
Action<long>	progressAction	Optional action called as the file is uploaded, passing the long percent complete.

Returns

Type	Description
(DownloadManifest manifest, string manifestUri)	The DownloadManifest information as well as the URI to the uploaded manifest.

Remarks

This method works by splitting the sourcePath file into parts no larger than maxPartSize bytes each and the uploading these parts to the specified bucket and path along with a file holding DownloadManifest information describing the download and its constituent parts. This information includes details about the download including the overall MD5 and size as well records describing each part including their URIs, sizes and MD5.

The DownloadManifest details returned include all of the information required by DownloadMultiPart(DownloadManifest, string, DownloadProgressDelegate, IRetryPolicy, TimeSpan, bool) and DownloadMultiPartAsync(DownloadManifest, string, DownloadProgressDelegate, TimeSpan, IRetryPolicy, bool, CancellationToken) to actually download the entire file and the URI returned references these msame details as uploaded to S3.

You'll need to pass sourcePath as the path to the file being uploaded and targetFolderUri as the S3 location where the download information and the file parts will be uploaded. targetFolderUri may use with the https:// or s3:// URI scheme.

By default the uploaded file and parts names will be based on the filename part of sourcePath, but this can be overridden via filename. The DownloadManifest information for the file will be uploaded as FILENAME.manifest and the parts will be written to a subfolder named FILENAME.parts. For example, uploading a large file named myfile.json to https://s3.uswest.amazonaws.com/mybucket will result S3 file layout like:

https://s3.uswest.amazonaws.com/mybucket
    myfile.json.manifest
    myfile.json.parts/
        part-0000
        part-0001
        part-0002
        ...

The URI returned in this case will be https://s3.uswest.amazonaws.com/mybucket/myfile.json.manifest.

note

This method uses two threads for uploading the parts because it seems to take the AWS CLI tool several seconds to actually start the upload, resulting in a lot of wasted uploading time.

S3UploadText(string, string, bool, string, bool, Encoding)

Uploads text to an S3 bucket.

Declaration

public static void S3UploadText(string text, string targetUri, bool gzip = false, string metadata = null, bool publicReadAccess = false, Encoding encoding = null)

Parameters

Type	Name	Description
string	text	The text being uploaded.
string	targetUri	The target S3 URI. This may be either an s3://BUCKET/KEY or a https://s3.REGION.amazonaws.com/BUCKET/KEY URI referencing an S3 bucket and key.
bool	gzip	Optionally indicates that the target content encoding should be set to gzip.
string	metadata	Optionally specifies HTTP metadata headers to be returned when the object is downloaded from S3. This formatted as as comma separated a list of key/value pairs like: Content-Type=text,app-version=1.0.0 note AWS supports system as well as custom headers. System headers include standard HTTP headers such as Content-Type and Content-Encoding. Custom headers are required to include the x-amz-meta- prefix. You don't need to specify the x-amz-meta- prefix for setting custom headers; the AWS-CLI detects custom header names and adds the prefix automatically. This method will strip the prefix if present before calling the AWS-CLI to ensure the prefix doesn't end up being duplicated.
bool	publicReadAccess	Optionally grant the upload public read access.
Encoding	encoding	Optionally specifies the text encoding. This defaults to UTF8.

SetCredentials(string, string, string, string)

Sets the AWS credential environment variables by loading them from 1Password.

Declaration

public static void SetCredentials(string awsAccessKeyId = "AWS_NEONFORGE[ACCESS_KEY_ID]", string awsSecretAccessKey = "AWS_NEONFORGE[SECRET_ACCESS_KEY]", string vault = null, string masterPassword = null)

Parameters

Type	Name	Description
string	awsAccessKeyId	Optionally overrides the AWS access key ID 1Password secret name.
string	awsSecretAccessKey	Optionally overrides the AWS access key 1Password secret name.
string	vault	Optionally overrides the current user's 1Password vault.
string	masterPassword	Optionally specifies the master 1Password.