API References
Grainite Homepage

Deploying on AWS with CloudFormation

Replicated 3 Node Cluster

Prerequisites

  • AWS cli

  • Kubectl

  • jq

  • zip

  • The CloudFormation script requires that the region in which Grainite will be deployed must have a minimum of 3 availability zones. For example, us-west-1 has only 2 available zones but us-east-2 has 3.

  • Tokens:

    • Helm deploy token (same as GitLab deploy token)

    • Helm username

    • Quay username

    • Quay password

Recommendation: All of the steps below should be performed from a linux VM running within the same virtual private cloud as the target cluster.

Download scripts

The scripts package contains scripts that make it easier to deploy and manage Grainite clusters with scripts for creating roles, VPCs, Grainite clusters, etc.

  1. Run the following to download the CloudFormation AWS and Terraform GCP scripts package tar:

curl -H "DEPLOY-TOKEN: <token>" https://gitlab.com/api/v4/projects/26443204/packages/generic/scripts/<version>/cluster_creation.tar -o cluster_creation.tar

Replace <token> with the deploy token provided to you. Also, replace <version> with the desired version of Grainite (e.g. 2316.1) that needs to be deployed or latest for the latest available version of Grainite.

2. Run the following to extract the script package tar:

tar xvf cluster_creation.tar

Setup and Create Roles

This step only needs to be performed once per region. Subsequent clusters can be created by reusing these roles.

1. Setup Roles

./grainite/scripts/bin/aws-grainite role-awsqs-create -h

role-awsqs-create: Create an AWSQS role to activate in the region to use

Usage:
    role-awsqs-create  STACK-NAME [flags]

Flags:
    -N:         dry run
    -h          help

Example:

./grainite/scripts/bin/aws-grainite role-awsqs-create tom-awstest

The command above should have automatically created and registered execution roles for AWSQS::EKS::CLUSTER and AWSQS::KUBERNETES::HELM in the same region where the role is created. Please confirm the roles were created and are privately registered as shown below.

2. Create a VPC Proxy Role

./grainite/scripts/bin/aws-grainite role-vpcproxy-create -h

role-vpcproxy-create: Create a VPC proxy role for AWSQS to use

Usage:
    role-vpcproxy-create STACK-NAME [flags]

Flags:
    -N:         dry run
    -h          help

As the role that will be created through this command is global, it only needs to be run once.

Setup VPC

Create a dedicated VPC with 1 public subnet and NatGateway. This step can be skipped if the workload will be deployed using an existing VPC (with a NatGateway).

./aws-grainite vpc-create -h

vpc-create: Create a VPC

Usage:
    role-awsqs-create STACK-NAME [flags]

Flags:
    -R:         region
    -V:         environment name
    -s:         public subnet CIDR in the VPC
    -v:         VPC CIDR
    -N:         dry run
    -h          help

Example:

./grainite/scripts/bin/aws-grainite vpc-create tom-awstestvpc -V tom-awstestenv -s 172.31.20.0/24 -v 172.31.0.0/16 -R us-east-2

Create an unsecured 3 node EKS cluster

Create a 3-node cluster in an existing VPC and deploy grainite workload using 3-node-with-vpc.yaml:

./grainite/scripts/bin/aws-grainite cluster-create -h
Creates a new unsecured grainite cluster with the given cluster name

Usage:
    cluster-create CLUSTER-NAME [flags]

Flags:
    -R:         region

    -i:         grainite image name
    -j:         jetty image name
    -r:         image repository URL
    -U:         image repository user name
    -T:         image repository access token

    -H:         helm repo URL
    -u:         helm repo user name
    -K:         helm repo access token
    -C:         helm chart version

    -S:         num servers in cluster, default: 3
    -n:         create the infrastructure without deploying Grainite workload

    -D:         type of persistent disks. default: gp3

    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti

    -c:         cloudformation template file name
    -v:         VPC stack name
    -V:         environment_name
    -k:         instance keypair name
    -s:         subnet1_cidr,subnet2_cidr,subnet3_cidr
    -x:         awsqs stack name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)
    -N:         dry run
    -W:         wait for creation completion, true or false

    -h|--help   help

Where:

  • helm chart version : The release version for the helm chart.

    • Example: for release 2316.1, specify -C 23.16.1 or for release 2317, specify -C 23.17.0

  • environment_name: Environment tag

  • subnet_cidr: CIDR for 3 private subnets to be created in the VPC. These subnets should be already created.

  • awsqs_stack_name: The name of the stack used to create the role to activate the AWSQS extensions

  • instance key: Name of the ssh key pair to connect to worker nodes

  • template_file: The path and filename of the template to use to create the cluster. Default to 3-node-with-vpc.yaml

Example (Need to replace username, passwords and ssh key):

./grainite/scripts/bin/aws-grainite cluster-create tom-awstestgx2
    -v tom-awstestvpc
    -V tom-awstestenv
    -s 172.31.10.0/24,172.31.11.0/24,172.31.12.0/24
    -x tom-awstest
    -R us-east-2
    -C 23.16.1 
    -r quay.io/grainite
    -U QUAY-USERNAME
    -T QUAY-PASSWORD
    -i cluster:2316.1
    -H https://gitlab.com/api/v4/projects/26443204/packages/helm/stable
    -u HELM-USERNAME
    -K HELM-TOKEN
    -k SSH-KEY
    -c 3-node-with-vpc.yaml

Deploy a cluster with TLS and encryption enabled

Encryption can also be enabled on an existing cluster, see the following page for details:

pageEnabling Disk Encryption

Optionally, the following script can be used to create a cluster with encryption and TLS enabled directly:

Note: This will not create the client certificates necessary for TLS. To create these, follow Step 2 under Enabling TLS.

./grainite/scripts/bin/aws-grainite cluster-create-secure -h

Creates a new secure grainite cluster with the given name

Usage:
    cluster-create-secure CLUSTER-NAME [flags]

Env:
    GXS_CERTS_KEYS_DIR:
        Default GXS_CERTS_KEYS_DIR=${HOME}/.gxscerts/

        For secure setup of grainite cluster and enabling disk encryption export
        this env. The server CA CERTS and encryption KEYS will be generated under

            ${GXS_CERTS_KEYS_DIR}/<cluster-name>/{server/, client/, keys/}

Flags:
    -R:         region
    -e:         optional, setup disk encryption also
                keys will be generated under ${GXS_CERTS_KEYS_DIR}/<cluster-name>

    -i:         grainite image name
    -j:         jetty image name
    -r:         image repository URL
    -U:         image repository user name
    -T:         image repository access token

    -H:         helm repo URL
    -u:         helm repo user name
    -K:         helm repo access token
    -C:         helm chart version

    -S:         num servers in cluster, default: 3

    -D:         type of persistent disks. default: gp3

    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti

    -v:         VPC stack name
    -V:         environment_name
    -k:         instance keypair name
    -s:         subnet1_cidr,subnet2_cidr,subnet3_cidr
    -x:         awsqs stack name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)
    -N:         dry run

    -h|--help   help

Where:

Access the Kubernetes cluster

The Kubernetes cluster control plane and load balancer are private endpoints and can only be accessible from the same VPC and peered VPCs:

First, connect to the cluster by running the following command:

./grainite/scripts/bin/aws-grainite cluster-switch CLUSTER_NAME

Next, get the cluster's service endpoint by running the following command:

./grainite/scripts/bin/aws-grainite cluster-ip

Example:

$> ./grainite/scripts/bin/aws-grainite cluster-ip
a7486243db9f948a48faf52c07a69c31-d52a444e0cf31de2.elb.us-east-2.amazonaws.com

Destroy the cluster

Run the following command to destroy the cluster:

./grainite/scripts/bin/aws-grainite cluster-delete CLUSTER_NAME

Some of the subnets might not be deleted due to lingering lambda functions. To manually cleanup:

  • aws lambda list-functions --region <region-name> to find the function name for the subnets

  • aws lambda delete-function --function-name < function_name > --region <region-name>

Persistent volumes for statefulsets are also automatically removed with aws-grainite cluster-delete.

Optional: Clean up persistent volumes if cluster-delete fails

1. List persistent volumes used in the cluster

kubectl -n default get pv

2. Get the volume ID

aws ec2 describe-volumes --filter Name=status,Values=available Name=tag:cluster_name,Values=$CLUSTER_NAME --query “Volumes[*].VolumeId”3. Delete the volume
aws ec2 delete-volume --volume-id <volume_id>

Where:

  • <volume id> is the ID of the volume.

PreviousCreating your ClusterNextDeploying on Azure with Terraform

Last updated