Deploying on GCP with Terraform

Replicated 3 Node Cluster

Prerequisites

VPC, Subnet, and NAT in GCP, to allow pods to pull images from the Granite repository.
- An example can be found at Example GKE setup | Cloud NAT | Google Cloud. Steps 1, 3, 4 & 6 from this link need to be completed. There is no need to perform step 2 because Grainite script creates the kubernetes cluster. Make sure for Step 6 to create a NAT per region in case your cluster and VM are in different regions.
gcloud CLI
Terraform
zip
Kubectl
grepcidr - sudo apt install -y grepcidr
Helm

curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

Tokens:
- Helm deploy token (same as GitLab deploy token)
- Helm username
- Quay username
- Quay password
The following permissions are required for the service account:
- Compute Admin
- Kubernetes Engine Admin
- Service Account User
- compute.subnetworks.delete (Compute Network Admin)

Recommendation: All of the steps below should be performed from a linux VM running within the same virtual private cloud as the target cluster.

Download scripts

The scripts package contains scripts that make it easier to deploy and manage Grainite clusters with scripts for creating roles, VPCs, Grainite clusters, etc.

Run the following to download the Terraform GCP and CloudFormation AWS scripts package tar:

curl -H "DEPLOY-TOKEN: <token>" https://gitlab.com/api/v4/projects/26443204/packages/generic/scripts/<version>/cluster_creation.tar -o cluster_creation.tar

Replace <token> with the deploy token provided to you. Also, replace <version> with the desired version of Grainite (e.g. 2316.1) that needs to be deployed or latest for the latest available version of Grainite.

2. Run the following to extract the script package tar:

tar xvf cluster_creation.tar

Initialize Terraform

Initialize Terraform by running the following command under grainite/scripts/package/k8s/tf/gcp:

terraform init

If successful, the following output will be shown:

Terraform has been successfully initialized!

Initialize gcloud CLI

Run the following command to Initialize gcloud by picking a project, region, and authenticating:

gcloud init

Create an unsecured 3 node GKE cluster

Create a 3-node cluster in an existing VPC by running the following script:

./grainite/scripts/bin/gcp-grainite cluster-create -h

Creates a new unsecured grainite cluster with the given cluster name

Usage:
    cluster-create CLUSTER-NAME [flags]

Flags:
    -R:         region

    -i:         grainite image name
    -j:         jetty image name
    -r:         image repository URL
    -U:         image repository user name
    -T:         image repository access token

    -H:         helm repo URL
    -u:         helm repo user name
    -K:         helm repo access token
    -C:         helm chart version

    -S:         num servers in cluster, default: 3
    -n:         create the infrastructure without deploying Grainite workload

    -D:         type of persistent disks. default: pd-standard

    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti

    -N:         subnet CIDR of the VM invoking the deployment
    -P:         project name in gcp
    -v:         vpc name
    -p:         use pre-emptible worker nodes or not, true/false
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)

    -h|--help   help

NOTE:

Grainite helm chart version : The Grainite release version for the helm chart.
- Example: for release 2316.1, specify -C 23.16.1 or for release 2317, specify -C 23.17.0
-m: meta size default value is 1Ti.
-d: dat size default value is 1Ti.
-p: default value is false.

Example (Need to replace username and passwords):

./grainite/scripts/bin/gcp-grainite cluster-create toms-cluster
    -N 172.30.0.0/16 
    -P toms-project   
    -v toms-vpc 
    -R us-west1  
    -i cluster:2316.1  
    -r quay.io/grainite 
    -U QUAY_USERNAME 
    -T QUAY_PASSWORD 
    -H https://gitlab.com/api/v4/projects/26443204/packages/helm/stable 
    -u gitlab+deploy-token-123456 
    -K 4354QeeGqdfdfEt454
    -C 23.16.1

Deploy a cluster with TLS and encryption enabled

Encryption can also be enabled on an existing cluster, see the following page for details:

pageEnabling Disk Encryption

Optionally, the following script can be used to create a cluster with encryption and TLS enabled directly:

Note: This will not create the client certificates necessary for TLS. To create these, follow Step 2 under Enabling TLS.

./grainite/scripts/bin/gcp-grainite cluster-create-secure -h

Creates a new secure grainite cluster with the given name

Usage:
    cluster-create-secure CLUSTER-NAME [flags]

Env:
    GXS_CERTS_KEYS_DIR:
        Default GXS_CERTS_KEYS_DIR=${HOME}/.gxscerts/

        For secure setup of grainite cluster and enabling disk encryption export
        this env. The server CA CERTS and encryption KEYS will be generated under

            ${GXS_CERTS_KEYS_DIR}/<cluster-name>/{server/, client/, keys/}

Flags:
    -R:         region
    -e:         optional, setup disk encryption also
                keys will be generated under ${GXS_CERTS_KEYS_DIR}/<cluster-name>

    -i:         grainite image name
    -j:         jetty image name
    -r          image repository URL
    -U          image repository user name
    -T          image repository access token

    -H          helm repo URL
    -u          helm repo user name
    -K          helm repo access token
    -C          helm chart version

    -S          num servers in cluster, default 3

    -D:         type of persistent disks. default: pd-standard

    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti

    -N:         subnet CIDR of the VM invoking the deployment
    -P:         project name in gcp
    -v:         vpc name
    -p:         use pre-emptible worker nodes or not, true/false
    -x          service account name for disk encryption with KMS
    -y          key ring name
    -z          key name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)

    -h|--help   help

Where:

All flags are the same as those passed in Create a 3 node GKE Cluster except for -e which when passed enables encryption at desk.
The -x, -y and -z options are needed for at-rest encryption using a key provided via Google Cloud KMS. More details here.

Access the Kubernetes cluster

The Kubernetes config is set up as part of cluster creation using prep_cluster.sh. However, you can also switch it, to use another cluster.

Use the following script to list all available clusters:

./grainite/scripts/bin/gcp-grainite cluster-list

Use the following script to connect to an existing cluster:

./grainite/scripts/bin/gcp-grainite cluster-switch CLUSTER-NAME

Use the following script to get the IP address of a cluster after connecting to it:

./grainite/scripts/bin/gcp-grainite cluster-ip

Use the following script to get the name of the cluster that is being used:

./grainite/scripts/bin/gcp-grainite cluster-current

Destroy the cluster

Running the cluster-delete will clean up the PVCs used in the cluster. To preserve the volumes after cluster deletion, delete the cluster from the GCP console instead.

./grainite/scripts/bin/gcp-grainite cluster-delete CLUSTER-NAME

PreviousDeploying on Azure with Terraform NextDeploying on VMWare Tanzu

Last updated 9 months ago