API References
Grainite Homepage

Deploying on Azure with Terraform

Prerequisites

  • Resource Group (RG) and Virtual Network (VNet) in Azure

  • The Microsoft.ContainerService resource provider needs to be registered for the Azure Subscription, if not already registered.

  • Azure CLI

    • From the Azure docs: Follow the steps in this guide relevant to the instance you plan to run the cluster setup from (we recommend setting up from a Linux VM)

  • Terraform

  • jq

    • sudo apt install -y jq

  • zip

    • sudo apt install zip

  • kubectl

    • sudo az aks install-cli

  • grepcidr

    • sudo apt install -y grepcidr

  • ipcalc

    • sudo apt install -y ipcalc

  • Make sure you have each of the below credentials which have been sent to you by the Grainite team. The credentials will be passed in as arguments to some of the commands in this guide:

    • Helm deploy token (same as GitLab deploy token)

    • Helm username

    • Quay username

    • Quay password

  • The cluster creation step will also take a helm repo URL parameter -H for which the value is https://gitlab.com/api/v4/projects/26443204/packages/helm/stable

  • The Azure user account being used for cluster setup needs to have the following permission(s) are required for the resource group:

    • "Owner"

  • In addition the same Azure user account should have the "Global Administrator" assigned role.

Recommendation: All of the steps below should be performed from a Linux VM running within the same virtual private cloud as the target cluster.

Download Scripts

The scripts package contains scripts that make it easier to deploy and manage Grainite clusters by automating creation of the necessary resources, roles, etc.

  1. Run the following curl command to download the Terraform Azure scripts package tar (also includes Terraform GCP and Cloud Formation AWS scripts):

curl -H "DEPLOY-TOKEN: <token>" https://gitlab.com/api/v4/projects/26443204/packages/generic/scripts/<version>/cluster_creation.tar -o cluster_creation.tar

Replace <token> with the helm/gitlab deploy token provided to you.

Also, replace <version> with the desired version of Grainite (e.g. 2316.1) that needs to be deployed or latest for the latest available version of Grainite.

  1. Run the following to extract the script package tar:

tar xvf cluster_creation.tar

Initialize Terraform

Initialize Terraform by running the following command under grainite/scripts/package/k8s/tf/azure:

terraform init

If successful, the following output will be shown:

Output
Terraform has been successfully initialized!

Log in to Azure

az login

Follow the subsequent onscreen instructions from the az login utility. After following these, you should see information displayed as below after a successful login.

Output
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ATG3RKKWW to authenticate.
[
{
  "cloudName": "AzureCloud",
  "homeTenantId": "b6759b27-d98d-4e1a-ab16-0dcd0d3a6a3d",
  "id": "768ae722-d64d-45e2-bbe8-87dab0c26f21",
  "isDefault": true,
  "managedByTenants": [],
  "name": "Azure Pass - Sponsorship",
  "state": "Enabled",
  "tenantId": "b6759b27-d98d-4e1a-ab16-0dcd0d3a6a3d",
  "user": {
    "name": "scott@kunaalatgrainiteoutlook.onmicrosoft.com",
    "type": "user"
  }
}
]

Create an unsecured 3 node AKS Cluster

To create a 3-node cluster without encryption (e.g. for a test environment) in an existing VPC, run the following script:

grainite/scripts/bin/azure-grainite cluster-create -h

Creates a new unsecured grainite cluster with the given cluster name

Usage:
    cluster-create CLUSTER-NAME [flags]

Flags:

    -i:         grainite image name
    -j:         jetty image name
    -r:         image repository URL
    -U:         image repository user name
    -T:         image repository access token

    -H:         helm repo URL
    -u:         helm repo user name
    -K:         helm repo access token
    -C:         helm chart version

    -S:         num gxs servers in cluster, default: 3
    -n:         create the infrastructure without deploying Grainite workload
    -I:         minimum num compute nodes in cluster, default: 0
    -M:         maximum num compute nodes in cluster, default: 0

    -D:         type of managed disks. default: Standard_LRS

    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti

    -g:         resource group name in azure
    -v:         vNet name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)

    -h|--help   help

Where:

  • helm chart version: The release version for the helm chart.

    • Example: for release 2316.1, specify -C 23.16.1 or for release 2317, specify -C 23.17.0

  • -m: meta size default value is 1Ti.

  • -d: dat size default value is 1Ti.

Example (Need to replace usernames and passwords with those provided to you):

azure-grainite cluster-create toms-cluster

 -v eng
 -g eng-cluster
 -i cluster:2316.1
 -r quay.io/grainite
 -U QUAY_USERNAME
 -T QUAY_PASSWORD
 -H https://gitlab.com/api/v4/projects/26443204/packages/helm/stable
 -u gitlab+deploy-token-123456
 -K 4354QeeGqdfdfEt454
 -C 23.16.1
azure-grainite cluster-create-secure toms-cluster

 -v eng
 -g eng-cluster
 -i cluster:2316.1
 -r quay.io/grainite
 -U QUAY_USERNAME
 -T QUAY_PASSWORD
 -H https://gitlab.com/api/v4/projects/26443204/packages/helm/stable
 -u gitlab+deploy-token-123456
 -K 4354QeeGqdfdfEt454
 -C 23.16.1
 -e

Deploy a cluster with TLS and encryption enabled

Encryption can also be enabled on an existing cluster, see the following page for details:

pageEnabling Disk Encryption

Optionally, the following script can be used to create a cluster with encryption and TLS enabled directly:

Note: This will not create the client certificates necessary for TLS. To create these, follow Step 2 under Enabling TLS.

grainite/scripts/bin/azure-grainite cluster-create-secure -h

Creates a new secure grainite cluster with the given name

Usage:
    cluster-create-secure CLUSTER-NAME [flags]

Env:
    GXS_CERTS_KEYS_DIR:
        Default GXS_CERTS_KEYS_DIR=${HOME}/.gxscerts/

        For secure setup of grainite cluster and enabling disk encryption export
        this env. The server CA CERTS and encryption KEYS will be generated under

            ${GXS_CERTS_KEYS_DIR}/<cluster-name>/{server/, client/, keys/}

Flags:

    -e:         optional, setup disk encryption also
                keys will be generated under ${GXS_CERTS_KEYS_DIR}/<cluster-name>

    -i:         grainite image name
    -j:         jetty image name
    -r          image repository URL
    -U          image repository user name
    -T          image repository access token

    -H          helm repo URL
    -u          helm repo user name
    -K          helm repo access token
    -C          helm chart version

    -S          num servers in cluster, default 3
    -I:         minimum num compute nodes in cluster, default: 0
    -M:         maximum num compute nodes in cluster, default: 0

    -D:         type of managed disks. default: ${GXS_MDISK_TYPE}

    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti

    -g:         resource group name in ${GXS_CLOUD_CTX}
    -v:         vNet name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)

    -x:         name of the user-assigned Managed Identity in Azure AD which has permission to access the Key Vault
    -y:         name of Azure Key Vault for disk encryption with KMS    (try: az keyvault list)
    -Y:         (optional) URL of Azure Key Vault (by default, this is "https://<key vault name>.vault.azure.net". Try: az keyvault show --name <vault name>. URL will be stored as 'vaultUri')
    -z:         name of Azure Key in Key Vault  (try:  az keyvault key list --vault-name <vault name>)

    -h|--help   help

Example (Need to replace username and passwords):

azure-grainite cluster-create-secure toms-cluster

 -v eng
 -g eng-cluster
 -i cluster:2316.1
 -r quay.io/grainite
 -U QUAY_USERNAME
 -T QUAY_PASSWORD
 -H https://gitlab.com/api/v4/projects/26443204/packages/helm/stable
 -u gitlab+deploy-token-123456
 -K 4354QeeGqdfdfEt454
 -C 23.16.1
 -e
 -x eng-cluster-managed-id1
 -y -keyvault1
 -z eng-cluster-keyvault1-key

Where:

  • All flags are the same as those passed in the instructions to Create an unsecured 3 node AKS Cluster except for the flag -e which, when used, enables encryption at rest along with TLS.

  • The -x, -y , -Y, -z options are needed for at-rest encryption using a key provided via Azure KMS. More details here.

Access the Kubernetes cluster

The Kubernetes config is set up as part of cluster creation using prep_cluster.sh. However, you can also modify it to use another cluster.

Use the following script to list all available clusters:

./grainite/scripts/bin/azure-grainite cluster-list

Use the following script to connect to an existing cluster:

./grainite/scripts/bin/azure-grainite cluster-switch <CLUSTER-NAME>

Use the following script to get the IP address of a cluster after connecting to it:

./grainite/scripts/bin/azure-grainite cluster-ip

Use the following script to get the name of the cluster that is being used:

./grainite/scripts/bin/azure-grainite cluster-current

Destroy the cluster

Running cluster-delete will clean up the PVCs used in the cluster. To preserve the volumes after cluster deletion, delete the cluster from the Azure console instead.

./grainite/scripts/bin/azure-grainite cluster-delete <CLUSTER-NAME>
PreviousDeploying on AWS with CloudFormationNextDeploying on GCP with Terraform

Last updated