Creation of Azure Kubernetes Cluster using Java and Terraform (IaC)

Introduction to IaC

Infrastructure as a code(IaC) is a way of managing and provisioning the cloud infrastructure using human and machine readable code unlike manual deployment and configuration of cloud resources. Another definition can be a mechanism to automate the deployment, management and configuration of the cloud resources using the code.

Manual configurations can be error prone and may lead to inconsistent infrastructure, erroneous, slow deployment process and security vulnerabilities. On the other side IaC guarantees the more consistency, reliability and ensures fast deployment lifecycle and can be cost effective with less efforts.

Below can be the proposed architecture of azure cluster creation from the custom user interface.

Flow diagram

I will be majorly focusing on “Terraform Azure Service” in this tutorial which is a spring boot microservice who deals with the azure cloud to create the cluster. Azure Storage is being used to store the terraform templates to be executed. Vault service stores the azure credentials like subscriptionId, tenantId, clientId and clientSecret.

Here is how it will work end to end

1. We can a create a user form on UI(React is being used here in flow diagram) which can have combination of text boxes(for cluster name, vpcCIDR block, Subnet CIDR block, disk size, disk type etc) and dropdown (for regions, machine types etc).
2. Node service is an orchestrator or bridge layer in between which will have interaction with Azure to get the static data like regions, machine types etc and as well interaction with event bus. Here are the links to the azure GET API’s

List Geo Regions - List Geo Regions - REST API (Azure App Service)

Availability Sets - List Available Sizes - REST API (Azure Compute)

Apart from that below is the CURL for Azure Authentication (oauth2) which will give you bearer token to be used to access azure API’s

curl — location — request POST ‘https://login.microsoftonline.com/{tenantId}/oauth2/token' \
— header ‘Content-Type: application/x-www-form-urlencoded’ \
— data-urlencode ‘grant_type=client_credentials’ \
— data-urlencode ‘client_id=*****-****–****–****’ \
— data-urlencode ‘client_secret=****~****_*** \
— data-urlencode ‘resource=https://management.azure.com'

Azure login (oauth2) response will look like below

3. Considering the fact that user have already configured the azure application using azure portal and saved it inside the our application (through some user interface which is out of scope for this tutorial), we can ask user to select the applicationId (while creating cluster)which can have reference to the other azure attributes like tenantId, subscriptionId, clientId and clientSecret. One azure subscription can have multiple applications configured within it and each application can have its own clientId and clientSecret. This step can be optional and for PoC perspective we can get the tenantId, subscriptionId, clientId and clientSecret from the properties file or runtime configuration as well.

4. Once user fill/select all the data from the react portal an event will be thrown to event bus (remember it is not advised to be synchronous as cluster creation can take up-to a minute or more than that) with following payload

{
“platform”: “AZURE_KUBERNETES_CLUSTER”,
“metaDatda”: {
“configId”: “60b9ac0e8b6396684738f3b5”,
“action” : “START”
},
“serviceType”: “AZURE_KUBERNETES”,
“terraformParameters”: {
“clustername”: “kubernetes-aks3”,
“region”: “West US”,
“disk_type”: “Managed”,
“disk_size_gb”: 50,
“node_min_count”: 1,
“node_max_count”: 10,
“machine_type”: “Standard_D2_v2”,
“vpcCIDRblock”: “10.1.0.0/18”,
“subnetCIDRblock”: “10.1.0.0/20”,
}
}

I have highlighted the compulsory parameters in bold (its always suggested to use 1 VPC and 1 Subnet, but you can avoid it providing default values from configuration like credentials). For the PoC purpose you can pass only highlighted values.

There are two ways to use terraform in this case

1. To use terraform CLI where we can collect all the inputs from the user preparing the terraform command to execute it via CLI.
2. To use azure terraform client where the we need to specify the terraform directory and terraform client will perform all the stages like INIT, WORKSPACE and APPLY (https://github.com/microsoft/terraform-spring-boot)

In this tutorial we are going to use option #1 as we need to replace dynamic attributes inside terraform whose values we have collected from the react portal(in above diagram)

Pseudo code below for TerraformClient

Pseudo code below for ProcessLauncher

TerraformRequest input DTO

Method to build terraform params

private void buildTerraformParams(TerraformRequest request) {

List<String> workspaceCommand = new ArrayList<>();

workspaceCommand.add(“new”);

String workspaceName = String.format(“%s-%s”, request.getPlatform(), System.currentTimeMillis());

workspaceCommand.add(workspaceName);

request.setWorkspaceCommand(workspaceCommand);

List<String> applyCommand = new ArrayList<>();

if (!CollectionUtils.isEmpty(request.getTerraformParameters())) {

request.getTerraformParameters().entrySet().stream().forEach(param -> applyCommand.add(String.format(“-var=%s=%s”, param.getKey(), param.getValue())));

}

applyCommand.add(“-auto-approve”); // Auto approving the terraform

request.setApplyCommand(applyCommand);

}

Create terraform base directory and pass it to TerraformClient

client.setWorkingDirectory(“/terraform-directory”);

client.workspace(request).get();

client.apply(request).get();

client.output(request).get();

client.apply(request).get() — This will run the terraform command using terraform CLI

5. Terraform Template will look like below

terraform {
required_providers {
helm = “1.3.2”
kubernetes = “1.13.3”
}
}

provider “azurerm” {
subscription_id = var.subscription_id
client_id = var.client_id
client_secret = var.client_secret
tenant_id = var.tenant_id
features {}
}

resource “azurerm_resource_group” “k8s” {
name = “${var.clustername}-resources”
location = var.region
}

resource “azurerm_virtual_network” “vnet” {
name = “${var.clustername}-vnet”
location = azurerm_resource_group.k8s.location
resource_group_name = azurerm_resource_group.k8s.name
address_space = [ var.vpcCIDRblock ]
}

resource “azurerm_subnet” “subnet” {
name = “${var.clustername}-subnet”
resource_group_name = azurerm_resource_group.k8s.name
address_prefixes = [ var.subnetCIDRblock ]
virtual_network_name = azurerm_virtual_network.vnet.name

depends_on = [
azurerm_virtual_network.vnet
]
}

resource “azurerm_kubernetes_cluster” “k8s” {
name = var.clustername
location = azurerm_resource_group.k8s.location
resource_group_name = azurerm_resource_group.k8s.name
dns_prefix = var.clustername
kubernetes_version = var.kubernetes_version

default_node_pool {
name = “default”
vm_size = var.machine_type
enable_auto_scaling = true

os_disk_size_gb = var.disk_size_gb
os_disk_type = var.disk_type
max_count = var.node_max_count
min_count = var.node_min_count

vnet_subnet_id = azurerm_subnet.subnet.id
}

network_profile {
network_plugin = “azure”

}

identity {
type = “SystemAssigned”
}

depends_on = [
azurerm_subnet.subnet
]
}

6. helm.tf will look like below

provider “helm” {
kubernetes {
host = azurerm_kubernetes_cluster.k8s.kube_config.0.host
# username = azurerm_kubernetes_cluster.k8s.kube_config.0.username
# password = azurerm_kubernetes_cluster.k8s.kube_config.0.password
client_key = base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_key)
client_certificate = base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate)
}
}

provider “kubernetes” {
host = azurerm_kubernetes_cluster.k8s.kube_config.0.host
client_key = base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_key)
client_certificate = base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate)
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate)
}

resource “helm_release” “nginx-ingress” {
name = “nginx-ingress”
chart = “/path-to/nginx-ingress”
namespace = “ingress-nginx”
create_namespace = true
timeout = 400

values = [
“${file(“nginx-value.yaml”)}”
]

depends_on = [
azurerm_kubernetes_cluster.k8s
]
}

data “kubernetes_service” “nginx-ingress-controller” {
metadata {
name = “nginx-ingress-controller”
namespace = “ingress-nginx”
}

depends_on = [
“helm_release.nginx-ingress”
]
}

7. Variable.tf will look like below

variable "clustername" {
default = "kubernetes-aks1"
}

variable "region" {
default = "East US"
}

variable "machine_type" {
default = "Standard_D2_v2"
}

variable "node_max_count" {
default = "1"
}

variable "node_min_count" {
default = "1"
}

variable "disk_size_gb" {
default = "50"
}

variable "disk_type" {
default = "Managed"
}

variable kubernetes_version {
default = "1.20.5"
}

variable vpcCIDRblock {
default = "10.1.0.0/16"
}

variable subnetCIDRblock {
default = "10.1.0.0/20"
}

variable "subscription_id" {
default = "eb41814d-980b-4b4a-bf45"
}
variable "client_id" {
default = "9f7d3bc4-3eb6-49f9-9e6f"
}
variable "client_secret" {
default = "QY2PIJwNkKZ~0fUOyS5v6z61IEmw.GPJR~"
}
variable "tenant_id" {
default = "0f0d68d4-e126-4a95-81cb"
}

Example terraform apply command to create AKS cluster

terraform apply -var clustername=kubernetes-cluster -var region=”West US” -var disk_type=Managed -var disk_size_gb=50 -var node_min_count=1 -var node_max_count=10 -var machine_type=Standard_D2_v2 -var vpcCIDRblock=10.1.0.0/18 -var subnetCIDRblock=10.1.0.0/20 — auto-approve