Terraform guide for Grid deployments

Alright … long post … Terraform guide :slight_smile:

Threefold maintains a Terraform provider for the grid. It’s a CLI tool, which might look complicated at first but it’s actually not. Deploying on the Grid using this tool gives you by far the most flexibility and access to all the current grid features. While Weblets are more accessible, Terraform gives you access to a lot more features.

First thing to do is installing it on the client (laptop, desktop, …) your using to deploy on the Grid.
Follow the appropriate guide for your OS.

For Linux this is quite simple:

wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

Once you have Terraform installed we can start to create a deployment file: main.tf
Put each deployment file in a separate folder. Once your deployment file is ready, we will initialize Terraform and the Grid provider for this specific deployment (later in this guide). Initialization will download and create some extra files in the folder, so it’s important to keep these separate with your other Terraform deployments.

One could make many different deployment files so this guide will concentrate on a simple full vm. It’s up to you to experiment with all the available examples and features.

terraform {
  required_providers {
    grid = {
      source = "threefoldtech/grid"
    }
  }
}

provider "grid" {
}

resource "grid_network" "net1" {
    nodes = [3000]
    ip_range = "10.20.0.0/16"
    name = "mynetwork"
    description = "My internal Grid network"
    add_wg_access = true
}
resource "grid_deployment" "d1" {
  node = 3000
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 3000, "")
  disks {
    name = "root"
    size = 50
  }
    vms {
    name = "tftest"
    description ="Terraform deployment test"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-22.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 8192
    mounts {
        disk_name = "root"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ADD YOUR SSH KEY HERE"
    }
  }
}
output "wg_config" {
    value = grid_network.net1.access_wg_config
}
output "node1_vm1_ip" {
    value = grid_deployment.d1.vms[0].ip
}
output "public_ip" {
    value = grid_deployment.d1.vms[0].computedip
}
output "public_ip6" {
    value = grid_deployment.d1.vms[0].computedip6
}
output "ygg_ip" {
    value = grid_deployment.d1.vms[0].ygg_ip
}

So with this example, we will deploy an Ubuntu 22.04 flist on node 3000, with 4 cpu’s / 8GB ram / 50GB disk size / public IPv4/6 / Yggdrasil IP and Wireguard config.
The above example consists of:

  • a provider for Terraform
  • a network resource
  • a deployment, being a full VM
  • output statements to get the IP’s and Wireguard config after the deployment. These statements will give you an easy summary after a deployment.

Deploying on the Grid like this will not provide you with automatic node selection. You have to manually look for one that suits your requirements here: https://dashboard.grid.tf/explorer/nodes

Once the main.tf deployment file has been created, we have to set some environment variables. Terraform will use these environment variables for it’s deployments to identify your wallet and to know where to send requests and query’s. The easiest way is to create a file which has these env vars and source it into your terminal sessions every time you want to deploy or destroy something on the gird.
!! Please be very careful with this file as it will contain your wallet mnemonic !!
Example:

export MNEMONICS="YOUR MNEMONIC HERE"
export NETWORK="main"

In Linux you set the environment variables like this: source <filename>
In Windows or Mac I have no idea, if someone could post below would be nice. If your using Windows, I would advice to install the Windows subsystem for Linux anyway :wink:
Some docs on environment variables in Linux or in Terraform.

Once the deployment file is made and environment variables set we can initialize. Move into the folder and execute the following:

cd <folder>
terraform init

Once initialization is ok you can start the actual deploy:

terraform apply -parallelism=1 -auto-approve

To destroy a deployment do:

terraform destroy -parallelism=1 -auto-approve

After a deployment is done you can show it’s details again with:

terraform show

Some important tips:

  • In the example you can see the Node ID is defined 3 times. If you want a different node you have to change it at all 3 places in the main.tf file. Following this example, replace ‘3000’ with your new Node ID 3 times.

  • The subnet you define for the network resource always have to be a /16. This is required if you for example deploy 2 or more vm’s on different nodes but within the same internal subnet / Grid network resource. Each node that is part of the network resource, will get a /24 defined.

  • A VM name can not be longer then 12 characters and can’t contain spaces or special signs.

  • Memory is in MB (Megabytes) and Disk size is in GB (Gigabyte)

  • add_wg_access = true will enable Wireguard on the network resource. If this is set to true you will get a wireguard config which you can use to connect to the internal subnet running on the gird

  • planetary = true will enable Planetary Network / Yggdrasil accessibility for your deployment.

  • SSH_KEY ="ssh-xx..xx..xx" will have to filled in with your public SSH key. This public key will be added to the vm, so you can use your private ssh key to login once a VM is deployed with the user root.

  • IP range of the internal subnet should always be within one of the known Internal Subnet ranges. Example: ip_range = "10.20.0.0/16"

  • Different flists can be found here for full VM’s and here for all other

  • If a new version of the Grid Terraform provider is out, you can update a deployment like this: terraform init -upgrade. This has to be done in each deployment folder, since the provider files are stored locally in the deployment folder itself.

  • If you encounter issues have a look if the issue is known or not, if not create one: https://github.com/threefoldtech/terraform-provider-grid/issues

  • The Grid Provider repo contains a lot of examples for experimenting and using all the features the Threefold Grid has to offer. You can combine these examples to build what you want.

  • Here you can find all the different Terraform options explained for each kind of deployment

There might be a few other / better ways to do things with Terraform. I’m no expert, only learned things along the way. Don’t hesitate to respond if something is wrong, something was forgotten or if there is a better way to do things.

5 Likes

Another example to create two VM’s on two different nodes within the same private subnet.

terraform {
  required_providers {
    grid = {
      source = "threefoldtech/grid"
    }
  }
}

provider "grid" {
}

resource "grid_network" "net1" {
    nodes = [311, 312]
    ip_range = "10.32.0.0/16"
    name = "internal"
    description = "Internal subnet"
    add_wg_access = true
}
resource "grid_deployment" "d1" {
  node = 311
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 311, "")
  disks {
    name = "data"
    size = 25
  }
    vms {
    name = "vm1"
    description ="Test vm 1"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-22.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 8192
    mounts {
        disk_name = "data"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ADD YOUR SSH KEY HERE"
    }
  }
}
resource "grid_deployment" "d2" {
  node = 312
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 312, "")
  disks {
    name = "data"
    size = 25
  }
    vms {
    name = "vm2"
    description ="Test vm 2"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-22.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 8192
    mounts {
        disk_name = "data"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ADD YOUR SSH KEY HERE"
    }
  }
}
output "wg_config" {
value = grid_network.net1.access_wg_config
}
output "node1_vm1_ip" {
value = grid_deployment.d1.vms[0].ip
}
output "public_ip" {
value = grid_deployment.d1.vms[0].computedip
}
output "public_ip6" {
value = grid_deployment.d1.vms[0].computedip6
}
output "ygg_ip" {
value = grid_deployment.d1.vms[0].ygg_ip
}

Notice:

  • You have to add each node to the network resource. If not the vm there will not be able to join the private subnet: nodes = [311, 312]

  • You can’t predefine private or public IP’s in advance.

  • Increment each deployment number: resource "grid_deployment" "d1" . So the next deployment would be resource "grid_deployment" "d2" and so on …

1 Like

Hi i need more then one public ip actually 3 ip’s how can i achieve this ?

Don’t think that is possible.
Could you explain the use case for that?

Thanks for the excellent post, @linkmark! We were long overdue for some extra guidance around working in Terraform :slight_smile:

The only thing I have to add right now is that you can also define your network and mnemonics directly in the main.tf file, within the provider block like this:

provider "grid" {
  mnemonics = "write your words here..."
  network = "main"
}

It’s especially helpful if you create deployments across different networks using different accounts.

You can get one public IP per VM right now.

Not sure how feasible this is in the current architecture or if it would apply to your use case, @manfred, but I think it could also be useful to have “floating IPs” available that aren’t linked to specific workloads.

For example, the metalLB load balancer for Kubernetes expects that you assign an IP range it can freely allocate from for new load balancers. Providers like Hetzner allow customers to reserve a “vswitch” with public IPs that can be dynamically assigned to workloads on servers or VMs that are also assigned to the switch.

I think this is a nice feature for creating redundancy, so that another node can claim the public IP and continue routing traffic if the node using it goes down. That said, I’m no expert in this realm and maybe there’s a different way to achieve the same ends within our current architecture.

1 Like

Nice, did not know that! While that is very handy indeed it makes the main.tf file containing the mnemonic thus it must be handled with care regarding where to save it. But good to know :slight_smile:

1 Like

I also need 2 I.p. Deployment for full utilization of DirectAdmins built in production ready domain resell platform :slight_smile:

Yeah, just don’t go pushing that into version control :laughing:

I would expect this should be simple to add to Zos and I don’t see any good reason not to.

I would go further and would write the critical information’s in a separate file named credentials.auto.tfvars terraform parse this every time you apply you don’t need to specify it.

you just need to add the variables to your main.tf file

credentials.auto.tfvars:

mnemonics = "bla bla bla blub"
sshkey = "ssh-rsa AAAAB3NzaC1yc2EAAxxxxxxxxxxxxxxxxxxxx=="

main.tf:

variable "mnemonics" {
  type = string
}

variable "sshkey" {
  type = string
}

provider "grid" {
    mnemonics = var.mnemonics
    network = "dev"
    rmb_proxy_url = "https://gridproxy.dev.grid.tf/"
    use_rmb_proxy = true
}

resource "grid_deployment" "vm1" {
  node = 31
  network_name = grid_network.net1.name
  disks {
    name = "disk1"
    size = 10
    description = "disk1"
  }
  vms {
    planetary = true
    publicip6 = true
    ip = "10.1.0.2"
    name = "vm1"
    flist = "https://hub.grid.tf/azmy.3bot/ubuntu-22.04-LTS.flist"
    cpu = 8
    memory = 8192
    entrypoint = "/sbin/zinit init"
    mounts {
      disk_name = "disk1"
      mount_point = "/opt"
    }
    env_vars = {
      SSH_KEY = var.sshkey
      }
  }

Further you can add the variables into an variables.tf file:

terraform-repo/
├── applicationGateway/
│   ├── main.tf
│   ├── variables.tf
├── appService/
│   ├── main.tf
│   └── variables.tf
├── main.tf
├── variables.tfvars
└── credentials.auto.tfvars

Nice. I was aware of the variables feature in Terraform but haven’t tried it out yet. Now I wonder… is there a way for the main.tf files in each subfolder to reference once variable specification in the parent folder?