Orso Labs

Keeping Packer and Terraform images versioned, synchronized and DRY

Posted at — Sep 2, 2020

A cloud agnostic approach to control which version of a Packer image is used in Terraform, allowing independent development by team members and git-based rollbacks.

Our goals

It is 2020. Why are we still dealing with VMs?

For many reasons. Sometimes we have to deal with non-Linux operating systems. Sometimes we are building our own infra to run our own container orchestrator. Sometimes we are stuck in legacy. Sometimes a good old ASG is good enough. Sometimes there are trade-offs.

Why not a simple “latest”?

For some Terraform providers it is possible to use data filters to obtain the equivalent of a Docker “latest” tag. For example, using the AWS AMI data source:

data "aws_ami" "example" {
  most_recent = true                   <== this is "latest"
  name_regex  = "^myami-\\d{3}"
  ...
  filter {
    name   = "name"
    values = ["myami-*"]
  }
  ...
}

But “latest In production environment” has several drawbacks:

Sure, the ID could be extracted from the Terraform state, but it is not immediately visible.

Versioning without tags: naming conventions

Packer allows to give a name to an image, but since it targets many disparate builders, it doesn’t have the notion of a Docker tag or equivalent. How could we convey versioning information then?

A simple solution is to use a naming convention. What we propose is:

<IMAGE-NAME> ::= <ROLE>--<BRANCH>-<REVISION>

where <ROLE> is the role of the instance, for example:

If the default branch for the prod environment is main, then some examples of image names are:

Since we construct the version out of the branch and revision, we are guaranteed that:

See section “Workflow” for complete examples.

Directory layout used for the examples below

├── packer
│   ├── minimal.pkr.hcl
│   └── versions.pkrvars.hcl
└── terraform
    └── demo
        ├── main.tf
        └── variables.tf

Packer now supports enough HCL to pull the trick

HCL is the HashiCorp Configuration Language , the default format of Terraform configuration files.

Although at the time of this writing (August 2020), HCL support for Packer (introduction, details) is still in beta, it is already enough to enjoy the enhancements over the prior JSON-based configuration.

This file (packer/minimal.pkr.hcl) is a minimal Packer HCL configuration file to build an image whose name is defined in variable service-a-img-name. As we can see, it supports the familiar Terraform variable syntax.

NOTE: This example uses the Scaleway builder, but there is nothing specific to it. Any cloud provider supported by Packer will work.

variable "service-a-img-name" {}

source "scaleway" "cfg" {
  region          = "par1"
  commercial_type = "DEV1-S"
  # Scaleway public image, ubuntu focal 20.04 x86_64, region: par1
  image           = "cf44b8f5-77e2-42ed-8f1e-09ed5bb028fc"
  ssh_username    = "root"
}

build {
  source "scaleway.cfg" {
    name          = "service-a"
    server_name   = "packer-service-a"
    image_name    = var.service-a-img-name
    snapshot_name = var.service-a-img-name
  }

  # Placeholder for a real provider
  provisioner "shell-local" {
    inline = ["echo hello I would provision: service-a"]
  }
}

We then define the value of the variable in a separate file, packer/versions.pkvars.hcl:

service-a-img-name = "service-a--main-3"

Finally, we can build the image with:

$ packer build -var-file=versions.pkrvars.hcl minimal.pkr.hcl

Note that, until here, we could have obtained the same with the old JSON format:

{
  "variables": {
    "service-a-img-name": "service-a--main-3"
  },

  "builders": [
    {
      "type": "scaleway",
      "image": "{{user `service-a-img-name`}}",
      // ...
    }
  ]
}

and it would have been enough to obtain versioned images between Packer and Terraform (although not DRY).

We get to DRY in the next section.

Using the versioning information from Terraform

NOTE: This example uses the Scaleway provider, but there is nothing specific to it. Any cloud provider supported by Terraform will work.

This is file terraform/demo/variables.tf:

variable "service-a-img-name" {}

This is file terraform/demo/main.tf:

terraform {
  required_version = "~> 0.13.1"
  required_providers {
    scaleway = {
      source  = "scaleway/scaleway"
      version = "~> 1.16"
    }
  }
}

data "scaleway_instance_image" "service-a" {
  name = var.service-a-img-name
}

resource "scaleway_instance_server" "demo-a" {
  type = "DEV1-S"
  image = data.scaleway_instance_image.service-a.id
  name  = "service-a"
  tags  = ["terraform"]     <= not needed, but tag everything!
}

And, since file packer/versions.pkvars.hcl is in HCL, we can read it from Terraform:

$ terraform apply -var-file=../../packer/versions.pkrvars.hcl

Yes. That’s it! DRY image versioning between Packer and Terraform :-)

Unfortunately there is some duplication, the variable service-a-img-name must be declared in two places:

  1. For Packer in packer/minimal.pkr.hcl
  2. For Terraform in terraform/demo/variables.tf

It is a very small price to pay. DRY is good, but fixating about it is useless and can be detrimental (can reduce readability).

But I cannot / don’t want to use a single repository

…so I cannot share the version pinning file between Packer and Terraform.

In this case, simply duplicate for Terraform the definition (that is, assign a value) of variable service-a-img-name:

  1. packer/versions.pkvars.hcl <== no changes
  2. terraform/variables.tf <== already contains the declaration; add the definition here

Also in this case the pinning and thus the control is still in place.

Possibility for drift or human forgetfulness must be taken into account and a procedure should be in place to reduce this risk. This is exactly the same problem as any kind of dependency with software packages.

Workflow

As a reminder, we use the naming convention

<IMAGE-NAME> ::= <ROLE>--<BRANCH>-<REVISION>

Example:

  1. There is an image provisioned with the Nomad client, called nomad-client. The current version in production is nomad-client--main-34. There is a team of multiple persons, of which two, Alice and Bob, are working at the same time on different features for the Nomad client image.

  2. In her Terraform development environment, Alice is working on branch feat-A. Her development images will be named nomad-client--feat-A-1, nomad-client--feat-A-2 and so on. Due to the fact that Packer and Terraform share the image versioning file packer/versions.pkvars.hcl, after having run packer build, it is enough to run terraform apply (in Alice dev environment!) to have the new image deployed.

  3. In his Terraform development environment, Bob is working on branch feat-B. His development images will be named nomad-client--feat-B-1, nomad-client--feat-B-2 and so on.

  4. After having ran the tests and having passed code review, Alice merges first. The prod image will be bumped from nomad-client--main-34 to nomad-client--main-35 and will be identical to the latest development image nomad-client--feat-A-2. We assume that the merge to the main branch corresponds (one way or the other) with the deployment to production.

  5. After having passed code review, it is now time for Bob to merge. He rebases branch feat-B on top of the current main branch. At this point branch feat-B contains also all the changes from the recently merged branch feat-A. He runs the tests for the last time and merges to the main branch. The prod image will be bumped from nomad-client--main-35 to nomad-client--main-36.

Other aspects to consider

How to test the images in order to promote them to production.

Possible enhancements

  1. Especially when dealing with N images, it makes sense to have a small script that bumps the image versions according to the current branch.
  2. It would be nice to tag the git commit that introduces a given image version, so that it is easier to rollback or to do code spelunking.
  3. If the repositories for Packer and Terraform are separated, we could imagine the equivalent of dependabot that makes a PR to update the image version in Terraform each time that a new Packer image is built :-)