The problem of “shifting all items” in a Terraform array

Photo by Paweł Czerwiński on Unsplash

In Terraform, it’s very easy to use count & count.index to create a series of similar resources from a single array.

locals {
  roles = [
    "roles/viewer",
    "roles/storage.admin",
    "roles/compute.admin",
    "roles/cloudsql.admin"
  ]
}
resource "google_project_iam_member" "group_access" {
  count  = length(local.roles)
  role   = local.roles[count.index]
  member = "group:devs@example.com"
}

The above would create a terraform plan like this:

# google_project_iam_member.group_access[0] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/viewer"
  }
# google_project_iam_member.group_access[1] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/storage.admin"
  }
# google_project_iam_member.group_access[2] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/compute.admin"
  }
# google_project_iam_member.group_access[3] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/cloudsql.admin"
  }

Notice that google_project_iam_member.group_access is an array with indexes [0], [1], [2], [3].

All fair & good. Let’s apply it.

✅

Adding a new item into the array

Now, let’s try adding a new role to the list.

locals {
  roles = [
    "roles/viewer",
    "roles/pubsub.admin", # <-- this role is being newly added
    "roles/storage.admin",
    "roles/compute.admin",
    "roles/cloudsql.admin"
  ]
}

This is where things get interesting. Even though it’s adding 1 single role (which is pretty simple), the plan that will be produced is not so simple.

# google_project_iam_member.group_access[1] must be replaced
-/+ resource "google_project_iam_member" "group_access" {
      ~ etag    = "BwWza97zq7A=" -> (known after apply)
      ~ id      = "my-gcp-project/roles/cloudsql.admin/group:devs@example.com" -> (known after apply)
        member  = "group:devs@example.com"
      ~ project = "my-gcp-project" -> (known after apply)
      ~ role    = "roles/storage.admin" -> "roles/pubsub.admin" # forces replacement
    }
  # google_project_iam_member.group_access[2] must be replaced
-/+ resource "google_project_iam_member" "group_access" {
      ~ etag    = "BwWza97zq7A=" -> (known after apply)
      ~ id      = "my-gcp-project/roles/cloudsql.admin/group:devs@example.com" -> (known after apply)
        member  = "group:devs@example.com"
      ~ project = "my-gcp-project" -> (known after apply)
      ~ role    = "roles/compute.admin" -> "roles/storage.admin" # forces replacement
    }
  # google_project_iam_member.group_access[3] must be replaced
-/+ resource "google_project_iam_member" "group_access" {
      ~ etag    = "BwWza97zq7A=" -> (known after apply)
      ~ id      = "my-gcp-project/roles/cloudsql.admin/group:devs@example.com" -> (known after apply)
        member  = "group:devs@example.com"
      ~ project = "my-gcp-project" -> (known after apply)
      ~ role    = "roles/cloudsql.admin" -> "roles/compute.admin" # forces replacement
    }
  # google_project_iam_member.group_access[4] will be created
  + resource "google_project_iam_member" "group_access" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = "group:devs@example.com"
      + project = (known after apply)
      + role    = "roles/cloudsql.admin"
    }

We wanted to create just 1 new resource. But the plan says it will replace several resources and then create a resource that was already there.

❌

What’s going on here?

If we notice carefully, what we are doing is we are adding one more item in the middle of the array. Thus, item1 in the array (given that first item is item0) which previously used to be "roles/storage.admin" is now becoming "roles/pubsub.admin"; then item2 in the array is changing from "roles/compute.admin" to "roles/storage.admin"; and likewise, it's doing that for all remaining items in the array that follow. Except, the last item in the array (which is now item4) is being treated as a new item with role "roles/cloudsql.admin"- even though this role had already existed and is not really a new entry in our list.

It is shifting all items in the array

This is not really a problem but, rather, the design of arrays — every item in it is with reference to the index of the array. What may seem like a simple addition into the middle of the array is actually a shift of every item in the array. And terraform will treat it exactly as that — remove the previous item that was occupying that index — create a new item that will replace that index.

Although the final result of applying this plan will give us our desired result all fine & well, but shifting all those items is unnecessary computation. And if we have a similar scenario with 20 to 30 roles for a user-group, we can imagine the number of shiftings that will be necessary for terraform to perform.

Also, the produced plan is a bit messy to look at and review.

How can we improve this?

What if, instead of using the data-structure of an array (which uses sequentially numbered indexes) we could use the data-structure of a hashmap (which uses named indexes)?

Enter for_each instead of count.

We can define our TF code a bit differently using for_each & each.value instead of count & count.index:

locals {
  roles = [
    "roles/viewer",
    "roles/storage.admin",
    "roles/compute.admin",
    "roles/cloudsql.admin"
  ]
}
resource "google_project_iam_member" "group_access" {
  for_each = toset(local.roles) # P.S. for_each can act on sets or hashmaps only
  role     = each.value
  member   = "group:devs@example.com"
}

This would produce our first plan (we will show the plan with new-addition later) to look like this:

# google_project_iam_member.group_access["roles/viewer"] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/viewer"
  }
# google_project_iam_member.group_access["roles/storage.admin"] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/storage.admin"
  }
# google_project_iam_member.group_access["roles/compute.admin"] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/compute.admin"
  }
# google_project_iam_member.group_access["roles/cloudsql.admin"] will be created
+ resource "google_project_iam_member" "group_access" {
    + etag    = (known after apply)
    + id      = (known after apply)
    + member  = "group:devs@example.com"
    + project = (known after apply)
    + role    = "roles/cloudsql.admin"
  }

Notice that, this time the google_project_iam_member.group_access is using named indexes like "roles/viewer", "roles/storage.admin", "roles/compute.admin", "roles/cloudsql.admin".

Looks all fair & good. Let’s apply it.

✅

Now, adding a new item in the array

Now, let’s try adding a new role to the list.

locals {
  roles = [
    "roles/viewer",
    "roles/pubsub.admin", # <-- this role is being newly added
    "roles/storage.admin",
    "roles/compute.admin",
    "roles/cloudsql.admin"
  ]
}

Let’s see what the plan for adding this new item would look like now.

# google_project_iam_member.group_access["roles/pubsub.admin"] will be created
  + resource "google_project_iam_member" "group_access" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = "group:devs@example.com"
      + project = (known after apply)
      + role    = "roles/pubsub.admin"
    }

And that’s it.

✅

Since google_project_iam_member.group_access is now using named indexes instead of using sequentially numbered indexes, terraform doesn't need to shift around items in an array to make room for a new item in the middle - that saves us A LOT of computations (especially if we have many items in the array).

And also, it produces a much cleaner plan showing only the specific differences which makes it much easier to review. 👍

Conclusion

It is certainly straight-forward to use count & count.index and, in fact, is recommended for count = <CONDITION> ? 1 : 0 situations where we want to decide whether to create a resource or not based on a condition.

However, for creating a series of similar resources, it is recommended to use for_each & each.value instead, as it saves us from having to shift all items in an array whenever an item needs to be added to (or removed from) the array.