Importing existing cloud infrastructure with Terraform
June 9, 2024 Terraform IaC
Overview
As a Cloud DevOps engineer, you sometimes need to import an existing cloud infrastructure into Terraform. Terraform has a capability to import existing cloud resources. It allows you to take your cloud resources created through ClickOps and import them into Terraform state file to be managed under Terraform IaC.
Before Terraform v1.5.0, the only way to import resources into Terraform was using the terraform import command line. This command has the following limitations:
- You can only import one resource at a time (painful if you have many resources to import)
- The Terraform state file is modified immediately, ie. you don’t get to preview the outcome
- A matching resource block needs to be manually coded, which can be a laborious process
In June 12 2023, HashiCorp released Terraform v1.5.0 and introduced a new config-driven import block. I have recently used this capability as I needed to import many (50+) AWS cloud resources that were created manually.
I should have experimented this feature when it was first released about a year ago because prior to this, I have been utilising the old terraform import command. Long story short, using the Terraform import block saved me a lot of grief and I feel like sharing my experience by writing a blog post about it.
Terraform import block
The Terraform import block allows you to define import operations in code. It addresses all of the terraform import CLI limitations. You can do resources import in bulk and the operations are part of the standard Terraform plan and apply process, which eliminates the risk of unintended state file modification. That means you can now run terraform plan to preview the operation first, and then run terraform apply to execute the import operation.
The import block has the following arguments:
to- the imported resource (HCL) address in the state fileid- the cloud resource ID stringprovider(optional) - an optional custom resource provider
To find out what resources are importable and the format, please check HashiCorp cloud provider documentation. An example of import block for Amazon EC2 instance:
import {
# HCL resource address
to = aws_instance.web
# Cloud resource ID
id = "i-12345678"
}
And the best part is you can now generate the imported resource code automatically. This saves a significant amount of time as you don’t need to spend time writing code to match the imported resources. You simply need to execute a plan operation with the new -generate-config-out parameter to automatically create the matching resource configuration code in a file you specify in the parameter value.
You can review and edit the generated code, and then run an terraform apply to complete the resources import into the state file. Please note that the code generation during import is currently experimental.
A simple demo
For the purpose of demo, I simply created the following AWS resources using the AWS console UI:
- A security group named
tf-import-sgwith an inbound rule for port 22 and outbound rule for port 443 - An S3 bucket named
tf-import-s3
Step 1: Create Terraform import configuration
I created the Terraform configuration file, eg. imports.tf, to place the import block for the resources that I want to import.
1# Security group import
2import {
3 to = aws_security_group.tf_import
4 id = "sg-05af4ed76c86e2b2b"
5}
6
7# S3 bucket import
8import {
9 to = aws_s3_bucket.tf_import
10 id = "tf-import-s3"
11}
Step 2: Preview and generate the configuration code
Next step is to run the terraform plan with the -generate-config-out parameter to preview and generate configuration code.
terraform plan -generate-config-out=generated.tf
The terraform plan output:
wkhoo-macbook:tf-import $ terraform plan -generate-config-out=generated.tf
aws_security_group.tf_import: Preparing import... [id=sg-05af4ed76c86e2b2b]
aws_s3_bucket.tf_import: Preparing import... [id=tf-import-s3]
aws_security_group.tf_import: Refreshing state... [id=sg-05af4ed76c86e2b2b]
aws_s3_bucket.tf_import: Refreshing state... [id=tf-import-s3]
Terraform will perform the following actions:
# aws_s3_bucket.tf_import will be imported
# (config will be generated)
resource "aws_s3_bucket" "tf_import" {
arn = "arn:aws:s3:::tf-import-s3"
bucket = "tf-import-s3"
bucket_domain_name = "tf-import-s3.s3.amazonaws.com"
bucket_regional_domain_name = "tf-import-s3.s3.ap-southeast-2.amazonaws.com"
hosted_zone_id = "Z1WCIGYICN2BYD"
id = "tf-import-s3"
object_lock_enabled = false
region = "ap-southeast-2"
request_payer = "BucketOwner"
tags = {}
tags_all = {}
grant {
id = "754c7d875755bfb329f9bd5cf61f8eacb3db030406e6765caf8e869da5036366"
permissions = [
"FULL_CONTROL",
]
type = "CanonicalUser"
}
server_side_encryption_configuration {
rule {
bucket_key_enabled = true
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
versioning {
enabled = false
mfa_delete = false
}
}
# aws_security_group.tf_import will be imported
# (config will be generated)
resource "aws_security_group" "tf_import" {
arn = "arn:aws:ec2:ap-southeast-2:308825204162:security-group/sg-05af4ed76c86e2b2b"
description = "Demo of security group Terraform import"
egress = [
{
cidr_blocks = [
"0.0.0.0/0",
]
description = ""
from_port = 443
ipv6_cidr_blocks = []
prefix_list_ids = []
protocol = "tcp"
security_groups = []
self = false
to_port = 443
},
]
id = "sg-05af4ed76c86e2b2b"
ingress = [
{
cidr_blocks = [
"10.0.0.0/8",
]
description = ""
from_port = 22
ipv6_cidr_blocks = []
prefix_list_ids = []
protocol = "tcp"
security_groups = []
self = false
to_port = 22
},
]
name = "tf-import-sg"
owner_id = "308825204162"
tags = {
"Name" = "tf-import-sg"
}
tags_all = {
"Name" = "tf-import-sg"
}
vpc_id = "vpc-0be2f04ae84a21152"
}
Plan: 2 to import, 0 to add, 0 to change, 0 to destroy.
╷
│ Warning: Config generation is experimental
│
│ Generating configuration during import is currently experimental, and the generated configuration format may change
│ in future versions.
╵
Step 3: (Optional) Review and edit the generated code
You can review the generated configuration code and edit it as necessary before proceeding to import.
wkhoo-macbook:tf-import $ cat generated.tf
# __generated__ by Terraform
# Please review these resources and move them into your main configuration files.
# __generated__ by Terraform from "sg-05af4ed76c86e2b2b"
resource "aws_security_group" "tf_import" {
description = "Demo of security group Terraform import"
egress = [{
cidr_blocks = ["0.0.0.0/0"]
description = ""
from_port = 443
ipv6_cidr_blocks = []
prefix_list_ids = []
protocol = "tcp"
security_groups = []
self = false
to_port = 443
}]
ingress = [{
cidr_blocks = ["10.0.0.0/8"]
description = ""
from_port = 22
ipv6_cidr_blocks = []
prefix_list_ids = []
protocol = "tcp"
security_groups = []
self = false
to_port = 22
}]
name = "tf-import-sg"
name_prefix = null
revoke_rules_on_delete = null
tags = {
Name = "tf-import-sg"
}
tags_all = {
Name = "tf-import-sg"
}
vpc_id = "vpc-0be2f04ae84a21152"
}
# __generated__ by Terraform from "tf-import-s3"
resource "aws_s3_bucket" "tf_import" {
bucket = "tf-import-s3"
bucket_prefix = null
force_destroy = null
object_lock_enabled = false
tags = {}
tags_all = {}
}
Step 4: Import the resources
Run the terraform apply command to import the resources.
terraform apply
The terraform apply output:
wkhoo-macbook:tf-import $ terraform apply -auto-approve
aws_s3_bucket.tf_import: Preparing import... [id=tf-import-s3]
aws_security_group.tf_import: Preparing import... [id=sg-05af4ed76c86e2b2b]
aws_security_group.tf_import: Refreshing state... [id=sg-05af4ed76c86e2b2b]
aws_s3_bucket.tf_import: Refreshing state... [id=tf-import-s3]
Terraform will perform the following actions:
# aws_s3_bucket.tf_import will be imported
resource "aws_s3_bucket" "tf_import" {
arn = "arn:aws:s3:::tf-import-s3"
bucket = "tf-import-s3"
bucket_domain_name = "tf-import-s3.s3.amazonaws.com"
bucket_regional_domain_name = "tf-import-s3.s3.ap-southeast-2.amazonaws.com"
hosted_zone_id = "Z1WCIGYICN2BYD"
id = "tf-import-s3"
object_lock_enabled = false
region = "ap-southeast-2"
request_payer = "BucketOwner"
tags = {}
tags_all = {}
grant {
id = "754c7d875755bfb329f9bd5cf61f8eacb3db030406e6765caf8e869da5036366"
permissions = [
"FULL_CONTROL",
]
type = "CanonicalUser"
}
server_side_encryption_configuration {
rule {
bucket_key_enabled = true
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
versioning {
enabled = false
mfa_delete = false
}
}
# aws_security_group.tf_import will be imported
resource "aws_security_group" "tf_import" {
arn = "arn:aws:ec2:ap-southeast-2:308825204162:security-group/sg-05af4ed76c86e2b2b"
description = "Demo of security group Terraform import"
egress = [
{
cidr_blocks = [
"0.0.0.0/0",
]
description = ""
from_port = 443
ipv6_cidr_blocks = []
prefix_list_ids = []
protocol = "tcp"
security_groups = []
self = false
to_port = 443
},
]
id = "sg-05af4ed76c86e2b2b"
ingress = [
{
cidr_blocks = [
"10.0.0.0/8",
]
description = ""
from_port = 22
ipv6_cidr_blocks = []
prefix_list_ids = []
protocol = "tcp"
security_groups = []
self = false
to_port = 22
},
]
name = "tf-import-sg"
owner_id = "308825204162"
tags = {
"Name" = "tf-import-sg"
}
tags_all = {
"Name" = "tf-import-sg"
}
vpc_id = "vpc-0be2f04ae84a21152"
}
Plan: 2 to import, 0 to add, 0 to change, 0 to destroy.
aws_security_group.tf_import: Importing... [id=sg-05af4ed76c86e2b2b]
aws_security_group.tf_import: Import complete [id=sg-05af4ed76c86e2b2b]
aws_s3_bucket.tf_import: Importing... [id=tf-import-s3]
aws_s3_bucket.tf_import: Import complete [id=tf-import-s3]
Apply complete! Resources: 2 imported, 0 added, 0 changed, 0 destroyed.
Step 5: (Optional) Verify the import
Verify the resources are imported successfully into Terraform state.
wkhoo-macbook:tf-import $ terraform state list
aws_s3_bucket.tf_import
aws_security_group.tf_import
Success! The S3 bucket and security group were imported successfully and now managed by Terraform.
With a few simple steps, I imported the resources into Terraform state file easily. This is just a simple demo with 2 simple AWS resources but the steps are the same for any other importable resources. You will still need to craft the import configuration file, which takes less time compared to writing the resource configuration code.
Clean up
As the S3 bucket and security group have been imported and managed by Terraform, you can run terraform destroy to delete the resources.
Some considerations
- The import block is only available in Terraform v1.5.0 and later version. You will get the following error if you try with an older version of Terraform:
╷
│ Error: Unsupported block type
│
│ on imports.tf line 12:
│ 12: import {
│
│ Blocks of type "import" are not expected here.
- Terraform import block currently does not support interpolation in the ID field. It must be a string value. So you can’t do something like this:
locals {
bucket_name = "tf-import-s3"
}
import {
to = module.s3.aws_s3_bucket.tf_import
id = local.bucket_name
}
- You can only define the import block in the root (parent) module. If you define the import block inside a module, you will get this error:
╷
│ Error: Invalid import configuration
│
│ on s3/main.tf line 2:
│ 2: import {
│
│ An import block was detected in "module.s3". Import blocks are only allowed in the root module.
╵
There is an open
GitHub issue
for the feature request. You can still use the terraform import CLI to import a resource into a Terraform module.
- Lastly, you would still take a backup of your Terraform state file just in case before importing so that you can roll back to the previous state quickly.
Summary
In this post, I talked about an overview of the Terraform import feature which you can use to import existing cloud infrastructure resources under Terraform IaC management. I have demonstrated how you can import an S3 bucket and a security group easily into Terraform state using the Terraform config-driven import block. I also highlighted some considerations when using the Terraform import block feature.
The terraform import command can import existing cloud resource but will not generate the imported resource configuration. You will need to write the code. The Terraform import block will import the existing resources into Terraform state and provide the capability to generate the resource configuration, which is very handy in my view even though it is an experimental feature.