This is a spin-off from my previous blog post Centralised SSM VPC Endpoints Cost-Benefit Analysis where we explored how the centralised SSM VPC endpoints architecture could help organisations to minimise costs by creating the endpoints in one VPC and sharing the endpoints with the other VPCs, therefore reducing the number of endpoints required and the associated endpoint-hour charges.

In this blog post, I will illustrate how you can scale the centralised SSM VPC endpoints solution across AWS accounts.

Overview

The similar steps apply for sharing SSM VPC Endpoints across multiple accounts and across multiple VPCs within the same account. Let’s briefly go through the steps at a high level.

1. Create SSM VPC interface endpoints in the Shared Services account ¹

When an interface endpoint is created with the default setting PrivateDnsEnabled=true, AWS creates a Route 53 Private Hosted Zone (PHZ) automatically during the endpoint creation and associates the PHZ to the VPC the endpoint is created. This PHZ is managed by AWS so you don’t see the hosted zone in your AWS account and therefore you can’t modify and share it with other VPCs.

2. Create the Route 53 Private Hosted Zones ²

In order to share the PHZs outside of the VPC, you will need to set PrivateDnsEnabled=false when creating the VPC endpoint and then create your own self-managed PHZ matching the service endpoint name, eg. ssm.ap-southeast-2.amazonaws.com, with an Alias record of the service endpoint name pointing to the interface endpoint DNS, eg. vpce-0456c1b582cbee519-wsde53xb.ssm.ap-southeast-2.vpce.amazonaws.com.

3. Associate Private Hosted Zones across different accounts ³

Setting up the PHZ associations across VPCs within the same AWS account is straightforward (using Route 53 console). However, if the VPCs reside in different accounts, you need to do the following steps:

• In the Shared Services account, create the VPC association authorisations.
• In the spoke accounts, associate the VPC with the Route 53 PHZ.
• In the Shared Services account, delete the VPC association authorisations.

Note: there’s an account-level limit of 1000 VPC association authorisations that you can create so it’s best practice to delete the authorisations. Deleting the authorisations do not affect the already associated VPCs.

The PHZ associations can be done through the AWS CLI, AWS SDK, or Route 53 API. It is currently not possible to authorise the VPC association using the Route 53 console. There is a KB article in AWS re:Post on how to associate a Route 53 PHZ with a VPC on a different account using AWS CLI.

I will demonstrate how to associate a VPC on a different AWS account with PHZ using Terraform (AWS SDK).

Demo setup

This is the demo environment architecture overview:

For the demo, I scale the centralised VPC endpoint solution across multiple VPCs and accounts by leveraging AWS VPC Peering instead of AWS Transit Gateway for 2 reasons: (1) simplicity as I only have two VPCs and (2) cost savings because VPC Peering connection is free (no connection fee and data transfer cost within an AZ).

One VPC is created on each AWS account (Hub and Spoke) and inter-connected using VPC Peering. The CIDR network routing between the peered VPCs are configured in the private route table. The SSM VPC interface endpoints are created and associated in the VPC private subnet on the Hub account, and an Amazon Linux (AML) EC2 instance is created in the VPC private subnet on the Spoke account. A Route 53 PHZ for each interface endpoint is created in the Hub VPC and associated with the Spoke VPC. Both VPCs have no Internet gateway attachment so all network traffic will be private.

I will connect to the AML EC2 instance in the Spoke VPC using SSM Session Manager. This is possible because the SSM agent running in the EC2 instance is able to connect to the SSM service endpoint. The network routes through the VPC Peering connection to the Hub VPC and access the SSM VPC interface endpoints to get to the SSM service.

I have crafted the Terraform code to deploy the demo environment easily. The code is available in my GitHub repository.

Deploying the demo environment

As a pre-requisite, you need to create an IAM role in the Spoke account for Terraform to assume the role to create AWS resources before deploying the demo environment.

Step 1: Provision the pre-requisite Terraform IAM service role

Follow these quick simple steps to provision the IAM service role.

• Clone the terraform-service-role repository locally.

git clone https://github.com/wtkhoo/terraform-service-role.git

• Initialise Terraform to set up the providers.

cd terraform-service-role
terraform init

• Create a terraform.tfvars file to customise your deployment.

# Trusted AWS account id
account_id  = "123456789012"
# STS external id
external_id = "some-random-characters"

• Provision the Terraform resources.

terraform apply -auto-approve

Step 2: Provision the demo environment resources

You must clone the demo environment repository cross-account-vpce-sharing on the same top level directory as terraform-service-role because the Terraform state outputs (IAM role ARN and STS external ID) are referenced by the demo environment Terraform stack.

• Clone the cross-account-vpce-sharing repository locally.

git clone https://github.com/wtkhoo/cross-account-vpce-sharing.git

• Initialise Terraform to set up the providers.

cd cross-account-vpce-sharing
terraform init

• Create a terraform.tfvars file to customise your deployment.

# Prefix name for resources
name     = "ssm-demo"
# Hub VPC CIDR block
hub_vpc_cidr = "10.0.0.0/16"
# Spoke VPC CIDR block
spoke_vpc_cidr = "10.1.0.0/16"

• Provision the Terraform resources.

terraform apply -auto-approve

Terraform code constructs

Before starting the demo test, let’s walk through some key construct of the Terraform code with a brief explanation.

Create SSM VPC interface endpoints in the Hub account

The following Terraform code snippet create the SSM VPC interface endpoints - ssm, ssmmessages, and ec2messages - in the Hub VPC and attach to a private subnet. Note that the private_dns_enabled attribute is set to false on each endpoint.

resource "aws_vpc_endpoint" "ssm" {
  ip_address_type      = "ipv4"
  private_dns_enabled  = false
  security_group_ids   = [aws_security_group.vpce.id]
  service_name         = "com.amazonaws.ap-southeast-2.ssm"
  subnet_ids           = [aws_subnet.hub_private_a.id]
  vpc_endpoint_type    = "Interface"
  vpc_id               = aws_vpc.hub.id
  dns_options {
    dns_record_ip_type = "ipv4"
  }
  tags = {
    Name = "ssm-vpc-endpoint"
  }
}

resource "aws_vpc_endpoint" "ec2messages" {
  ip_address_type      = "ipv4"
  private_dns_enabled  = false
  security_group_ids   = [aws_security_group.vpce.id]
  service_name         = "com.amazonaws.ap-southeast-2.ec2messages"
  subnet_ids           = [aws_subnet.hub_private_a.id]
  vpc_endpoint_type    = "Interface"
  vpc_id               = aws_vpc.hub.id
  dns_options {
    dns_record_ip_type = "ipv4"
  }
  tags = {
    Name = "ssm-ec2messages-vpc-endpoint"
  }
}

resource "aws_vpc_endpoint" "ssmmessages" {
  ip_address_type      = "ipv4"
  private_dns_enabled  = false
  security_group_ids   = [aws_security_group.vpce.id]
  service_name         = "com.amazonaws.ap-southeast-2.ssmmessages"
  subnet_ids           = [aws_subnet.hub_private_a.id]
  vpc_endpoint_type    = "Interface"
  vpc_id               = aws_vpc.hub.id
  dns_options {
    dns_record_ip_type = "ipv4"
  }
  tags = {
    Name = "ssmmessages-vpc-endpoint"
  }
}

The Route 53 PHZ for each endpoint will not be created and associated to the VPC automatically during the endpoint creation process.

Create the Route 53 Private Hosted Zones manually

As the attribute PrivateDnsEnabled=false was set when creating the VPC endpoint, a Route 53 PHZ needs to be created manually for each endpoint in the Hub account. The Route 53 zone name must match the service endpoint name, eg. ssm.ap-southeast-2.amazonaws.com.

I created a local variable endpoints to map the SSM VPC endpoints attributes:

locals {
  endpoints = {
    ssm = {
      zone_name = join(".", (reverse(split(".", aws_vpc_endpoint.ssm.service_name))))
      dns_name = aws_vpc_endpoint.ssm.dns_entry[0]["dns_name"]
      hosted_zone_id = aws_vpc_endpoint.ssm.dns_entry[0]["hosted_zone_id"]
    }
    ssmmessages = {
      zone_name = join(".", (reverse(split(".", aws_vpc_endpoint.ssmmessages.service_name))))
      dns_name = aws_vpc_endpoint.ssmmessages.dns_entry[0]["dns_name"]
      hosted_zone_id = aws_vpc_endpoint.ssmmessages.dns_entry[0]["hosted_zone_id"]
    }
    ec2messages = {
      zone_name = join(".", (reverse(split(".", aws_vpc_endpoint.ec2messages.service_name))))
      dns_name = aws_vpc_endpoint.ec2messages.dns_entry[0]["dns_name"]
      hosted_zone_id = aws_vpc_endpoint.ec2messages.dns_entry[0]["hosted_zone_id"]
    }
  }
}

• zone_name - Derive the Route 53 PHZ name from reversing the endpoint service name, eg. com.amazonaws.ap-southeast-2.ssm to ssm.ap-southeast-2.amazonaws.com
• dns_name - Get interface endpoint DNS name, eg. vpce-0456c1b582cbee519-wsde53xb.ssm.ap-southeast-2.vpce.amazonaws.com
• hosted_zone_id - this is the hosted zone ID created for the interface endpoint DNS.

The Terraform code iterates the local variable endpoints, and creates a PHZ with matching name to the service endpoint name for each endpoint and an Alias record pointing to the interface endpoint DNS respectively.

resource "aws_route53_zone" "zone" {
  for_each = local.endpoints

  name     = each.value["zone_name"]

  vpc {
    vpc_id = aws_vpc.hub.id
  }

  lifecycle {
    ignore_changes = [vpc]
  }
}

resource "aws_route53_record" "root" {
  for_each = local.endpoints

  zone_id  = aws_route53_zone.zone[each.key].zone_id
  name     = aws_route53_zone.zone[each.key].name
  type     = "A"

  alias {
    name                   = each.value["dns_name"]
    zone_id                = each.value["hosted_zone_id"]
    evaluate_target_health = false
  }
}

Associate Route 53 Private Hosted Zones to the Spoke VPC

After the PHZs have been created, the VPC in the Spoke account must be authorised to associate the PHZ for each endpoint.

The following code creates the authorisations for setting up cross-account associations for each PHZ created in Hub account and the VPC in Spoke account:

resource "aws_route53_vpc_association_authorization" "zone" {
  for_each = local.endpoints

  vpc_id   = aws_vpc.spoke.id
  zone_id  = aws_route53_zone.zone[each.key].id
}

This code creates the association between each PHZ in Hub account and the VPC in Spoke account.

resource "aws_route53_zone_association" "zone" {
  provider = aws.spoke
  for_each = local.endpoints

  vpc_id   = aws_route53_vpc_association_authorization.zone[each.key].vpc_id
  zone_id  = aws_route53_vpc_association_authorization.zone[each.key].zone_id
}

Be aware that the VPC association authorisations are not deleted as Terraform maintains it in the state file. So if you’re creating more than 1000 authorisations using Terraform, you will hit the quota limit of 1000 which seems to be a hard limit.

Testing the setup

To verify whether the Spoke VPC is successfully associated with the PHZ, log on to the Hub account and check one of the Route 53 hosted zone details. Route 53 -> Hosted zones -> ssm.ap-southeast-2.amazonaws.com

There are 2 VPC IDs associated - one ID is the Hub VPC (default VPC association when the PHZ was created) and the other ID is the Spoke VPC (through the manual VPC association).

Let’s see if we can connect to the AML EC2 instance using SSM Session Manager. Log onto the Spoke account. Navigate to EC2 -> Instances. Select the EC2 instance ssm-demo-linux.

Click on Connect and select the Session Manager tab.

Looks like we are able to connect to Session Manager. Click Connect to start the session and run some DNS queries.

All the SSM endpoint service name queries resolve to private IP addresses. These private IP addresses are from CIDR 10.0.1.0/24 assigned to the SSM interface endpoints in the Hub VPC.

This testing simply verifies that the Spoke VPC can correctly resolve the endpoint service names through the VPC association with the PHZs in the Hub VPC and utilise the centralised SSM endpoints.

Clean up

To clean up the resources created to avoid unnecessary charges.

First, delete the demo environment Terraform stack.

cd cross-account-vpce-sharing
terraform destroy -auto-approve

Then, delete the Terraform IAM service role stack.

cd ../terraform-service-role
terraform destroy -auto-approve

Conclusion

I have demonstrated a simple architecture of sharing SSM interface VPC endpoint across multiple VPCs and accounts. This conceptual architecture can be applied for other VPC private interface endpoints. It is a testament of how you could build a secure and scalable multi-VPC AWS network infrastructure by utilising centralised VPC private endpoints and enabling cross-account VPC associations.

The AWS whitepaper Centralized Access to VPC Private Endpoints provides the insights for this blog post.