Problem statement

Do you have application servers reside in private subnets, or behind layers of firewall and NAT gateway that you need to get remote access to perform application troubleshooting or debugging connectivity issues?

Do you need a method to gain access to your servers, whether to provide temporary remote access to third-party vendors or when migrating servers to the cloud, you want to connect to your Amazon EC2 instances to perform some operational or validation tasks?

Traditionally, you would use a bastion host or jump host as the industry best practice. A bastion host is a special purpose EC2 instance designed to be the primary access point from the Internet and acts as a proxy to your EC2 instances in private subnets. This helps to reduce the surface of attack by not exposing your application servers to the Internet directly and reduce the blast radius should there be a security breach on your bastion host.

Bastion hosts come with operational burden (eg. patching, access management) and the additional EC2 running costs when you factor in the high availability and redundancy functionality. There is an alternative if you want to avoid using bastion hosts to eliminate the operation burden and the incurred operating costs.

AWS Systems Manager Session Manager

AWS Systems Manager (SSM) Session Manager allows you to connect to your EC2 instances securely without the need of bastion hosts. You don’t even need to open an inbound SSH or RDP port. When SSM agent is installed on your EC2 instance and able to communicate to the AWS SSM service, you can either use the AWS Console UI or the AWS Command Line Interface (AWS CLI) to connect to your Linux or Windows instance.

The Session Manager uses the SSM document or runbooks to perform pre-defined actions on the managed instances. There are currently over 100 pre-configured documents that you can use out-of-the-box by specifying parameters at runtime. A subset of these documents are Session documents that determine which type of session to start, such as a session to run interactive command, a session to create an SSH tunnel, or a port forwarding session.

In this blog post, I’m going to talk specifically about Session Manager port forwarding and demonstrate how you can use the feature. And in case you don’t already know, Session Manager service is available at no additional cost. 👍

Demo environment setup

Beforehand, let me walk through the demo environment that I used in my sandbox account. If you want to try this on your own, you can check out the Terraform code in my GitHub repo.

Important note: Deploying the demo environment will incur some cost in your AWS account even if you’re on free tier because of the SSM endpoints and data transfer charges.

This is the high level diagram of my architecture:

The demo environment was made up of the following resources:

• A VPC with private subnets

  I created an isolated VPC with private subnets, without an Internet gateway or NAT gateway. The EC2 instances connect to the AWS SSM service using the private interface VPC endpoints. The VPC attributes Enable DNS hostnames and Enable DNS support are pre-requisite to enable private DNS for the endpoints.

• Interface VPC endpoints for SSM

  As the VPC does not have Internet connectivity, we need to create the SSM interface VPC endpoints in the private subnet to use AWS PrivateLink connectivity to the SSM service. Private DNS on the endpoints is required, which is enabled by default in the endpoints creation.

  Endpoint	Description
  com.amazonaws.region.ssm	The endpoint for the Systems Manager service.
  com.amazonaws.region.ec2messages	Systems Manager uses this endpoint to make calls from the Systems Manager agent to the Systems Manager service.
  com.amazonaws.region.ssmmessages	This endpoint is required for connecting to your instances through a secure data channel using Session Manager, in this case our port forwarding requirement.

  Note: the EC2 interface VPC endpoint com.amazonaws.region.ec2 is not required because we’re not using SSM to create VSS-enabled snapshots.

• Two EC2 instances (Linux and Windows) and one RDS MySQL instance

  An Amazon Linux 2 EC2 instance, a Windows 2022 EC2 instance, and an RDS MySQL instance were launched to simulate application workloads inside the private subnet. Both the Amazon Linux 2 and Windows 2022 Base AMI have the SSM agent pre-installed by default.

• IAM role and EC2 instance profile

  An IAM role with the AWS managed IAM policy AmazonSSMManagedInstanceCore created and associated to an EC2 instance profile. The managed IAM policy has the required permissions to call the AWS SSM API to establish successful connection to the SSM service.

• Security groups

  The following security groups were created and attached to the EC2 instances, RDS instance, and interface VPC endpoints respectively.

  Security Group name	Ingress	Egress
  ssm-demo-ec2	no rule	tcp 3306
  ssm-demo-rds	tcp 3306	no rule
  ssm-demo-endpoint	tcp 443	no rule

Port forwarding for AWS SSM Session Manager

The AWS SSM Session Manager port forwarding feature allows you to create secure (encrypted) tunnel between your EC2 instances deployed in private subnet without the need to open the SSH or RDP port in the security group.

If you have used SSH port forwarding before, it is a similar concept with the exception that instead of going over the SSH tunnel, the connection is established between AWS SSM service and the SSM agent running on the managed instance. The session is encrypted by default using TLS 1.2 and you can further encrypt the data in transit using an AWS KMS key in the Session Manager Preferences.

The port forwarding tunnel allows you to forward traffic between your local machine (workstation) to the network port inside your EC2 instance. Once the tunnel is established, you can connect to the local port using localhost from your workstation and the traffic will go through the tunnel to the port on the remote server.

There is no special networking setup required on the local machine and the EC2 instances. The networking traffic, mapping and routing are handled by AWS internally. As long as you have the SSM agent configured and running on the instance, and your AWS credentials have the required permissions, you can run simple AWS CLI command locally to start the Session Manager session and specify the SSM document AWS-StartPortForwardingSession to set up the port forwarding tunnel.

Here is an example of the AWS CLI command to start the port forwarding session:

aws ssm start-session \
    --target instance-id \
    --document-name AWS-StartPortForwardingSession \
    --parameters '{"portNumber":["443"], "localPortNumber":["10443"]}'

You need to specify the EC2 instance ID (server) that you want to connect, ie. to establish the port forwarding session, and the SSM document name AWS-StartPortForwardingSession with the parameters portNumber (the listening network port on the server) and localPortNumber (your local machine network port).

Pre-requisites

This feature was announced in August 2019. You need SSM agent version 2.3.672.0 or later installed on the managed node. The AWS CLI version 1.16.12 or later must be installed on your local machine and you also need to install the Session Manager plugin version 1.1.26.0 or later extension for the AWS CLI.

Demo Linux SSH port forwarding

The first demo that I want to show you is how I can run SSH connection into an Amazon Linux EC2 instance in a private subnet. The security group attached to the EC2 instance has no ingress rule for port 22 but I’m able to connect to the SSH service running on the Linux instance.

This simple network diagram depicts what I’m trying to achieve:

Step 1: Set AWS credentials

To be able to start the Session Manager session, I need an AWS user credentials with the required IAM permissions. For this demo, I simply going to copy and paste my AWS access keys from IAM Identity Center (SSO) into my terminal.

Step 2: Obtain the EC2 instance ID

I need the EC2 instance ID as the target parameter to establish the Session Manager port forwarding session. I can use the AWS CLI command to get the EC2 instance ID:

INSTANCE_ID=$(aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=ssm-demo-linux" \
  --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text)

Step 3: Establish the SSM session

Run the AWS CLI command to start the SSM session:

aws ssm start-session --target $INSTANCE_ID \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["22"],"localPortNumber":["8022"]}'

The port forwarding session is connected successfully and waiting for connections.

Starting session with SessionId: william.khoo@abcde.onmicrosoft.com-bsuhv5pqxoz6vveuhw354wt2ke
Port 8022 opened for sessionId william.khoo@abcde.onmicrosoft.com-bsuhv5pqxoz6vveuhw354wt2ke.
Waiting for connections...

Step 4: Verify the connection

Note: In the Amazon Linux 2 EC2 instance bootstrap, I created a local user ssm-demo and set the user password, and also enabled SSH password login (PasswordAuthentication yes) in the /etc/ssh/sshd_config.

Run this shell command to create a SSH connection over the port forwarding tunnel.

ssh -p 8022 -o "StrictHostKeyChecking no" ssm-demo@localhost

I set the port as 8022 (the port forwarding tunnel port) instead of standard SSH port 22 and specify localhost as the destination target.

$ ssh -p 8022 -o "StrictHostKeyChecking no" ssm-demo@localhost
Warning: Permanently added '[localhost]:8022' (ED25519) to the list of known hosts.
ssm-demo@localhost's password:
   ,     #_
   ~\_  ####_        Amazon Linux 2
  ~~  \_#####\
  ~~     \###|       AL2 End of Life is 2025-06-30.
  ~~       \#/ ___
   ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
      ~~._.   _/
         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

[ssm-demo@ip-10-0-1-221 ~]$

I’m logged on successfully into the Amazon Linux 2 instance.

Demo Windows RDP port forwarding

For the second demo, I will connect to a Windows EC2 instance using a RDP connection. Like the Linux instance, the security group attached to the Windows instance has no ingress rule as well.

Note: I’m still using the AWS credentials from previous demo, which are still valid, so I don’t need to refresh the access keys.

Step 1: Obtain the EC2 instance ID

Grabbing the Windows EC2 instance ID using the AWS CLI command:

INSTANCE_ID=$(aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=ssm-demo-windows" \
  --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text)

Step 2: Establish the SSM session

Set up RDP forwarding session by running this command:

aws ssm start-session --target $INSTANCE_ID \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["3389"],"localPortNumber":["13389"]}'

Again, I get the successful connection.

Starting session with SessionId: william.khoo@abcde.onmicrosoft.com-xnrgaafguqaxqc742tyat25iky
Port 13389 opened for sessionId william.khoo@abcde.onmicrosoft.com-xnrgaafguqaxqc742tyat25iky.
Waiting for connections...

Step 3: Verify the connection

I use a RDP client to connect to localhost at port 13389. In this demo, I used Remote Desktop for Mac.

I’m prompted to enter user account credentials. I login as local user Administrator using the password set during the Windows EC2 bootstrapping.

Note: I get the warning prompt about the RDP certificate root CA verification, I simply click Continue.

And, I’m logged in successfully onto the Windows desktop.

Remote host port forwarding for AWS SSM Session Manager

In May 2022, AWS Systems Manager announced the support for forwarding connections from a client machine to ports on remote hosts using Session Manager.

With remote port forwarding, you can actually use a managed instance as a jump host to securely connect to application port on another remote server, such as an RDS database or a web server on another EC2 instance, without exposing those servers to public network. The remote server does not need to be managed by SSM.

You will need SSM Agent version 3.1.1374.0 or later installed on the managed instances that you are establishing a port forwarding session with. You can start a port forwarding session from the command line using the AWS-StartPortForwardingSessionToRemoteHost Session Manager document.

An example of the AWS CLI command to start a remote host port forwarding session:

aws ssm start-session \
    --target instance-id \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["mydb.example.us-east-2.rds.amazonaws.com"],"portNumber":["3306"], "localPortNumber":["3306"]}'

The host parameter value represents the hostname or IP address of the remote host that you want to connect to. The portNumber is the port on the remote host and localPortNumber is the local port on your local machine.

However, the general network connectivity requirements (routing, NACL, security group, DNS resolution) between the managed node and the remote host still apply.

Demo Remote host RDS (MySQL) port forwarding

In this demo, I will demonstrate using an Linux EC2 instance (it can be a Windows instance too) to connect to a remote host, which is an RDS instance MySQL database, as shown in this diagram:

In the Linux EC2 security group, an inbound rule is not required however an outbound rule for port 3306 is necessary to enable network connectivity between the managed EC2 instance and the remote host RDS instance. The RDS instance security group needs an inbound rule for port 3306.

Note: I’m still using the AWS credentials from previous demo, which are still valid, so I don’t need to refresh the access keys.

Step 1: Obtain the EC2 instance ID

Grab the Linux EC2 instance ID using the AWS CLI command:

INSTANCE_ID=$(aws ec2 describe-instances \
  --filter "Name=tag:Name,Values=ssm-demo-linux" \
  --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text)  

Step 2: Get the RDS instance host address

I also need the RDS instance host address. I can get the endpoint address with this AWS CLI command:

RDS_INSTANCE_ADDR=$(aws rds describe-db-instances \
  --db-instance-identifier ssm-demo-mysql \
  --query "DBInstances[].Endpoint.Address" --output text)

Step 3: Establish the SSM session

Set up remote host port forwarding session by running this command:

aws ssm start-session --target $INSTANCE_ID \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters "{\"portNumber\":[\"3306\"],\"localPortNumber\":[\"13306\"],\"host\":[\"$RDS_INSTANCE_ADDR\"]}"

The port forwarding tunnel is waiting for connections.

Starting session with SessionId: william.khoo@abcde.onmicrosoft.com-wwyiwey3kglekv4g2kubektyi4
Port 13306 opened for sessionId william.khoo@abcde.onmicrosoft.com-wwyiwey3kglekv4g2kubektyi4.
Waiting for connections...

Step 4: Verify the connection

I have MySQL client installed on my local machine so I can verify the forwarding connection is working by running the following command:

mysql -h 127.0.0.1 --port 13306 -u admin -pPassword123

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 63
Server version: 8.0.35 Source distribution

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| demodb             |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.18 sec)

mysql> use demodb;
Database changed
mysql> show tables;
Empty set (0.04 sec)

mysql> create table demo (FirstName varchar(255), LastName varchar(255));
Query OK, 0 rows affected (0.16 sec)

mysql> show tables;
+------------------+
| Tables_in_demodb |
+------------------+
| demo             |
+------------------+
1 row in set (0.05 sec)

mysql> quit
Bye

Security Considerations

From the demos that I have shown you, it is very easy to use the SSM Session Manager port forwarding feature and it does not require much configuration if you are using AWS SSM to manage your EC2 instances. It can be a risk if you use it out-of-the-box without considering the security aspects and access control.

As long as the SSM agent is installed and running on the EC2 instances and connected to the AWS SSM service, you can connect to any EC2 instance in your account. If you give a user or role generic access to SSM, they can open up port forwarding tunnel to any SSM-connected instance and possibly run whatever commands they want on that instance.

If you want to tighten the security around this, you will need to restrict the Session Manager through IAM policies on the SSM API access and the port forwarding SSM Document to control who in your organisation is authorised to create the tunnels.

Conclusion

In this post, I walked you through how to use the Session Manager port forwarding feature to access server application on EC2 instances running in private subnets. I have demonstrated that the port forwarding works for Linux and Windows EC2 instances, and you could also do port forwarding to a remote host using a managed instance as a jump host. I also talked briefly about security considerations when using SSM port forwarding.

There is no additional cost when connecting to Amazon EC2 instances using AWS SSM Session Manager however you will be charged for the egress traffic from the NAT Gateway or interface VPC endpoints.

With the port forwarding remote host feature, you could potentially place your bastion host inside a private subnet so that you don’t have to expose your bastion host directly to the public Internet. This is probably not an enterprise grade solution but food for thought.