Solving PAM in hybrid environment using multi-hop sessions with HCP Boundary

In this blog learn how to solve Privileged Access Management (PAM) problems using multi-hop sessions with HashiCorp Boundary.

Background

• Is your organization using HCP Boundary as a PAM (privileged access management) solution for hosts residing in private networks across multiple cloud providers?
• Does your organization have strict networking constraints, such as resources residing in private networks having only outbound access?
• Does your organization require using your own Boundary workers instead of HCP workers?

In this article, I am going to highlight and demo the solution using multi-hop sessions with HCP Boundary and show you how you can access resources in private networks across Azure and AWS.

What we'll be seeing in this article/demo?

• Working with self managed PKI ingress workers (Installation & Registration)
• Creating worker-aware targets
• Connecting to targets using Boundary Desktop Application
• Challenges with ingress workers
• Introducing multi-hop sessions with HCP Boundary
• Working demo of multi-hop sessions

Working with self managed workers

Self-managed workers allow Boundary users to securely connect to private resources without exposing their organizations' networks to the public. Self-managed workers use public key infrastructure (PKI) for authentication. PKI workers authenticate to Boundary using a certificate-based method

Networking requirements

1. Outbound access to HCP Boundary Control Plane (worker -> HCP control plane)

2. Outbound access to the private target (worker -> host)

3. Inbound access from user trying to establish the session (user -> worker on port 9202)

Installation of Ingress worker

Code can be found here

Assuming you have a running HCP Boundary cluster, follow below steps:

1. Create Azure VM and AWS EC2 linux machines with just private IP (no public IPs associated).

2. Create Azure and AWS Ingress workers in the respective cloud providers using below steps:

• Install boundary-worker utility on both workers

• Add pki-worker.hcl on both workers at path /home/username/boundary. Replace value of hcp_boundary_cluster_id with actual value.

• Run boundary worker using above config file.

• Copy the Worker Auth Registration Request field in the output of above command which will be used next for registering this worker to HCP Boundary control plane.

Registration of Ingress Worker

1. Go to HCP Boundary cluster URL and authenticate with admin credentials.

2. Once logged in, navigate to the Workers page.

3. Click New and scroll down to the bottom of the New PKI Worker page and paste the Worker Auth Registration Request key you copied earlier.

4. Click Register Worker.

5. Follow the above steps for next Ingress worker.

Awesome ! You now have two self-managed PKI Ingress workers (one for Azure and one for AWS) registered in HCP Boundary cluster. But how do you tell which ingress worker to use while connecting Azure or AWS target? Let's see this now.

Creating worker-aware targets

1. Open the HCP Boundary Admin Console UI and create a organization called "multi-hop" followed by a project called "multi-hop".

2. Navigate to the targets page within the multi-hop project.

3. Create 2 targets (one for Azure and one for AWS) with following details:

• Target name (aws-target and azure-target)
• Target address (private IP of AWS EC2 and Azure VM respectively)
• Default port 22
• Toggle the Ingress worker filter switch to add an ingress worker filter that searches for workers that have the following tag:

Connecting to targets using Boundary Desktop Application

With the ingress filters assigned above, any connections to target will be forced to proxy through the assigned ingress worker.

1. The end user authenticates to HCP Boundary cluster using Boundary Desktop application.

2. The user navigates to Targets section and will the authorized targets based on his roles and permissions.

3. The user connects to the aws/azure target using "Connect" button.

4. A proxy URL is published which user uses to establish a session

Challenges with Ingress workers

• The Boundary ingress workers have to be accessed by user/clients which means either those users need to have some VPN access or the ingress worker is publicly accessible using a public IP. Some organisations may not allow this.
• Other organizations might give access to boundary ingress workers residing in public network but resources are in private network with only outbound access, hence boundary ingress worker won't be able to proxy the connection.

Managing multi-hop sessions with HCP Boundary

In order to mitigate the above networking challenges with some organisations, 0.12 release of HCP Boundary introduced a feature of multi-hop sessions wherein now you can configure different types of Boundary workers based on their networking requirements and eventually create a chain of workers through reverse-proxy connections. This creates multiple "hops" in the chain from a worker to a controller.

• The new type of worker which is introduced as part of this feature is called an "Egress" worker.
• This worker has network access into the hosts and connects to an existing upstream/ingress worker which is accessible to the clients.
• The "Egress" worker can reside in a private network with only outbound access and inaccessible to the clients.

Installation of Egress Worker

1. Create 2 Egress workers (one in each cloud) with private IP's and only outbound access.

2. Install same boundary-worker utility similar to Ingress worker.

3. Add pki-worker.hcl on both workers at path /home/username/boundary. Note initial_upstreams parameter which tells which ingress worker to connect to.

• Run boundary worker using above config file.

• Copy the Worker Auth Registration Request field in the output of above command which will be used next for registering this worker to HCP Boundary control plane.

Registration of Egress Worker

Follow same steps as ingress worker for registering the egress worker.

Modifying worker-aware targets

1. Open the Boundary Admin Console UI navigate to the targets page within the "multi-hop" org.

2. Click on the aws-target/azure-target. Scroll to the bottom the the target details page and click Edit Form.

3. Toggle the Egress worker filter switch to add an egress worker filter that searches for workers that have the following tag:

Refresh the targets in Boundary Desktop Application and you should be able to connect the target again.

Working Demo

Conclusion

Awesome ! You have now connected to a target residing in a private network with only outbound access by creating a chain of "ingress" and "egress" workers and establishing a "hop" in the session.

See Zero Trust Security in action !

If you are new to Hashicorp Boundary and would like to understand how Boundary-Vault integration helps us in achieving Zero Trust Security, you can watch my HashiTalk where I explain the traditional workflow of privileged access management (PAM), its challenges and how we solved couple of PAM use-cases for Windows and Linux servers.