Migrating Network Traffic from Transit Gateway to AWS Cloud WAN

In this Cloud WAN article series, we have so far covered the service fundamentals, what its logical components are, and how to create and configure our global network based on policies.

We have also seen how to add inspection capabilities, both for east-west traffic and north-south traffic, simply by modifying the policy to redirect traffic without having to use static routes.

The reality for many organizations regarding networking is not finding ourselves with a green field where we can create a network from scratch. In multi-region environments, it’s common to find a network composed of several transit gateways interconnected by peering.

For this reason, in this post we will see a migration strategy from complex multi-region networks, based on transit gateway, towards a single global network managed with AWS Cloud WAN.

From Transit Gateway to Cloud WAN

#

To migrate from a Transit Gateway-based network to a Cloud WAN-based network, a two-phase strategy can be proposed. A first phase where we create the Cloud WAN Core Network, define the segments and federate Transit Gateway so that traffic flows between regions through Cloud WAN instead of through peerings between Transit Gateways. And a second phase where we change the VPC attachments to connect them to Cloud WAN instead of Transit Gateway. In this last phase, Transit Gateway resources are also deprovisioned.

To better illustrate the process, we will look at three scenarios that reflect each moment of the migration.

1. Starting situation: Transit Gateway in each region connected by peering.

2. Phase 1: Federation with Cloud WAN. This phase is divided into two sub-phases:

   1. Phase 1a: Deployment of Cloud WAN Core Network and Transit Gateway federation.
   2. Phase 1b: Elimination of peering between Transit Gateways

3. Phase 2: Transit Gateway substitution. This phase is divided into three sub-phases:

   1. Phase 2a: VPC attachment to Cloud WAN
   2. Phase 2b: Changing VPC routing towards Cloud WAN
   3. Phase 2c: Elimination of Transit Gateway infrastructure.

To facilitate understanding of the migration process, a repository structured in different branches is offered with the Terraform configuration for each of the phases. This way, if you wish, you can deploy the configuration of each phase in order or, if you prefer, deploy the branch corresponding to the starting situation and apply the changes, either by editing the Terraform code or directly from the AWS console.

Before starting

#

If you finally choose to follow the procedure, which I highly recommend, there are a couple of things you need to consider. The first and most important:

The second are the instructions for cloning and initializing Terraform.

# Clone the repository
git clone https://codeberg.org/scambelo/aws-cloud-wan-migration-demo.git
cd aws-cloud-wan-migration-demo

# Initialize Terraform (only needed once)
git checkout phase-0-starting-situation
terraform init

Starting situation

#

git checkout phase-0-starting-situation
terraform plan
terraform apply

We start with a Transit Gateway-based network. Both Transit Gateways are connected by peering that establishes connectivity between the two regions.

This network is segmented into two layers: one layer for production VPCs and another layer for development VPCs. This separation is achieved through a route table for each layer. Additionally, there is a third route table that handles traffic arriving through peering.

It should be noted that in this configuration, the routes that direct traffic through peering are static.

Assuming we have two EC2 instances, one in VPC A and another in VPC C, both belonging to the production segment, traffic from VPC A to VPC C flows as follows:

1. Outbound traffic from VPC A reaches the Transit Gateway and, according to the PROD 1 route table, the next hop corresponds to Transit Gateway Peering.
2. Traffic passes from the Transit Gateway in the primary region to the secondary through Transit Gateway Peering.
3. Finally, traffic exits the Transit Gateway in the secondary region to go to VPC C, as indicated in the PEERING 2 route table.

The return traffic is completely symmetric and returns by the same path, taking into account that it passes through the PROD 2 and PEERING 1 route tables to reach its destination.

Phase 1: Federation with Cloud WAN

#

Phase 1a: Cloud WAN Core Network deployment and Transit Gateway federation

#

git checkout phase-1a-tgw-federation
terraform plan
terraform apply

The next step is to deploy Cloud WAN. To do this, we create a Core Network and define the dev and prod segments in our policy. In addition to defining the Core Network Policy, we create peering between the Core Network and Transit Gateway in each region.

Finally, we need to attach the Transit Gateway route tables to the Core Network and also associate them with their corresponding segment. This configuration requires three components:

• Transit Gateway Policy Table: this component allows Cloud WAN to propagate routes in Transit Gateway route tables. This component is unique per Transit Gateway.
• Transit Gateway Policy Table Association: this is the relationship of the policy table with the peering between the Core Network and Transit Gateway. This component is unique per Transit Gateway.
• Transit Gateway Route Table Attachment: this is the route table attachment of Transit Gateway itself. Conceptually, it is very similar to a VPC attachment. The association of each attachment with its corresponding segment is done through tags. This component is unique for each route table of Transit Gateway that we want to associate.

For now, nothing has changed at the routing level. Traffic continues to flow in the same way as in the previous phase, but we have already laid the foundations for the next step of this phase.

Phase 1b: Elimination of peering between Transit Gateways

#

git checkout phase-1b-remove-peering
terraform plan
terraform apply

In the second part of this phase, we eliminate the peering between the Transit Gateways in each region and the associated route tables. We also eliminate the static routes from the Transit Gateway route tables. These route tables are dynamically updated by the Transit Gateway Policy Tables that we created in the previous step.

With these changes in infrastructure and routing, inter-region traffic now routes through Cloud WAN.

1. Following the previous example, traffic continues to exit from VPC A to the PROD 1 route table in Transit Gateway.
2. From Transit Gateway it continues its path to Cloud WAN.
3. In Cloud WAN, since the Transit Gateway PROD 1 route table is associated with the prod segment, the routing of the Core Network internal route table corresponding to that segment and region is applied, directing traffic towards the destination region.
4. In the destination region, the internal route table redirects traffic to the Transit Gateway PROD 2 route table.
5. Finally, Transit Gateway redirects traffic to its final destination in VPC C.

The return traffic in this case is completely symmetric, following the reverse path to the origin.

At this point in phase 1, we already have a fully functional inter-region interconnection solution, so we could finish the migration at this point. However, we still maintain Transit Gateway infrastructure that can represent an additional cost.

In the next phase, we will see how to complete the migration to not have this dependency on Transit Gateway.

Phase 2: Transit Gateway substitution

#

Phase 2a: VPC attachment to Cloud WAN

#

git checkout phase-2a-vpc-attachment
terraform plan
terraform apply

In the first step of this second phase, we attach VPCs directly to Cloud WAN while still maintaining the current attachments with Transit Gateway. Each VPC is associated with its corresponding segment thanks to tags, which allows us not to have to touch the Cloud WAN policy. In fact, during this entire migration, the Core Network Policy is not touched.

The new attachments to Cloud WAN dynamically modify its internal routes and now, instead of redirecting outbound traffic to Transit Gateway route tables, it redirects it directly to VPCs. Let’s see how the flow looks:

1. Traffic continues to exit from VPC A to the PROD 1 route table in Transit Gateway.
2. From Transit Gateway it continues its path to Cloud WAN.
3. In Cloud WAN, the routing of the Core Network internal route table corresponding to the segment and region of the VPC A attachment is applied and traffic is directed towards the destination region.
4. In the destination region, the internal route table redirects traffic to its final destination in VPC C.

The return traffic follows an analogous path. At this point, we have flow asymmetry, since outbound traffic continues to exit through Transit Gateway but return traffic enters the VPC from Cloud WAN. But this will change in the next step.

Phase 2b: Changing VPC routing towards Cloud WAN

#

git checkout phase-2b-routing-change
terraform plan
terraform apply

Now, in this step, once we have the VPCs connected to Cloud WAN, what we do is change the routing in the VPCs themselves so that their default route (0.0.0.0/0) points to Cloud WAN instead of Transit Gateway.

With this change, the flow is as follows:

1. Traffic exits from VPC A to Cloud WAN directly.
2. In Cloud WAN, the routing of the Core Network internal route table corresponding to the segment and region of the VPC A attachment is applied and traffic is directed towards the destination region.
3. In the destination region, the internal route table redirects traffic to its final destination in VPC C.

Therefore, now the return traffic follows the reverse path and we recover traffic symmetry. At this point, we have almost completed the migration, since all inter-VPC traffic is performed through Cloud WAN. We only have one thing left to do.

Phase 2c: Elimination of Transit Gateway infrastructure

#

git checkout phase-2c-tgw-cleanup
terraform plan
terraform apply

Finally, we only need to clean up the remaining resources from Transit Gateway that are no longer necessary. In this last step, there are no changes in the flow that was already defined in the previous step and we only do infrastructure cleanup that is no longer necessary, leaving the resulting architecture as shown in the diagram of this section.

Conclusion

#

In this article, we have explored a complete migration strategy from a Transit Gateway-based network architecture towards a modern solution with AWS Cloud WAN. The migration is structured in two main phases: the first focused on federation with Cloud WAN while maintaining existing infrastructure, and the second focused on complete Transit Gateway substitution. This gradual approach allows the transition to be carried out in a controlled and uninterrupted manner, minimizing operational risk.

During the process, we have seen how Cloud WAN introduces concepts such as policy-based dynamic routing and automatic route propagation, eliminating the need to manage static routes manually. Transit Gateway federation allows maintaining existing network segmentation while leveraging Core Network capabilities. The final result is an architecture where inter-regional traffic flows more directly and connectivity management is centralized in a single network policy.

And here ends this three-part monographic series on Cloud WAN.

Enjoyed this article? Subscribe to the newsletter and be the first to know when a new one is published!

References

#

• AWS Cloud WAN and AWS Transit Gateway migration and interoperability patterns
• Peerings in AWS Cloud WAN
• Transit gateway route table attachments in AWS Cloud WAN
• Transit gateway policy tables in Amazon VPC Transit Gateways