Network and edge

How traffic gets from the internet to a container, and how the network inside one region is laid out. Read this before you add a service, a load balancer, a WebSocket, or a VPC endpoint.

How it works

One region's VPC

Term	What it means
VPC	Your private network inside AWS.
Subnet	A slice of the VPC's addresses, in one AZ.
Public subnet	Has a route to the internet. Load balancers and NAT live here.
Private subnet	No inbound route from the internet. Containers live here.
Isolated subnet	No internet route at all, in or out. Databases live here.
Security group	A firewall attached to a resource. We chain them: internet to ALB, ALB to tasks, tasks to data.
NAT gateway	Lets private resources call out to the internet, but not be called.
VPC endpoint	A private door from your VPC to an AWS service, so the call never touches the internet.

Address plan

Give each environment its own /16 block (for example 10.0.0.0/16 dev, 10.1.0.0/16 staging, 10.2.0.0/16 prod). Inside it, reserve one slot per future region now, even if you only run one. Then networks never overlap when you connect regions later.

One ALB, many hostnames

A listener is the port the load balancer accepts traffic on. Rules on the listener pick a target group by the Host header. One ALB can serve many services this way, which saves money.

Mutual TLS is per listener, not per hostname

Mutual TLS (mTLS) means the browser must show a client certificate before the load balancer lets it in. We used it to keep pre-release web apps private to the team.

On an ALB, mTLS is a setting on the whole listener. Turn it on, and every hostname on that listener demands a certificate, including the API the mobile app uses. That is why browser apps got their own ALB.

• The ALB trusts the certificate authority (CA), not each person. Adding a person needs no AWS change, only a new signed certificate.
• Keep the CA private key out of Terraform state. Apply in stages: create the bucket, upload the CA bundle, then create the trust store.
• Cost, as of September 2026: a trust store costs a few dollars a month per load balancer. A managed client VPN for the same team was far more.

Long-lived WebSockets

The ALB closes any connection that sends no data for its idle timeout. The default is 60 seconds. A voice or chat session can be quiet longer than that.

• Raise the ALB idle timeout above your longest session. We used 600 seconds.
• Set the target group deregistration delay to at least a session length, so old tasks finish their sessions during a deploy instead of cutting them.
• Browsers cannot set an Authorization header on a WebSocket, so the token goes in the query string. Check it at upgrade time.
• Exempt the upgrade path from per-IP request rate limits, and rate-limit messages per connection instead.

Static files through CloudFront

• Static files go to S3, served by CloudFront. Containers never serve them.
• CloudFront price classes decide which edge locations you pay for. As of October 2026, the cheapest class (Price Class 100) covers the USA, Canada, Europe and Israel only. India and most of Asia start at Price Class 200. If your users are in Asia, you need the higher class.
• Set the bucket's allowed web origins (CORS) in Terraform, for every hostname that uploads.

WAF in front of public APIs

AWS WAF checks each request against rules (known attack patterns, rate limits) before it reaches the ALB's targets. Every rule can Count (just log) or Block.

How to build it

1. One VPC per environment per region, three subnet tiers across at least two AZs.
2. Gateway endpoints for S3 and DynamoDB always (they are free).
3. Interface endpoints in staging and prod, the same list in both.
4. One NAT per AZ in prod. One is fine in dev.
5. One API ALB per region, host-header rules per service. A second ALB only when a listener-level setting (like mTLS) must differ.
6. Open port 80 on the ALB security group if you have a port-80 redirect listener.
7. WAF on every public ALB. Start in Count, read the logs for a few days, then switch to Block.

NAT or VPC endpoints?

	NAT gateway	Interface endpoint	Gateway endpoint
What it reaches	Anything on the internet	One AWS service	S3 or DynamoDB only
Billed by	Hour plus data processed	Hour per AZ plus data processed	Free
Traffic path	Public internet	Private AWS network	Private AWS network

As of August 2026, in our main region, nine to ten interface endpoints across two AZs cost more than twice as much as the NAT gateways. NAT data charges for the same traffic were only a few dollars a month. So endpoints are not a cost saving at our size. We kept them anyway, on purpose: AWS API traffic stays off the public internet.

Rules

Rule

Never turn on mTLS on a listener that serves an API used by mobile apps or servers. mTLS applies to the whole listener. Give browser apps their own load balancer instead.

Rule

VPC endpoints are both-or-neither for staging and prod. Staging exists so prod does not surprise you. A different network path in staging tests nothing.

Rule

A WAF rule in Count mode protects nothing. Move it to Block once you have read what it would block.

Rule

Any service with long-lived connections needs a raised ALB idle timeout, set in Terraform on the main branch. A value set only on a feature branch is reverted by the next apply from main.

Mistakes we made

• mTLS on the shared listener would have locked out the mobile app. The one dev listener carried the APIs, discovery, admin and every website. Turning on mTLS there would have demanded certificates from phones. A second listener on a non-standard port was free, but sign-in redirect addresses on odd ports are rejected or awkward. Instead: a second ALB for browser apps. Moving a service between ALBs means moving the listener rule, the deployment group's listener and the DNS alias together. Check the rule forwards to the real target group first.
• The WebSocket idle timeout lived on an unmerged branch. The live dev ALB had 600 seconds, applied from a feature branch. Main still had the 60-second default. Any apply from main would silently cut every live session. Instead: merge infra changes before or with the feature that needs them.
• We blamed NAT for an endpoint bill. The bill line named after the VPC service was interface endpoint hours. NAT bills under a different line ("EC2 - Other"). Instead: check the usage type in Cost Explorer before you blame a resource.
• We paid for an endpoint nobody used. An SQS endpoint stayed up in two environments for queues that received zero messages. Instead: before adding an endpoint, check that a service in a private subnet calls that AWS service. Removing one is safe if a NAT exists, because traffic falls back to NAT.
• Dev could not catch endpoint bugs. Dev turned endpoints off to save money, so only staging matched prod's network path. Instead: accept this knowingly, and never also remove them from staging.
• Plain HTTP timed out on every web host. The ALB had a port-80 listener that redirects to HTTPS, but the security group only opened 443. Instead: open 80 wherever a redirect listener exists. Do not fix it by editing a security group's description: that field cannot be changed, so Terraform replaces the whole group.
• The WAF was installed but only counting. Every rule was in Count mode. It logged attacks and blocked none. Instead: treat "Count" as a temporary state with an end date.
• The cheapest CloudFront price class had no edge near our users. Dev and staging used the cheapest class, which has no South Asia edge. Prod used a higher class. Instead: pick the price class by where your users are, and know dev latency will differ.
• Blue/green swapped the wrong thing. Our deploy tool's blue/green swap updated the listener's default action, but our routing used host-header rules. Deploys finished, yet traffic still went to the old, empty target group. Instead: make sure your deploy swaps the rule your traffic actually uses, and tell Terraform to ignore the rule action so it does not undo the swap.

Checklist

• Databases are in isolated subnets.
• My service's security group only accepts traffic from the ALB's security group.
• My hostname has a listener rule with a unique priority.
• If my service holds long connections, the idle timeout and deregistration delay are raised on main.
• My public ALB has WAF in Block mode.
• Static files are served by CloudFront, not by my container.