Understanding the Impact of Load Balancing on Multi-AZ Kubernetes Deployments

If your multi-AZ Kubernetes cluster is integrated with a service mesh, you might be incurring additional costs and latency that aren't immediately evident on your monitoring dashboards.

The Default Behavior of Load Balancers

Kubernetes Services and Istio's Envoy sidecars typically distribute incoming traffic randomly among all available endpoints, without considering their availability zones (AZs). While this makes sense in a single-AZ context, it presents challenges in multi-AZ environments where services are intentionally spread across different zones for resilience.

For instance, when a pod in us-east-1a accesses an internal service, there's a significant chance it will connect to a pod located in another AZ, which can lead to inefficient routing and increased latency. The implications of this cross-AZ communication can multiply quickly as a single user request may traverse several services, accumulating more cross-AZ hops along the way.

Compounding this issue, AWS amplifies the inefficiencies. With cross-zone load balancing enabled on a Network Load Balancer (NLB) in front of an Istio ingress gateway, an incoming request might get routed to an endpoint in a different AZ, even if a nearby endpoint could handle the traffic.

The Real Costs of Inefficient Load Balancing

Two primary concerns arise from this configuration: latency and cost.

Latency: In mid-sized deployments with around 3,500 requests per second across multiple AZs, data reveals that requests managed within the same AZ yield a median response time of 15-18ms. In contrast, cross-AZ requests take considerably longer, often clocking in at 25-30ms. With a significant portion of requests being cross-AZ, the overall weighted median response time can elevate to around 24ms, which is detrimental to user experience, particularly for systems reliant on swift response times.

Cost: AWS invoices $0.01 per GB for inter-AZ data transfers. This may seem minor per request but can escalate quickly because internal service traffic tends to be far greater than external traffic due to numerous API calls and database interactions. For instance, in a given scenario, service-to-service and database communication racked up an estimated $600 monthly in crossed-AZ instances. This figure doesn't account for additional costs related to Aurora replication and log shipping, which would further inflate the expenses. It's essential for users to examine their specific traffic through AWS Cost Explorer for tailored insights.

Implementing Locality-Aware Load Balancing

To counter these problems, Istio allows for locality-aware load balancing, which can dramatically improve both latency and cost management. However, the simplistic fix of routing 100% of traffic to the local AZ and none to others is flawed and compromises resilience. Instead, a more nuanced weighted approach is prudent: directing 80% of traffic to the local AZ, while allocating 10% each to the other two zones. This balances the load fairly while mitigating risk from local hotspots and potential failures.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: checkout-service-locality-lb
spec:
  host: checkout-service.prod.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
          - from: us-east-1a/*
            to:
              "us-east-1a/*": 80
              "us-east-1b/*": 10
              "us-east-1c/*": 10
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

For this approach to be effective, three key conditions must be met:

1. Even distribution of pods across AZs: If one zone hosts more than half of a service's pods, the traffic policy may inadvertently send more requests to that AZ, negating the intended balance. Employ topologySpreadConstraints with maxSkew: 1 to maintain pod equilibrium.
2. Ingress pods must be distributed across AZs: Locality-aware routing won't be beneficial if incoming requests are concentrated within a single AZ.
3. Disable NLB cross-zone load balancing: Use the annotation service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false". Neglecting this leads to the reintroduction of the random nature of load distribution that the mesh layer was meant to eliminate.

Evaluating the Benefits and Trade-offs

After adjusting to a locality-aware strategy, expect improved response latency and a reduced failure blast radius. Previously, a complete AZ failure could dramatically increase load on the remaining zones, but with the weighted traffic policy, the impact is easier to manage.

However, transitioning to this system comes at the cost of added operational complexity. Understanding why certain AZs receive disproportionate traffic requires knowledge of the locality policy. It's vital to document these changes to prevent confusion among team members monitoring systems.

Testing Before Deployment

Always validate this setup in a lower environment with realistic load testing. Tools like k6 or Locust can expose potential imbalances that may arise from uneven pod distributions. For production, roll out changes to lesser-used services first—monitoring error rates and latency carefully instead of solely focusing on cross-AZ traffic statistics.

Locality-aware load balancing is not a novel concept within Istio but often overlooked because the service mesh generally functions adequately without it—until the benefits, or drawbacks, become apparent during a crisis. Therefore, consider evaluating your configurations to ensure your multi-AZ cluster operates optimally.