> Lincoln Stoll’s k8s-vpcnet, and more recently, Amazon’s amazon-vpc-cni-k8s CNI stacks use Elastic Network Interfaces (ENIs) and secondary private IPs to achieve an overlay-free AWS VPC-native solutions for Kubernetes networking. While both of these solutions achieve the same base goal of drastically simplifying the network complexity of deploying Kubernetes at scale on AWS, they do not focus on minimizing network latency and kernel overhead as part of implementing a compliant networking stack.
>"To run Kubernetes over AWS VPC, we would like to reach following additional goals:
Networking for Pods must support high throughput and availability, low latency and minimal jitter comparable to the characteristics a user would get from EC2 networking"
Both the Lyft and AWS CNIs use ENIs, Romana's CNI does not. But more specifically, vpc-router works along with Romana's IPAM to aggregate routes so that each VPC route can forward traffic for multiple instances. So, instead of one route per instance, you need only 1 routes per n instances. Where n is set by how much aggregation you want (configurable).
The net effect is that you can build large clusters without running out of VPC routes and no overlay is needed when traffic crosses AZs.
When a route is used to forward traffic for multiple instances, the target instance acts as router and forwards traffic to the final destination instance. This works because instances within an AZ have routes installed on them to the pod CIDRs on the other instances in the zone, so any one of them can perform this forwarding function.
Romana only piggybacks routes when there are no more VPC routes available, so for small cluster it's just like kubenet. For large clusters routes it uses all the instances to forward traffic so that none of them become a bottleneck.
> Lincoln Stoll’s k8s-vpcnet, and more recently, Amazon’s amazon-vpc-cni-k8s CNI stacks use Elastic Network Interfaces (ENIs) and secondary private IPs to achieve an overlay-free AWS VPC-native solutions for Kubernetes networking. While both of these solutions achieve the same base goal of drastically simplifying the network complexity of deploying Kubernetes at scale on AWS, they do not focus on minimizing network latency and kernel overhead as part of implementing a compliant networking stack.