aws vpc-cni 插件源码解析

aws vpc-cni 插件源码解析

前言

aws vpc cni 模型采用的是大二层underlay的网络方案，pod分配到的是vpc ip,与节点主机同属于一层网络中，相对于flannel等overlay方案更加简单直接，这样做带来的优势是：

1. 实现集群内外ip直接互通

在k8s的落地过程中，很多企业往往都有强烈的需求保证k8s集群内外网络的直连互通

2. 减少了overlay方案中数据包的封包和拆包的网络损耗，能够提升性能

原理

主要实现逻辑:

Worker节点启动的时候挂载多个虚拟网卡ENI(Elastic Netowrk Interface) 每个ENI都绑定了一个主IP(Primary ip)  和 多个 Secondry ip ipamd(Local IP Address Manager)运行在每个worker 节点上,将所有ENI的所有secondary ip 加入到本地ip地址池中 当cni接受到创建pod事件请求时，就会通过grpc请求ipamd拿到ip并设置pod网络栈;反之，当接收到删除pod请求时就会通知ipamd释放ip并同时删除pod网络栈

CNI

遵守k8S CNI网络模型的接口规范,主要实现了cmdAdd cmdDel接口,分别处理pod网络的创建和销毁事件

代码路径: cmd/routed-eni-cni-plugin/cni.go

cmdAdd

func cmdAdd(args *skel.CmdArgs) error { return add(args, typeswrapper.New(), grpcwrapper.New(), rpcwrapper.New(), driver.New()) } func add(args *skel.CmdArgs, cniTypes typeswrapper.CNITYPES, grpcClient grpcwrapper.GRPC, rpcClient rpcwrapper.RPC, driverClient driver.NetworkAPIs) error { conf, log, err := LoadNetConf(args.StdinData) ... // 解析 k8s参数 var k8sArgs K8sArgs if err := cniTypes.LoadArgs(args.Args, &k8sArgs); err != nil { log.Errorf("Failed to load k8s config from arg: %v", err) return errors.Wrap(err, "add cmd: failed to load k8s config from arg") } ... // 通过grpc发起请求到ipamd server conn, err := grpcClient.Dial(ipamdAddress, grpc.WithInsecure()) ... c := rpcClient.NewCNIBackendClient(conn) // 调用ipamd的AddNetwork接口获取ip地址 r, err := c.AddNetwork(context.Background(), &pb.AddNetworkRequest{ ClientVersion: version, K8S_POD_NAME: string(k8sArgs.K8S_POD_NAME), K8S_POD_NAMESPACE: string(k8sArgs.K8S_POD_NAMESPACE), K8S_POD_INFRA_CONTAINER_ID: string(k8sArgs.K8S_POD_INFRA_CONTAINER_ID), Netns: args.Netns, ContainerID: args.ContainerID, NetworkName: conf.Name, IfName: args.IfName, }) ... addr := &net.IPNet{ IP: net.ParseIP(r.IPv4Addr), Mask: net.IPv4Mask(255, 255, 255, 255), } ... // 获取到ip后,调用driver模块配置pod的network namespace err = driverClient.SetupNS(hostVethName, args.IfName, args.Netns, addr, int(r.DeviceNumber), r.VPCcidrs, r.UseExternalSNAT, mtu, log) } ... ips := []*current.IPConfig{ { Version: "4", Address: *addr, }, } result := ¤t.Result{ IPs: ips, } return cniTypes.PrintResult(result, conf.CNIVersion) }

dataStore dataStore 是一个通过结构体构造的本地DB,维护着本地节点ENI信息,以及ENI上绑定的所有ip，每条ip信息都以ipamkey作为主键;当ip被分配,则会以(network name, CNI_CONTAINERID, CNI_IFNAME)作为主键值;反之，ip没有被分配,ipamkey会被设置为空值 代码路径 /pkg/ipamd/datastore/data_store.go type DataStore struct { total int assigned int allocatedPrefix int eniPool ENIPool lock sync.Mutex log logger.Logger CheckpointMigrationPhase int backingStore Checkpointer cri cri.APIs isPDEnabled bool } type ENI struct { ID string createTime time.Time IsPrimary bool IsTrunk bool IsEFA bool DeviceNumber int AvailableIPv4Cidrs map[string]*CidrInfo } type AddressInfo struct { IPAMKey IPAMKey Address string UnassignedTime time.Time } type CidrInfo struct { Cidr net.IPNet // 192.168.1.1/24 IPv4Addresses map[string]*AddressInfo IsPrefix bool } type ENIPool map[string]*ENI //['eniid]eni cni本质上是直接调用datastore的  AssignPodIPv4Address和UnAssignPodIPv4Address方法来获取ip和释放ip AssignPodIPv4Address // 将ip分配给pod func (ds *DataStore) AssignPodIPv4Address(ipamKey IPAMKey) (ipv4address string, deviceNumber int, err error) { // 对 dataStore操作加互斥锁 ds.lock.Lock() defer ds.lock.Unlock() ... // 遍历dataStore的eniPool拿到ip for _, eni := range ds.eniPool { for _, availableCidr := range eni.AvailableIPv4Cidrs { var addr *AddressInfo var strPrivateIPv4 string var err error if (ds.isPDEnabled && availableCidr.IsPrefix) || (!ds.isPDEnabled && !availableCidr.IsPrefix) { strPrivateIPv4, err = ds.getFreeIPv4AddrfromCidr(availableCidr) if err != nil { ds.log.Debugf("Unable to get IP address from CIDR: %v", err) //Check in next CIDR continue } ... addr = availableCidr.IPv4Addresses[strPrivateIPv4] ... availableCidr.IPv4Addresses[strPrivateIPv4] = addr // 对于已分配的ip,设置其ipamkey ds.assignPodIPv4AddressUnsafe(ipamKey, eni, addr) ... return addr.Address, eni.DeviceNumber, nil } } ... } UnAssignPodIPv4Address // 释放ip地址 func (ds *DataStore) UnassignPodIPv4Address(ipamKey IPAMKey) (e *ENI, ip string, deviceNumber int, err error) { ... // 通过主键ipamKey 在enipool中找对对应的pod ip地址 eni, availableCidr, addr := ds.eniPool.FindAddressForSandbox(ipamKey) ... // 调用unassignPodIPv4AddressUnsafe 设置ip为未分配状态,即将IP地址对应的主键ipamkey设置为空 ds.unassignPodIPv4AddressUnsafe(addr) ... // 设置ip释放时间为当前时间 addr.UnassignedTime = time.Now() ... return eni, addr.Address, eni.DeviceNumber, nil }

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。