How to run loxilb with calico CNI in BGP mode

This article describes how to integrate loxilb using calico CNI in Kubernetes.


For this example, kubernetes and loxilb are setup as follows:


Kubernetes uses a single master cluster for simplicity, all clusters use the same subnet. The load balancer node where the loxilb container is running is also connected to the same subnet as kubernetes. Externally, all kubernetes connections are configured to go through the "loxilb" load balancer node.

This example uses docker to run the loxilb container. These examples assume that kubernetes & calico are already installed.

1. loxilb setup

1.1 docker network setup

In order to integrate loxilb and kubernetes, loxilb needs to be able to communicate with Kubernetes. First, we create a docker network so that the loxilb container is also connected to the same network which is used by kubernetes nodes. Currently, the load balancer node is connected to kubernetes through the eno6 interface. Therefore, we will create a macvlan-type docker network that uses the eno6 interface as a parent and provide it to the loxilb docker. Create a docker network with the following command:

sudo docker network create -d macvlan -o parent=eno6 \
  --subnet \
  --gateway \
  --aux-address 'cp1=' \
  --aux-address 'cp2=' \
  --aux-address 'cp3=' k8snet
Options Description
-d macvlan Specify network type as macvlan
-o parent=eno6 Create a macvlan type network using the eno6 interface as parent
--subnet Specify network subnet
--gateway Set gateway (optional)
--aux-address 'serverName=serverIP' Option to register in advance so that IP addresses already in use on the network are not duplicated
k8snet Name the network k8snet

A docker network that can communicate with the outside is also created so that traffic accessing the kubernetes service from outside also goes through loxilb. The load balancer node is connected to the outside through eno8.

sudo docker network create -d macvlan -o parent=eno8 \
  --subnet \
  --gateway llbnet

We can check the network created with the docker network list command.

netlox@nd8:~$ sudo docker network list
5c97ae74fc32   bridge    bridge    local
6142f53e8be6   host      host      local
24ee7dbd7707   k8snet    macvlan   local
81c96ceda375   llbnet    macvlan   local
7bcd1738501b   none      null      local

1.2 loxilb docker setup

The loxilb container image is provided at github. To download the docker image, use the following command.

docker pull

To run loxilb docker, we can use the following command.

sudo docker run -u root --cap-add SYS_ADMIN --restart unless-stopped \
  --privileged -dit -v /dev/log:/dev/log \
  --net=k8snet --ip= --name loxilb \

The options that need to specified are: |Options|Description| |----|----| |--net=k8snet|Network to connect to container| |--ip=|Specifies the IP address the container will use. If not specified, use any IP within the network subnet range| |--name loxilb|Set container name|

We can check the docker created with the docker ps command.

netlox@nd8:~$ sudo docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED       STATUS       PORTS       NAMES
eae349a283ae   loxilbio/loxilb:beta   "/root/loxilb-io/lox…"   11 days ago   Up 11 days               loxilb

Since we only connected the kubernetes network (k8snet) when running the docker above, we also need to connect to the docker network for external communication. Currently, docker only supports one network to be connected when running "docker run" command. But, its easy to connect other network to the docker with the following command:

sudo docker network connect llbnet loxilb

Once the connection is complete, we can see the docker container's interfaces as follows:

netlox@netlox:~$ sudo docker exec -ti loxilb ip route
default via dev eth0 dev eth0 proto kernel scope link src dev eth1 proto kernel scope link src

2. loxi-ccm setup in kubernetes

loxi-ccm is a ccm provider to provide a loxilb load balancer to kubernetes, and it is essential for interworking with kubernetes and loxilb. Refer to the relevant document, change the apiServerURL of configMap to the IP address of loxilb created above and install it in kubernetes. If loxi-ccm is installed properly, the setup is complete.

3.Basic load-balancer test

We can now give an External IP when you create a LoadBalancer type service in kubernetes. Create a test-nginx-svc.yaml file for testing as follows:

apiVersion: v1
kind: Pod
  name: nginx
  labels: proxy
  - name: nginx
    image: nginx:latest
      - containerPort: 80
        name: http-web-svc
apiVersion: v1
kind: Service
  name: nginx-service
  type: LoadBalancer
  selector: proxy
  - name: name-of-service-port
    protocol: TCP
    port: 8888
    targetPort: http-web-svc

The above steps create the nginx pod and then associates a LoadBalancer service to it. The command to apply is as follows

kubectl apply -f test-nginx-svc.yaml

We can verify that the service nginx-service has been created as a LoadBalancer type and has been assigned an External IP. Now you can access the kubernetes service from outside using IP and port 8888.

vagrant@node1:~$ sudo kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
kubernetes      ClusterIP      <none>           443/TCP          28d
nginx-service   LoadBalancer   8888:31655/TCP   3s

The LoadBalancer rule is also created in the loxilb container. We can check in the loxilb load-balancer node as follows

netlox@nd8:~$ sudo docker exec -ti loxilb loxicmd get lb
| | 8888 | tcp      |      0 |              2 |

4. calico BGP & loxilb setup

If calico configures the network in BGP mode, loxilb must also operate in BGP mode. loxilb supports BGP functions based on goBGP. The following description assumes that calico is already set to use BGP mode.

4.1 loxilb BGP mode setup

If we create a loxilb container with the following command, it will run in BGP mode. The -b option at the end of the command is to enable BGP mode in loxilb.

sudo docker run -u root --cap-add SYS_ADMIN --restart unless-stopped \
  --privileged -dit -v /dev/log:/dev/log \
  --net=k8snet --ip= --name loxilb \
  --host= -b

4.2 gobgp_loxilb.yaml file setup

Create a gobgp_loxilb.yaml file in the /opt/loxilb/ directory of the loxilb container.

        as: 65002
    - config:
        peer-as: 64512
    - config:
        peer-as: 64001

BGP information such as as-id and router-id of the loxilb container must be registered as global items. The neighbors item need info about the IP address and as-id information of the BGP router peering with loxilb. In this example, calico's BGP information ( and external BGP information ( were registered.

4.3 Add router-id to the lo interface of the loxilb container

We need to add the IP registered as loxilb router-id in the gobgp_loxilb.yaml file to the lo interface of loxilb docker.

sudo docker exec -ti loxilb ip addr add dev lo

4.4 loxilb docker restart

Restart the loxilb docker for the settings in gobgp_loxilb.yaml to take effect.

sudo docker stop loxilb
sudo docker start loxilb

4.5 Setup BGP Peer information in Calico

We also need to add loxilb's BGP peer information to calico. Create the calico-bgp-config.yaml file as follows:

kind: BGPPeer
  name: my-global-peers2
  asNumber: 65002

In peerIP, enter the IP address of loxilb. In asNumber, enter the as-ID of the loxilb BGP set above. After creating the file, add BGP peer information to calico with the command below.

sudo calicoctl apply -f calico-bgp-config.yaml

4.6 Check BGP status

We now check the BGP connectivity in the loxilb docker like this:

netlox@nd8:~$ sudo docker exec -ti loxilb3 gobgp neigh
Peer              AS  Up/Down State       |#Received  Accepted 64512 00:00:59 Establ      |        4         4

If the connection is successful, the State will be shown as "Established". We can check the route information of calico with the gobgp global rib command.

netlox@nd8:~$ sudo docker exec -ti loxilb3 gobgp global rib
   Network              Next Hop             AS_PATH              Age        Attrs
*>       64512                01:02:03   [{Origin: i}]
*>       64512                01:02:03   [{Origin: i}]
*>       64512                01:02:03   [{Origin: i}]
*>       64512                01:02:03   [{Origin: i}]