loxilb - How to debug

  • Check loxilb logs

loxilb logs its various important events and logs in the file /var/log/loxilb.log. Users can check it by using tail -f or any other command of choice.

root@752531364e2c:/# tail -f /var/log/loxilb.log 
DBG:  2022/07/10 12:49:27 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:49:37 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:49:47 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:49:57 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:50:07 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:50:17 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:50:27 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:50:37 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:50:47 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
DBG:  2022/07/10 12:50:57 1:dst-10.10.10.1/32,proto-6,dport-2020,,do-dnat:eip-31.31.31.1,ep-5001,w-1,alive|eip-32.32.32.1,ep-5001,w-2,alive|eip-100.100.100.1,ep-5001,w-2,alive| pc 0 bc 0 
  • Check loxicmd to debug loxilb's internal state
## Spawn a bash shell of loxilb docker 
docker exec -it loxilb bash

root@752531364e2c:/# loxicmd get lb       
| EXTERNALIP | PORT | PROTOCOL | SELECT | # OF ENDPOINTS |
|------------|------|----------|--------|----------------|
| 10.10.10.1 | 2020 | tcp      |      0 |              3 |


root@752531364e2c:/# loxicmd get lb -o wide
| EXTERNALIP | PORT | PROTOCOL | SELECT |  ENDPOINTIP   | TARGETPORT | WEIGHT |
|------------|------|----------|--------|---------------|------------|--------|
| 10.10.10.1 | 2020 | tcp      |      0 | 31.31.31.1    |       5001 |      1 |
|            |      |          |        | 32.32.32.1    |       5001 |      2 |
|            |      |          |        | 100.100.100.1 |       5001 |      2 |


root@0c4f9175c983:/# loxicmd get conntrack
| DESTINATIONIP |  SOURCEIP  | DESTINATIONPORT | SOURCEPORT | PROTOCOL |    STATE    | ACT |
|---------------|------------|-----------------|------------|----------|-------------|-----|
| 127.0.0.1     | 127.0.0.1  |           11111 |      47180 | tcp      | closed-wait |     |
| 127.0.0.1     | 127.0.0.1  |           11111 |      47182 | tcp      | est         |     |
| 32.32.32.1    | 31.31.31.1 |           35068 |      35068 | icmp     | bidir       |     |


root@65ad9b2f1b7f:/# loxicmd get port
| INDEX | PORTNAME |        MAC        | LINK/STATE  |    L3INFO     |    L2INFO     |
|-------|----------|-------------------|-------------|---------------|---------------|
|     1 | lo       | 00:00:00:00:00:00 | true/false  | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3801    |
|       |          |                   |             | IPv6 : []     |               |
|     2 | vlan3801 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3801    |
|       |          |                   |             | IPv6 : []     |               |
|     3 | llb0     | 42:6e:9b:7f:ff:36 | true/false  | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3803    |
|       |          |                   |             | IPv6 : []     |               |
|     4 | vlan3803 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3803    |
|       |          |                   |             | IPv6 : []     |               |
|     5 | eth0     | 02:42:ac:1e:01:c1 | true/true   | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3805    |
|       |          |                   |             | IPv6 : []     |               |
|     6 | vlan3805 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3805    |
|       |          |                   |             | IPv6 : []     |               |
|     7 | enp1     | fe:84:23:ac:41:31 | false/false | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3807    |
|       |          |                   |             | IPv6 : []     |               |
|     8 | vlan3807 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3807    |
|       |          |                   |             | IPv6 : []     |               |
|     9 | enp2     | d6:3c:7f:9e:58:5c | false/false | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3809    |
|       |          |                   |             | IPv6 : []     |               |
|    10 | vlan3809 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3809    |
|       |          |                   |             | IPv6 : []     |               |
|    11 | enp2v15  | 8a:9e:99:aa:f9:c3 | false/false | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3811    |
|       |          |                   |             | IPv6 : []     |               |
|    12 | vlan3811 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3811    |
|       |          |                   |             | IPv6 : []     |               |
|    13 | enp3     | f2:c7:4b:ac:fd:3e | false/false | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3813    |
|       |          |                   |             | IPv6 : []     |               |
|    14 | vlan3813 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3813    |
|       |          |                   |             | IPv6 : []     |               |
|    15 | enp4     | 12:d2:c3:79:f3:6a | false/false | Routed: false | IsPVID: true  |
|       |          |                   |             | IPv4 : []     | VID : 3815    |
|       |          |                   |             | IPv6 : []     |               |
|    16 | vlan3815 | aa:bb:cc:dd:ee:ff | true/true   | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 3815    |
|       |          |                   |             | IPv6 : []     |               |
|    17 | vlan100  | 56:2e:76:b2:71:48 | false/false | Routed: false | IsPVID: false |
|       |          |                   |             | IPv4 : []     | VID : 100     |
|       |          |                   |             | IPv6 : []     |               |

  • Debug loxilb kernel and eBPF components

loxilb uses various eBPF maps as part of its DP implementation. These maps are pinned to OS filesystem and can be further used with bpftool to debug.

root@0c4f9175c983:/# ls -lart /opt/loxilb/dp/bpf/
total 0
-rw------- 1 root root 0 Jul 10 11:32 xfis
-rw------- 1 root root 0 Jul 10 11:32 xfck
-rw------- 1 root root 0 Jul 10 11:32 xctk
-rw------- 1 root root 0 Jul 10 11:32 tx_intf_stats_map
-rw------- 1 root root 0 Jul 10 11:32 tx_intf_map
-rw------- 1 root root 0 Jul 10 11:32 tx_bd_stats_map
-rw------- 1 root root 0 Jul 10 11:32 tmac_stats_map
-rw------- 1 root root 0 Jul 10 11:32 tmac_map
-rw------- 1 root root 0 Jul 10 11:32 smac_map
-rw------- 1 root root 0 Jul 10 11:32 sess_v4_stats_map
-rw------- 1 root root 0 Jul 10 11:32 sess_v4_map
-rw------- 1 root root 0 Jul 10 11:32 rt_v6_stats_map
-rw------- 1 root root 0 Jul 10 11:32 rt_v4_stats_map
-rw------- 1 root root 0 Jul 10 11:32 rt_v4_map
-rw------- 1 root root 0 Jul 10 11:32 polx_map
-rw------- 1 root root 0 Jul 10 11:32 pkts
-rw------- 1 root root 0 Jul 10 11:32 pkt_ring
-rw------- 1 root root 0 Jul 10 11:32 pgm_tbl
-rw------- 1 root root 0 Jul 10 11:32 nat_v4_map
-rw------- 1 root root 0 Jul 10 11:32 mirr_map
-rw------- 1 root root 0 Jul 10 11:32 intf_stats_map
-rw------- 1 root root 0 Jul 10 11:32 intf_map
-rw------- 1 root root 0 Jul 10 11:32 fcas
-rw------- 1 root root 0 Jul 10 11:32 fc_v4_stats_map
-rw------- 1 root root 0 Jul 10 11:32 fc_v4_map
-rw------- 1 root root 0 Jul 10 11:32 dmac_map
-rw------- 1 root root 0 Jul 10 11:32 ct_v4_map
-rw------- 1 root root 0 Jul 10 11:32 bd_stats_map
-rw------- 1 root root 0 Jul 10 11:32 acl_v6_stats_map
-rw------- 1 root root 0 Jul 10 11:32 acl_v4_stats_map
-rw------- 1 root root 0 Jul 10 11:32 acl_v4_map
drwxrwxrwt 3 root root 0 Jul 10 11:32 ..
lrwxrwxrwx 1 root root 0 Jul 10 11:32 xdp -> /opt/loxilb/dp/bpf//tc/
drwx------ 3 root root 0 Jul 10 11:32 tc
lrwxrwxrwx 1 root root 0 Jul 10 11:32 ip -> /opt/loxilb/dp/bpf//tc/


root@752531364e2c:/# bpftool map dump pinned /opt/loxilb/dp/bpf/intf_map 
[{
        "key": {
            "ifindex": 2,
            "ing_vid": 0,
            "pad": 0
        },
        "value": {
            "ca": {
                "act_type": 11,
                "ftrap": 0,
                "oif": 0,
                "cidx": 0
            },
            "": {
                "set_ifi": {
                    "xdp_ifidx": 1,
                    "zone": 0,
                    "bd": 3801,
                    "mirr": 0,
                    "polid": 0,
                    "r": [0,0,0,0,0,0
                    ]
                }
            }
        }
    },{
        "key": {
            "ifindex": 3,
            "ing_vid": 0,
            "pad": 0
        },
        "value": {
            "ca": {
                "act_type": 11,
                "ftrap": 0,
                "oif": 0,
                "cidx": 0
            },
            "": {
                "set_ifi": {
                    "xdp_ifidx": 3,
                    "zone": 0,
                    "bd": 3803,
                    "mirr": 0,
                    "polid": 0,
                    "r": [0,0,0,0,0,0
                    ]
                }
            }
        }
    }
]


root@752531364e2c:/# bpftool map dump pinned /opt/loxilb/dp/bpf/nat_v4_map
[{
        "key": {
            "daddr": 17435146,
            "dport": 58375,
            "zone": 0,
            "l4proto": 6
        },
        "value": {
            "ca": {
                "act_type": 5,
                "ftrap": 0,
                "oif": 0,
                "cidx": 1
            },
            "lock": {
                "val": 0
            },
            "nxfrm": 3,
            "sel_hint": 0,
            "sel_type": 0,
            "nxfrms": [{
                    "nat_flags": 0,
                    "inactive": 0,
                    "wprio": 1,
                    "res": 0,
                    "nat_xport": 35091,
                    "nat_xip": 18816799
                },{
                    "nat_flags": 0,
                    "inactive": 0,
                    "wprio": 2,
                    "res": 0,
                    "nat_xport": 35091,
                    "nat_xip": 18882592
                },{
                    "nat_flags": 0,
                    "inactive": 0,
                    "wprio": 2,
                    "res": 0,
                    "nat_xport": 35091,
                    "nat_xip": 23356516
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                },{
                    "nat_flags": 0,
                    "inactive": 1,
                    "wprio": 0,
                    "res": 0,
                    "nat_xport": 0,
                    "nat_xip": 0
                }
            ]
        }
    }
]
  • Check eBPF kernel debug logs

Last but not the least, linux kernel outputs generic eBPF debug logs to /sys/kernel/debug/tracing/trace_pipe. Although loxilb eBPF modules do not emit logs in normal mode of operation, logs can be enabled after a recompilation.

root@752531364e2c:/# cat /sys/kernel/debug/tracing/trace_pipe        
         loxicmd-30524   [001] d.s1 27870.170790: bpf_trace_printk: out-dir
         loxicmd-30524   [001] d.s1 27870.170791: bpf_trace_printk: smr 4
         loxicmd-30529   [000] d.s1 27871.617467: bpf_trace_printk: [CTRK] start

         loxicmd-30529   [000] d.s1 27871.617484: bpf_trace_printk: new-ct4
         loxicmd-30529   [000] d.s1 27871.617486: bpf_trace_printk: in-dir
         loxicmd-30529   [000] d.s1 27871.617488: bpf_trace_printk: smr 0
         loxicmd-30529   [000] d.s1 27871.617503: bpf_trace_printk: [CTRK] start

         loxicmd-30529   [000] d.s1 27871.617503: bpf_trace_printk: out-dir
         loxicmd-30529   [000] d.s1 27871.617504: bpf_trace_printk: smr 4
            sshd-30790   [000] d.s1 27970.031847: bpf_trace_printk: [CTRK] start

            sshd-30790   [000] d.s1 27970.031866: bpf_trace_printk: new-ct4
            sshd-30790   [000] d.s1 27970.031868: bpf_trace_printk: in-dir
            sshd-30790   [000] d.s1 27970.031870: bpf_trace_printk: smr 0
            sshd-30790   [000] d.s1 27970.031887: bpf_trace_printk: [CTRK] start

            sshd-30790   [000] d.s1 27970.031887: bpf_trace_printk: out-dir
            sshd-30790   [000] d.s1 27970.031888: bpf_trace_printk: smr 0
            sshd-30790   [000] d.s1 27970.031900: bpf_trace_printk: [CTRK] start