Table of Contents

• What is loxicmd
• How to build
• How to run and configure loxilb
  • Load balancer
  • Endpoint
  • BFD
  • Session
  • SessionUlCl
  • IPaddress
  • FDB
  • Route
  • Neighbor
  • Vlan
  • Vxlan
  • Firewall
  • Mirror
  • Policy
  • Session Recorder

What is loxicmd

loxicmd is command tool for loxilb's configuration. loxicmd aims to provide all configuation related to loxilb and is based on kubectl's look and feel. When running k8s, kube-loxilb usually takes care of loxilb configuration but nonetheless loxicmd can be used for enhanced config, debugging and observability.

How to build

Note - loxilb docker has this built-in and there is no need to build it when using loxilb docker

Install package dependencies

go get .

Make loxicmd

make

Install loxicmd

sudo cp -f ./loxicmd /usr/local/sbin

How to run and configure loxilb

Load Balancer

Get load-balancer rules

Get basic information

loxicmd get lb

Get detailed information

loxicmd get lb -o wide

Get info in json

loxicmd get lb -o json

Configure load-balancer rule

Simple NAT44 tcp (round-robin) load-balancer

loxicmd create lb 1.1.1.1 --tcp=1828:1920 --endpoints=2.2.3.4:1

Note:
- Round-robin is default mode in loxilb
- End-point format is specified as <CIDR:weight>. For round-robin, weight(1) has no significance.

NAT66 (round-robin) load-balancer

loxicmd create lb  2001::1 --tcp=2020:8080 --endpoints=4ffe::1:1,5ffe::1:1,6ffe::1:1

NAT64 (round-robin) load-balancer

loxicmd create lb  2001::1 --tcp=2020:8080 --endpoints=31.31.31.1:1,32.32.32.1:1,33.33.33.1:1

WRR (Weighted round-robin) load-balancer (Divide traffic in 40%, 40% and 20% ratio among end-points)

loxicmd create lb 20.20.20.1 --select=priority --tcp=2020:8080 --endpoints=31.31.31.1:40,32.32.32.1:40,33.33.33.1:20

Sticky end-point selection load-balancer (select end-points based on traffic hash)

loxicmd create lb 20.20.20.1 --select=hash --tcp=2020:8080 --endpoints=31.31.31.1:1,32.32.32.1:1,33.33.33.1:1

Load-balancer with forceful tcp-reset session timeout after inactivity of 30s

loxicmd create lb 20.20.20.1 --tcp=2020:8080 --endpoints=31.31.31.1:1,32.32.32.1:1,33.33.33.1:1 --inatimeout=30

Load-balancer with one-arm mode

loxicmd create lb 20.20.20.1 --tcp=2020:8080 --endpoints=100.100.100.2:1,100.100.100.3:1,100.100.100.4:1 --mode=onearm

Load-balancer with fullnat mode

loxicmd create lb 88.88.88.1 --sctp=38412:38412 --endpoints=192.168.70.3:1 --mode=fullnat

• For more information on one-arm and full-nat mode, please check this post

Load-balancer config in DSR(direct-server return) mode

loxicmd create lb 20.20.20.1 --tcp=2020:8080 --endpoints=31.31.31.1:1,32.32.32.1:1 --mode=dsr

Load-balancer config with active endpoint monitoring

loxicmd create lb 20.20.20.1 --tcp=2020:8080 --endpoints=31.31.31.1:1,32.32.32.1:1 --monitor

Note:
- By default loxilb does not do active endpoint monitoring i.e it will continue to select end-points which might be inactive
- This is due to the fact kubernetes also has its own service monitoring mechanism and it can notify loxilb of any such endpoint health state
- Based on user's requirements, one can specify active endpoint checks using "--monitor" flag
- loxilb has extensive endpoint monitoring methods. Further details can be found in endpoint section

Load-balancer yaml example

apiVersion: netlox/v1
kind: Loadbalancer
metadata:
  name: test
spec:
  serviceArguments:
    externalIP: 1.2.3.1
    port: 80
    protocol: tcp
    sel: 0
  endpoints:
  - endpointIP: 4.3.2.1
    weight: 1
    targetPort: 8080
  - endpointIP: 4.3.2.2
    weight: 1
    targetPort: 8080
  - endpointIP: 4.3.2.3
    weight: 1
    targetPort: 8080

Delete a load-balancer rule

loxicmd delete lb 1.1.1.1 --tcp=1828 

---

Endpoint

Get load-balancer end-point health information

loxicmd get ep

Create end-point for health probing

# loxicmd create endpoint IP [--name=<id>] [--probetype=<probetype>] [--probereq=<probereq>] [--proberesp=<proberesp>] [--probeport=<port>] [--period=<period>] [--retries=<retries>]
loxicmd create endpoint 32.32.32.1 --probetype=http --probeport=8080 --period=60 --retries=2

IP(string) : Endpoint target IPaddress
name(string) : Endpoint Identifier
probetype(string): Probe-type:ping,http,https,udp,tcp,sctp,none
probereq(string): If probe is http/https, one can specify additional uri path
proberesp(string): If probe is http/https, one can specify custom response string
probeport(int): If probe is http,https,tcp,udp,sctp one can specify custom l4port to use
period(int): Period of probing
retries(int): Number of retries before marking endPoint inactive

Notes:
- "name" is not required when endpoint is created initially. loxilb will allocate the name which can be checked with "loxicmd get ep". "name" can be given as an Identifier when user wants to modify endpoint probe parameters
- Initial state of endpoint will be decided within 15 seconds of rule addition (We cant be sure if service is immediately up so this is the init liveness check timeout. It is not configurable at this time)
- After init liveness check, probes will be done as per default (60s) or whatever value is set by the user
- When endpoint is inactive we have internal logic and timeouts to minimize blocking calls and maintain stability. Only when endpoint is active, we use probe timeout given by user
- For UDP end-points and probe-type, there are two ways to check end-point health currently:
- If the service can respond to probe requests with pre-defined responses sent over UDP, we can use the following :
loxicmd create endpoint 172.1.217.133 --name="udpep1" --probetype=udp --probeport=32031 --period=60 --retries=2 --probereq="probe" --proberesp="hello"
- If the services cannot support the above mechanism, loxilb will try to check for "ICMP Port unreachable" after sending UDP probes. If an "ICMP Port unreachable" is received, it means the endpoint is not up.

Examples :

loxicmd create endpoint 32.32.32.1 --probetype=http --probeport=8080 --period=60 --retries=2

loxicmd get ep
|    HOST    |         NAME         | PTYPE | PORT | DURATION | RETRIES | MINDELAY | AVGDELAY | MAXDELAY | STATE |
|------------|----------------------|-------|------|----------|---------|----------|----------|----------|-------|
| 32.32.32.1 | 32.32.32.1_http_8080 | http: | 8080 |       60 |       2 | 0s       | 0s       | 0s       | ok    |

# Modify duration and retry count using name

loxicmd create endpoint 32.32.32.1 --name=32.32.32.1_http_8080 --probetype=http --probeport=8080 --period=40 --retries=4

loxicmd get ep
|    HOST    |         NAME         | PTYPE | PORT | DURATION | RETRIES | MINDELAY | AVGDELAY  | MAXDELAY  | STATE |
|------------|----------------------|-------|------|----------|---------|----------|-----------|-----------|-------|
| 32.32.32.1 | 32.32.32.1_http_8080 | http: | 8080 |       40 |       4 | 0s       | 0s        | 0s        | ok    |

Create end-point with https probing information

# loxicmd create endpoint IP [--name=<id>] [--probetype=<probetype>] [--probereq=<probereq>] [--proberesp=<proberesp>] [--probeport=<port>] [--period=<period>] [--retries=<retries>]
loxicmd create endpoint 32.32.32.1 --probetype=https --probeport=8080 --probereq="health" --proberesp="OK" --period=60 --retries=2

Note: loxilb requires CA certificate for TLS connection and private certificate and private key for mTLS connection. Admin can keep a common(default) CA certificate for all the endpoints at "/opt/loxilb/cert/rootCA.crt" or per-endpoint certificates can be kept as "/opt/loxilb/cert/\<IP>/rootCA.crt", private key must be kept at "/opt/loxilb/cert/server.key" and private certificate at "/opt/loxilb/cert/server.crt". Please see Minica or Certstrap or this CICD test case to know how to generate certificates.

Endpoint yaml example

apiVersion: netlox/v1
kind: Endpoint
metadata:
  name: test
spec:
  hostName: "Test"
  description: string
  inactiveReTries: 0
  probeType: string
  probeReqUrl: string
  probeDuration: 0
  probePort: 0

Delete end-point informtion

loxicmd delete endpoint 31.31.31.31

---

BFD

Get BFD Session information

loxicmd get bfd

Create BFD Session

#loxicmd create bfd <remoteIP> --sourceIP=<sourceIP> --interval=<time in usecs> --retryCount=<count>
loxicmd create bfd 192.168.80.253 --sourceIP=192.168.80.252 --interval=500000 --retryCount=3

remoteIP(string): Remote IP address
sourceIP(string): Source IP address for binding
interval(int): BFD packet Tx Interval Time in microseconds
retryCount(int): Number of retry counts to detect failure.

Set BFD Session

#loxicmd set bfd <remoteIP> --interval=<time in usecs> --retryCount=<count>
loxicmd set bfd 192.168.80.253 --interval=400000 --retryCount=5

interval(int): BFD packet Tx Interval Time in microseconds
retryCount(int): Number of retry counts to detect failure.

Delete BFD Session

#loxicmd delete bfd <remoteIP>
loxicmd delete bfd 192.168.80.253

remoteIP(string): Remote IP address
sourceIP(string): Source IP address for binding
interval(int): BFD packet Tx Interval Time in microseconds
retryCount(int): Number of retry counts to detect failure.

BFD yaml example

apiVersion: netlox/v1
kind: BFD
metadata:
  name: test
spec:
  instance: "default"
  remoteIp: "192.168.80.253"
  sourceIp: "192.168.80.252"
  interval: 300000
  retryCount: 4

Session

Get Session information

loxicmd get session

Create Session information

#loxicmd create session <userID> <sessionIP> --accessNetworkTunnel=<TeID>:<TunnelIP> --coreNetworkTunnel=<TeID>:<TunnelIP>
loxicmd create session user1 192.168.20.1 --accessNetworkTunnel=1:1.232.16.1 coreNetworkTunnel=1:1.233.16.1

userID(string): User Identifier
sessionIP(string): Session IP address
accessNetworkTunnel(string): accessNetworkTunnel has pairs that can be specified as 'TeID:IP'
coreNetworkTunnel(string): coreNetworkTunnel has pairs that can be specified as 'TeID:IP'

Session yaml example

apiVersion: netlox/v1
kind: Session
metadata:
  name: test
spec:
  ident: user1
  sessionIP: 88.88.88.88
  accessNetworkTunnel:
    TeID: 1
    tunnelIP: 11.11.11.11
  coreNetworkTunnel:
    TeID: 1
    tunnelIP: 22.22.22.22

Delete Session information

loxicmd delete session user1

---

SessionUlCl

Get SessionUlCl information

loxicmd get sessionulcl

Create SessionUlCl information

#loxicmd create sessionulcl <userID> --ulclArgs=<QFI>:<ulclIP>,...
loxicmd create sessionulcl user1 --ulclArgs=16:192.33.125.1

userID(string): User Identifier
ulclArgs(string): Port pairs can be specified as 'QFI:UlClIP'

SessionUlCl yaml example

apiVersion: netlox/v1
kind: SessionULCL
metadata:
  name: test
spec:
  ulclIdent: user1
  ulclArgument:
    qfi: 11
    ulclIP: 8.8.8.8

Delete SessionUlCl information

loxicmd delete sessionulcl --ulclArgs=192.33.125.1

ulclArgs(string): UlCl IP address can be specified as 'UlClIP'. It don't need QFI.

---

IPaddress

Get IPaddress information

loxicmd get ip

Create IPaddress information

#loxicmd create ip <DeviceIPNet> <device>
loxicmd create ip 192.168.0.1/24 eno7

DeviceIPNet(string): Actual IP address with mask
device(string): name of the related device

IPaddress yaml example

apiVersion: netlox/v1
kind: IPaddress
metadata:
  name: test
spec:
  dev: eno8
  ipAddress: 192.168.23.1/32

Delete IPaddress information

#loxicmd delete ip <DeviceIPNet> <device>
loxicmd delete ip 192.168.0.1/24 eno7

---

FDB

Get FDB information

loxicmd get fdb

Create FDB information

#loxicmd create fdb <MacAddress> <DeviceName>
loxicmd create fdb aa:aa:aa:aa:bb:bb eno7

MacAddress(string): mac address
DeviceName(string): name of the related device

FDB yaml example

apiVersion: netlox/v1
kind: FDB
metadata:
  name: test
spec:
  dev: eno8
  macAddress: aa:aa:aa:aa:aa:aa

Delete FDB information

#loxicmd delete fdb <MacAddress> <DeviceName>
loxicmd delete fdb aa:aa:aa:aa:bb:bb eno7

---

Route

Get Route information

loxicmd get route

Create Route information

#loxicmd create route <DestinationIPNet> <gateway>
loxicmd create route 192.168.212.0/24 172.17.0.254

DestinationIPNet(string): Actual IP address route with mask
gateway(string): gateway information if any

Route yaml example

apiVersion: netlox/v1
kind: Route
metadata:
  name: test
spec:
  destinationIPNet: 192.168.30.0/24
  gateway: 172.17.0.1

Delete Route information

#loxicmd delete route <DestinationIPNet>
loxicmd delete route 192.168.212.0/24 

---

Neighbor

Get Neighbor information

loxicmd get neighbor

Create Neighbor information

#loxicmd create neighbor <DeviceIP> <DeviceName> [--macAddress=aa:aa:aa:aa:aa:aa]
loxicmd create neighbor 192.168.0.1 eno7 --macAddress=aa:aa:aa:aa:aa:aa

DeviceIP(string): The IP address
DeviceName(string): name of the related device
macAddress(string): resolved hardware address if any

Neighbor yaml example

apiVersion: netlox/v1
kind: Neighbor
metadata:
  name: test
spec:
  dev: eno8
  macAddress: aa:aa:aa:aa:aa:aa
  ipAddress: 192.168.23.21

Delete Neighbor information

#loxicmd delete neighbor <DeviceIP> <device>
loxicmd delete neighbor 192.168.0.1 eno7

---

Vlan

Get Vlan and Vlan Member information

loxicmd get vlan

loxicmd get vlanmember

Create Vlan and Vlan Member information

#loxicmd create vlan <Vid>
loxicmd create vlan 100

Vid(int): vlan identifier

#loxicmd create vlanmember <Vid> <DeviceName> --tagged=<Tagged>
loxicmd create vlanmember 100 eno7 --tagged=true
loxicmd create vlanmember 100 eno7

Vid(int): vlan identifier
DeviceName(string): name of the related device
tagged(boolean): tagged or not (default is false)

Vlan yaml example

apiVersion: netlox/v1
kind: Vlan
metadata:
  name: test
spec:
  vid: 100

Vlan Member yaml example

apiVersion: netlox/v1
kind: VlanMember
metadata:
  name: test
  vid: 100
spec:
  dev: eno8
  Tagged: true

Delete Vlan and Vlan Member information

#loxicmd delete vlan <Vid>
loxicmd delete vlan 100

#loxicmd delete vlanmember <Vid> <DeviceName> --tagged=<Tagged>
loxicmd delete vlanmember 100 eno7 --tagged=true
loxicmd delete vlanmember 100 eno7

---

Vxlan

Get Vxlan and Vxlan Peer information

loxicmd get vxlan

loxicmd get vxlanpeer

Create Vxlan and Vxlan Peer information

#loxicmd create vxlan <VxlanID> <EndpointDeviceName>
loxicmd create vxlan 100 eno7

VxlanID(int): Vxlan Identifier
EndpointDeviceName(string): VTEP Device name(It must have own IP address for peering)

#loxicmd create vxlanpeer <VxlanID> <PeerIP>
loxicmd create vxlan-peer 100 30.1.3.1

VxlanID(int): Vxlan Identifier
PeerIP(string): Vxlan peer device IP address

Vxlan yaml example

apiVersion: netlox/v1
kind: Vxlan
metadata:
  name: test
spec:
  epIntf: eno8
  vxlanID: 100

Vxlan Peer yaml example

apiVersion: netlox/v1
kind: VxlanPeer
metadata:
  name: test
  vxlanID: 100
spec:
  peerIP: 21.21.21.1

Delete Vxlan and Vxlan Peer information

#loxicmd delete vxlan <VxlanID>
loxicmd delete vxlan 100

#loxicmd delete vxlanpeer <VxlanID> <PeerIP>
loxicmd delete vxlan-peer 100 30.1.3.1

---

Firewall

Get Firewall information

loxicmd get firewall

Create Firewall information

#loxicmd create firewall --firewallRule=<ruleKey>:<ruleValue>, [--allow] [--drop] [--trap] [--redirect=<PortName>] [--setmark=<FwMark>
loxicmd create firewall --firewallRule="sourceIP:1.2.3.2/32,destinationIP:2.3.1.2/32,preference:200" --allow
loxicmd create firewall --firewallRule="sourceIP:1.2.3.2/32,destinationIP:2.3.1.2/32,preference:200" --allow --setmark=10
loxicmd create firewall --firewallRule="sourceIP:1.2.3.2/32,destinationIP:2.3.1.2/32,preference:200" --drop
loxicmd create firewall --firewallRule="sourceIP:1.2.3.2/32,destinationIP:2.3.1.2/32,preference:200" --trap
loxicmd create firewall --firewallRule="sourceIP:1.2.3.2/32,destinationIP:2.3.1.2/32,preference:200" --redirect=ensp0

firewallRule sourceIP(string) - Source IP in CIDR notation
destinationIP(string) - Destination IP in CIDR notation
minSourcePort(int) - Minimum source port range
maxSourcePort(int) - Maximum source port range
minDestinationPort(int) - Minimum destination port range
maxDestinationPort(int) - Maximum source port range
protocol(int) - the protocol
portName(string) - the incoming port
preference(int) - User preference for ordering

Firewall yaml example

apiVersion: netlox/v1
kind: Firewall
metadata:
  name: test
spec:
  ruleArguments:
    sourceIP: 192.169.1.2/24
    destinationIP: 192.169.2.1/24
    preference: 200
  opts:
    allow: true

Delete Firewall information

#loxicmd delete firewall --firewallRule=<ruleKey>:<ruleValue>
loxicmd delete firewall --firewallRule="sourceIP:1.2.3.2/32,destinationIP:2.3.1.2/32,preference:200

---

Mirror

Get Mirror information

loxicmd get mirror

Create Mirror information

#loxicmd create mirror <mirrorIdent> --mirrorInfo=<InfoOption>:<InfoValue>,... --targetObject=attachement:<port1,rule2>,mirrObjName:<ObjectName>
loxicmd create mirror mirr-1 --mirrorInfo="type:0,port:ensp0" --targetObject="attachement:1,mirrObjName:ensp1

mirrorIdent(string): Mirror identifier
type(int) : Mirroring type as like 0 == SPAN, 1 == RSPAN, 2 == ERSPAN
port(string) : The port where mirrored traffic needs to be sent
vlan(int) : for RSPAN we may need to send tagged mirror traffic
remoteIP(string) : For ERSPAN we may need to send tunnelled mirror traffic
sourceIP(string): For ERSPAN we may need to send tunnelled mirror traffic
tunnelID(int): For ERSPAN we may need to send tunnelled mirror traffic

Mirror yaml example

apiVersion: netlox/v1
kind: Mirror
metadata:
  name: test
spec:
  mirrorIdent: mirr-1
  mirrorInfo:
    type: 0
    port: eno1
  targetObject:
    attachment: 1
    mirrObjName: eno2

Delete Mirror information

#loxicmd delete mirror <mirrorIdent>
loxicmd delete mirror mirr-1

---

Policy

Get Policy information

loxicmd get policy

Create Policy information

#loxicmd create policy IDENT --rate=<Peak>:<Commited> --target=<ObjectName>:<Attachment> [--block-size=<Excess>:<Committed>] [--color] [--pol-type=<policy type>]
loxicmd create policy pol-0 --rate=100:100 --target=ensp0:1
loxicmd create policy pol-1 --rate=100:100 --target=ensp0:1 --block-size=12000:6000
loxicmd create policy pol-1 --rate=100:100 --target=ensp0:1 --color
loxicmd create policy pol-1 --rate=100:100 --target=ensp0:1 --color --pol-type 0

rate(string): Rate pairs can be specified as 'Peak:Commited'. rate unit : Mbps
block-size(string): Block Size pairs can be specified as 'Excess:Committed'. block-size unit : bps
target(string): Target Interface pairs can be specified as 'ObjectName:Attachment'
color(boolean): Policy color enbale or not
pol-type(int): Policy traffic control type. 0 : TrTCM, 1 : SrTCM

Policy yaml example

apiVersion: netlox/v1
kind: Policy
metadata:
  name: test
spec:
  policyIdent: pol-eno8
  policyInfo:
    type: 0
    colorAware: false
    committedInfoRate: 100
    peakInfoRate: 100
  targetObject:
    attachment: 1
    polObjName: eno8

Delete Policy information

#loxicmd delete policy <Polident>
loxicmd delete policy pol-1

---

Session Recorder

Set n-tuple policy for recording

loxicmd create firewall --firewallRule="destinationIP:31.31.31.0/24,preference:200" --allow --record

loxilb will record any connection track entry which matches this policy (even for reverse direction) as a way to provide extended visibility for debugging

Check or record with tcpdump

tcpdump -i llb0 -n

Any valid tcpdump option can be used including saving to a pcap file

---

Get live connection-track information

loxicmd get conntrack

Get port-dump information

loxicmd get port

Save all loxilb's operational information in DBStore

loxicmd save -a

** This will ensure that whenever loxilb restarts, it will start with last saved state from DBStore

Configure loxicmd with yaml(Beta)

The loxicmd support yaml based configuration. The format is same as Kubernetes. This beta version support only one configuraion per one file. That means "Do not use --- in yaml file." . It will be supported at next release.

Command

#loxicmd apply -f <file.yaml>
#loxicmd delete -f <file.yaml>
loxicmd apply -f lb.yaml
loxicmd delete -f lb.yaml

File example(lb.yaml)

apiVersion: netlox/v1
kind: Loadbalancer
metadata:
  name: load
spec:
  serviceArguments:
    externalIP: 123.123.123.1
    port: 80
    protocol: tcp
    sel: 0
  endpoints:
  - endpointIP: 4.3.2.1
    weight: 1
    targetPort: 8080
  - endpointIP: 4.3.2.2
    weight: 1
    targetPort: 8080
  - endpointIP: 4.3.2.3
    weight: 1
    targetPort: 8080

It reuse API's json body as a "Spec". If the API URL has no param, it don't need to use "metadata". For example, The body of load Balancer rule is shown below.

{
  "serviceArguments": {
    "externalIP": "123.123.123.1",
    "port": 80,
    "protocol": "tcp",
    "sel": 0
  },
  "endpoints": [
    {
      "endpointIP": "4.3.2.1",
      "weight": 1,
      "targetPort": 8080
    },
    {
      "endpointIP": "4.3.2.2",
      "weight": 1,
      "targetPort": 8080
    },
    {
      "endpointIP": "4.3.2.3",
      "weight": 1,
      "targetPort": 8080
    }
}

This json format can be converted Yaml format as shown below.

  serviceArguments:
    externalIP: 123.123.123.1
    port: 80
    protocol: tcp
    sel: 0
  endpoints:
  - endpointIP: 4.3.2.1
    weight: 1
    targetPort: 8080
  - endpointIP: 4.3.2.2
    weight: 1
    targetPort: 8080
  - endpointIP: 4.3.2.3
    weight: 1
    targetPort: 8080

Finally, this is located in the Spec of the entire configuration file as File example(lb.yaml)

If you want to add Vlan bridge, IPaddress or something else. Just change the Kind value from Loadbalancer to VlanBridge, IPaddress as like below example.

apiVersion: netlox/v1
kind: IPaddress
metadata:
  name: test
spec:
  dev: eno8
  ipAddress: 192.168.23.1/32

If the URL has param such as adding vlan-member, it must have metadata.

apiVersion: netlox/v1
kind: VlanMember
metadata:
  name: test
  vid: 100
spec:
  dev: eno8
  Tagged: true

The example of all the settings below, so please refer to it.

More information

There are tons of other commands, use help option!

loxicmd help