1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2010-2015 Intel Corporation.
4 Link Bonding Poll Mode Driver Library
5 =====================================
7 In addition to Poll Mode Drivers (PMDs) for physical and virtual hardware,
8 DPDK also includes a pure-software library that
9 allows physical PMDs to be bonded together to create a single logical PMD.
11 .. figure:: img/bond-overview.*
16 The Link Bonding PMD library(librte_net_bond) supports bonding of groups of
17 ``rte_eth_dev`` ports of the same speed and duplex to provide similar
18 capabilities to that found in Linux bonding driver to allow the aggregation
19 of multiple (slave) NICs into a single logical interface between a server
20 and a switch. The new bonded PMD will then process these interfaces based on
21 the mode of operation specified to provide support for features such as
22 redundant links, fault tolerance and/or load balancing.
24 The librte_net_bond library exports a C API which provides an API for the
25 creation of bonded devices as well as the configuration and management of the
26 bonded device and its slave devices.
30 The Link Bonding PMD Library is enabled by default in the build
31 configuration, the library can be disabled using the meson option
32 "-Ddisable_drivers=net/bond".
35 Link Bonding Modes Overview
36 ---------------------------
38 Currently the Link Bonding PMD library supports following modes of operation:
40 * **Round-Robin (Mode 0):**
42 .. figure:: img/bond-mode-0.*
47 This mode provides load balancing and fault tolerance by transmission of
48 packets in sequential order from the first available slave device through
49 the last. Packets are bulk dequeued from devices then serviced in a
50 round-robin manner. This mode does not guarantee in order reception of
51 packets and down stream should be able to handle out of order packets.
53 * **Active Backup (Mode 1):**
55 .. figure:: img/bond-mode-1.*
57 Active Backup (Mode 1)
60 In this mode only one slave in the bond is active at any time, a different
61 slave becomes active if, and only if, the primary active slave fails,
62 thereby providing fault tolerance to slave failure. The single logical
63 bonded interface's MAC address is externally visible on only one NIC (port)
64 to avoid confusing the network switch.
66 * **Balance XOR (Mode 2):**
68 .. figure:: img/bond-mode-2.*
73 This mode provides transmit load balancing (based on the selected
74 transmission policy) and fault tolerance. The default policy (layer2) uses
75 a simple calculation based on the packet flow source and destination MAC
76 addresses as well as the number of active slaves available to the bonded
77 device to classify the packet to a specific slave to transmit on. Alternate
78 transmission policies supported are layer 2+3, this takes the IP source and
79 destination addresses into the calculation of the transmit slave port and
80 the final supported policy is layer 3+4, this uses IP source and
81 destination addresses as well as the TCP/UDP source and destination port.
84 The coloring differences of the packets are used to identify different flow
85 classification calculated by the selected transmit policy
88 * **Broadcast (Mode 3):**
90 .. figure:: img/bond-mode-3.*
95 This mode provides fault tolerance by transmission of packets on all slave
98 * **Link Aggregation 802.3AD (Mode 4):**
100 .. figure:: img/bond-mode-4.*
102 Link Aggregation 802.3AD (Mode 4)
105 This mode provides dynamic link aggregation according to the 802.3ad
106 specification. It negotiates and monitors aggregation groups that share the
107 same speed and duplex settings using the selected balance transmit policy
108 for balancing outgoing traffic.
110 DPDK implementation of this mode provide some additional requirements of
113 #. It needs to call ``rte_eth_tx_burst`` and ``rte_eth_rx_burst`` with
114 intervals period of less than 100ms.
116 #. Calls to ``rte_eth_tx_burst`` must have a buffer size of at least 2xN,
117 where N is the number of slaves. This is a space required for LACP
118 frames. Additionally LACP packets are included in the statistics, but
119 they are not returned to the application.
121 * **Transmit Load Balancing (Mode 5):**
123 .. figure:: img/bond-mode-5.*
125 Transmit Load Balancing (Mode 5)
128 This mode provides an adaptive transmit load balancing. It dynamically
129 changes the transmitting slave, according to the computed load. Statistics
130 are collected in 100ms intervals and scheduled every 10ms.
133 Implementation Details
134 ----------------------
136 The librte_net_bond bonded device are compatible with the Ethernet device API
137 exported by the Ethernet PMDs described in the *DPDK API Reference*.
139 The Link Bonding Library supports the creation of bonded devices at application
140 startup time during EAL initialization using the ``--vdev`` option as well as
141 programmatically via the C API ``rte_eth_bond_create`` function.
143 Bonded devices support the dynamical addition and removal of slave devices using
144 the ``rte_eth_bond_slave_add`` / ``rte_eth_bond_slave_remove`` APIs.
146 After a slave device is added to a bonded device slave is stopped using
147 ``rte_eth_dev_stop`` and then reconfigured using ``rte_eth_dev_configure``
148 the RX and TX queues are also reconfigured using ``rte_eth_tx_queue_setup`` /
149 ``rte_eth_rx_queue_setup`` with the parameters use to configure the bonding
150 device. If RSS is enabled for bonding device, this mode is also enabled on new
151 slave and configured as well.
152 Any flow which was configured to the bond device also is configured to the added
155 Setting up multi-queue mode for bonding device to RSS, makes it fully
156 RSS-capable, so all slaves are synchronized with its configuration. This mode is
157 intended to provide RSS configuration on slaves transparent for client
158 application implementation.
160 Bonding device stores its own version of RSS settings i.e. RETA, RSS hash
161 function and RSS key, used to set up its slaves. That let to define the meaning
162 of RSS configuration of bonding device as desired configuration of whole bonding
163 (as one unit), without pointing any of slave inside. It is required to ensure
164 consistency and made it more error-proof.
166 RSS hash function set for bonding device, is a maximal set of RSS hash functions
167 supported by all bonded slaves. RETA size is a GCD of all its RETA's sizes, so
168 it can be easily used as a pattern providing expected behavior, even if slave
169 RETAs' sizes are different. If RSS Key is not set for bonded device, it's not
170 changed on the slaves and default key for device is used.
172 As RSS configurations, there is flow consistency in the bonded slaves for the
173 next rte flow operations:
176 - Validate flow for each slave, failure at least for one slave causes to
177 bond validation failure.
180 - Create the flow in all slaves.
181 - Save all the slaves created flows objects in bonding internal flow
183 - Failure in flow creation for existed slave rejects the flow.
184 - Failure in flow creation for new slaves in slave adding time rejects
188 - Destroy the flow in all slaves and release the bond internal flow
192 - Destroy all the bonding PMD flows in all the slaves.
196 Don't call slaves flush directly, It destroys all the slave flows which
197 may include external flows or the bond internal LACP flow.
200 - Summarize flow counters from all the slaves, relevant only for
201 ``RTE_FLOW_ACTION_TYPE_COUNT``.
204 - Call to flow isolate for all slaves.
205 - Failure in flow isolation for existed slave rejects the isolate mode.
206 - Failure in flow isolation for new slaves in slave adding time rejects
209 All settings are managed through the bonding port API and always are propagated
210 in one direction (from bonding to slaves).
212 Link Status Change Interrupts / Polling
213 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
215 Link bonding devices support the registration of a link status change callback,
216 using the ``rte_eth_dev_callback_register`` API, this will be called when the
217 status of the bonding device changes. For example in the case of a bonding
218 device which has 3 slaves, the link status will change to up when one slave
219 becomes active or change to down when all slaves become inactive. There is no
220 callback notification when a single slave changes state and the previous
221 conditions are not met. If a user wishes to monitor individual slaves then they
222 must register callbacks with that slave directly.
224 The link bonding library also supports devices which do not implement link
225 status change interrupts, this is achieved by polling the devices link status at
226 a defined period which is set using the ``rte_eth_bond_link_monitoring_set``
227 API, the default polling interval is 10ms. When a device is added as a slave to
228 a bonding device it is determined using the ``RTE_PCI_DRV_INTR_LSC`` flag
229 whether the device supports interrupts or whether the link status should be
230 monitored by polling it.
232 Requirements / Limitations
233 ~~~~~~~~~~~~~~~~~~~~~~~~~~
235 The current implementation only supports devices that support the same speed
236 and duplex to be added as a slaves to the same bonded device. The bonded device
237 inherits these attributes from the first active slave added to the bonded
238 device and then all further slaves added to the bonded device must support
241 A bonding device must have a minimum of one slave before the bonding device
242 itself can be started.
244 To use a bonding device dynamic RSS configuration feature effectively, it is
245 also required, that all slaves should be RSS-capable and support, at least one
246 common hash function available for each of them. Changing RSS key is only
247 possible, when all slave devices support the same key size.
249 To prevent inconsistency on how slaves process packets, once a device is added
250 to a bonding device, RSS and rte flow configurations should be managed through
251 the bonding device API, and not directly on the slave.
253 Like all other PMD, all functions exported by a PMD are lock-free functions
254 that are assumed not to be invoked in parallel on different logical cores to
255 work on the same target object.
257 It should also be noted that the PMD receive function should not be invoked
258 directly on a slave devices after they have been to a bonded device since
259 packets read directly from the slave device will no longer be available to the
260 bonded device to read.
265 Link bonding devices are created using the ``rte_eth_bond_create`` API
266 which requires a unique device name, the bonding mode,
267 and the socket Id to allocate the bonding device's resources on.
268 The other configurable parameters for a bonded device are its slave devices,
269 its primary slave, a user defined MAC address and transmission policy to use if
270 the device is in balance XOR mode.
275 Bonding devices support up to a maximum of ``RTE_MAX_ETHPORTS`` slave devices
276 of the same speed and duplex. Ethernet devices can be added as a slave to a
277 maximum of one bonded device. Slave devices are reconfigured with the
278 configuration of the bonded device on being added to a bonded device.
280 The bonded also guarantees to return the MAC address of the slave device to its
281 original value of removal of a slave from it.
286 The primary slave is used to define the default port to use when a bonded
287 device is in active backup mode. A different port will only be used if, and
288 only if, the current primary port goes down. If the user does not specify a
289 primary port it will default to being the first port added to the bonded device.
294 The bonded device can be configured with a user specified MAC address, this
295 address will be inherited by the some/all slave devices depending on the
296 operating mode. If the device is in active backup mode then only the primary
297 device will have the user specified MAC, all other slaves will retain their
298 original MAC address. In mode 0, 2, 3, 4 all slaves devices are configure with
299 the bonded devices MAC address.
301 If a user defined MAC address is not defined then the bonded device will
302 default to using the primary slaves MAC address.
304 Balance XOR Transmit Policies
305 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
307 There are 3 supported transmission policies for bonded device running in
308 Balance XOR mode. Layer 2, Layer 2+3, Layer 3+4.
310 * **Layer 2:** Ethernet MAC address based balancing is the default
311 transmission policy for Balance XOR bonding mode. It uses a simple XOR
312 calculation on the source MAC address and destination MAC address of the
313 packet and then calculate the modulus of this value to calculate the slave
314 device to transmit the packet on.
316 * **Layer 2 + 3:** Ethernet MAC address & IP Address based balancing uses a
317 combination of source/destination MAC addresses and the source/destination
318 IP addresses of the data packet to decide which slave port the packet will
321 * **Layer 3 + 4:** IP Address & UDP Port based balancing uses a combination
322 of source/destination IP Address and the source/destination UDP ports of
323 the packet of the data packet to decide which slave port the packet will be
326 All these policies support 802.1Q VLAN Ethernet packets, as well as IPv4, IPv6
327 and UDP protocols for load balancing.
329 Using Link Bonding Devices
330 --------------------------
332 The librte_net_bond library supports two modes of device creation, the libraries
333 export full C API or using the EAL command line to statically configure link
334 bonding devices at application startup. Using the EAL option it is possible to
335 use link bonding functionality transparently without specific knowledge of the
336 libraries API, this can be used, for example, to add bonding functionality,
337 such as active backup, to an existing application which has no knowledge of
338 the link bonding C API.
340 Using the Poll Mode Driver from an Application
341 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
343 Using the librte_net_bond libraries API it is possible to dynamically create
344 and manage link bonding device from within any application. Link bonding
345 devices are created using the ``rte_eth_bond_create`` API which requires a
346 unique device name, the link bonding mode to initial the device in and finally
347 the socket Id which to allocate the devices resources onto. After successful
348 creation of a bonding device it must be configured using the generic Ethernet
349 device configure API ``rte_eth_dev_configure`` and then the RX and TX queues
350 which will be used must be setup using ``rte_eth_tx_queue_setup`` /
351 ``rte_eth_rx_queue_setup``.
353 Slave devices can be dynamically added and removed from a link bonding device
354 using the ``rte_eth_bond_slave_add`` / ``rte_eth_bond_slave_remove``
355 APIs but at least one slave device must be added to the link bonding device
356 before it can be started using ``rte_eth_dev_start``.
358 The link status of a bonded device is dictated by that of its slaves, if all
359 slave device link status are down or if all slaves are removed from the link
360 bonding device then the link status of the bonding device will go down.
362 It is also possible to configure / query the configuration of the control
363 parameters of a bonded device using the provided APIs
364 ``rte_eth_bond_mode_set/ get``, ``rte_eth_bond_primary_set/get``,
365 ``rte_eth_bond_mac_set/reset`` and ``rte_eth_bond_xmit_policy_set/get``.
367 Using Link Bonding Devices from the EAL Command Line
368 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
370 Link bonding devices can be created at application startup time using the
371 ``--vdev`` EAL command line option. The device name must start with the
372 net_bonding prefix followed by numbers or letters. The name must be unique for
373 each device. Each device can have multiple options arranged in a comma
374 separated list. Multiple devices definitions can be arranged by calling the
375 ``--vdev`` option multiple times.
377 Device names and bonding options must be separated by commas as shown below:
379 .. code-block:: console
381 ./<build_dir>/app/dpdk-testpmd -l 0-3 -n 4 --vdev 'net_bonding0,bond_opt0=..,bond opt1=..'--vdev 'net_bonding1,bond _opt0=..,bond_opt1=..'
383 Link Bonding EAL Options
384 ^^^^^^^^^^^^^^^^^^^^^^^^
386 There are multiple ways of definitions that can be assessed and combined as
387 long as the following two rules are respected:
389 * A unique device name, in the format of net_bondingX is provided,
390 where X can be any combination of numbers and/or letters,
391 and the name is no greater than 32 characters long.
393 * A least one slave device is provided with for each bonded device definition.
395 * The operation mode of the bonded device being created is provided.
397 The different options are:
399 * mode: Integer value defining the bonding mode of the device.
400 Currently supports modes 0,1,2,3,4,5 (round-robin, active backup, balance,
401 broadcast, link aggregation, transmit load balancing).
403 .. code-block:: console
407 * slave: Defines the PMD device which will be added as slave to the bonded
408 device. This option can be selected multiple times, for each device to be
409 added as a slave. Physical devices should be specified using their PCI
410 address, in the format domain:bus:devid.function
412 .. code-block:: console
414 slave=0000:0a:00.0,slave=0000:0a:00.1
416 * primary: Optional parameter which defines the primary slave port,
417 is used in active backup mode to select the primary slave for data TX/RX if
418 it is available. The primary port also is used to select the MAC address to
419 use when it is not defined by the user. This defaults to the first slave
420 added to the device if it is specified. The primary device must be a slave
421 of the bonded device.
423 .. code-block:: console
427 * socket_id: Optional parameter used to select which socket on a NUMA device
428 the bonded devices resources will be allocated on.
430 .. code-block:: console
434 * mac: Optional parameter to select a MAC address for link bonding device,
435 this overrides the value of the primary slave device.
437 .. code-block:: console
439 mac=00:1e:67:1d:fd:1d
441 * xmit_policy: Optional parameter which defines the transmission policy when
442 the bonded device is in balance mode. If not user specified this defaults
443 to l2 (layer 2) forwarding, the other transmission policies available are
444 l23 (layer 2+3) and l34 (layer 3+4)
446 .. code-block:: console
450 * lsc_poll_period_ms: Optional parameter which defines the polling interval
451 in milli-seconds at which devices which don't support lsc interrupts are
452 checked for a change in the devices link status
454 .. code-block:: console
456 lsc_poll_period_ms=100
458 * up_delay: Optional parameter which adds a delay in milli-seconds to the
459 propagation of a devices link status changing to up, by default this
462 .. code-block:: console
466 * down_delay: Optional parameter which adds a delay in milli-seconds to the
467 propagation of a devices link status changing to down, by default this
470 .. code-block:: console
477 Create a bonded device in round robin mode with two slaves specified by their PCI address:
479 .. code-block:: console
481 ./<build_dir>/app/dpdk-testpmd -l 0-3 -n 4 --vdev 'net_bonding0,mode=0,slave=0000:0a:00.01,slave=0000:04:00.00' -- --port-topology=chained
483 Create a bonded device in round robin mode with two slaves specified by their PCI address and an overriding MAC address:
485 .. code-block:: console
487 ./<build_dir>/app/dpdk-testpmd -l 0-3 -n 4 --vdev 'net_bonding0,mode=0,slave=0000:0a:00.01,slave=0000:04:00.00,mac=00:1e:67:1d:fd:1d' -- --port-topology=chained
489 Create a bonded device in active backup mode with two slaves specified, and a primary slave specified by their PCI addresses:
491 .. code-block:: console
493 ./<build_dir>/app/dpdk-testpmd -l 0-3 -n 4 --vdev 'net_bonding0,mode=1,slave=0000:0a:00.01,slave=0000:04:00.00,primary=0000:0a:00.01' -- --port-topology=chained
495 Create a bonded device in balance mode with two slaves specified by their PCI addresses, and a transmission policy of layer 3 + 4 forwarding:
497 .. code-block:: console
499 ./<build_dir>/app/dpdk-testpmd -l 0-3 -n 4 --vdev 'net_bonding0,mode=2,slave=0000:0a:00.01,slave=0000:04:00.00,xmit_policy=l34' -- --port-topology=chained