2 Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Intel Corporation nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31 Link Bonding Poll Mode Driver Library
32 =====================================
34 In addition to Poll Mode Drivers (PMDs) for physical and virtual hardware,
35 DPDK also includes a pure-software library that
36 allows physical PMD's to be bonded together to create a single logical PMD.
40 The Link Bonding PMD library(librte_pmd_bond) supports bonding of groups of
41 ``rte_eth_dev`` ports of the same speed and duplex to provide
42 similar the capabilities to that found in Linux bonding driver to allow the
43 aggregation of multiple (slave) NICs into a single logical interface between a
44 server and a switch. The new bonded PMD will then process these interfaces
45 based on the mode of operation specified to provide support for features such
46 as redundant links, fault tolerance and/or load balancing.
48 The librte_pmd_bond library exports a C API which provides an API for the
49 creation of bonded devices as well as the configuration and management of the
50 bonded device and its slave devices.
54 The Link Bonding PMD Library is enabled by default in the build
55 configuration files, the library can be disabled by setting
56 ``CONFIG_RTE_LIBRTE_PMD_BOND=n`` and recompiling the DPDK.
58 Link Bonding Modes Overview
59 ---------------------------
61 Currently the Link Bonding PMD library supports 4 modes of operation:
63 * **Round-Robin (Mode 0):**
67 This mode provides load balancing and fault tolerance by transmission of
68 packets in sequential order from the first available slave device through
69 the last. Packets are bulk dequeued from devices then serviced in a
70 round-robin manner. This mode does not guarantee in order reception of
71 packets and down stream should be able to handle out of order packets.
73 * **Active Backup (Mode 1):**
77 In this mode only one slave in the bond is active at any time, a different
78 slave becomes active if, and only if, the primary active slave fails,
79 thereby providing fault tolerance to slave failure. The single logical
80 bonded interface's MAC address is externally visible on only one NIC (port)
81 to avoid confusing the network switch.
83 * **Balance XOR (Mode 2):**
87 This mode provides transmit load balancing (based on the selected
88 transmission policy) and fault tolerance. The default policy (layer2) uses
89 a simple calculation based on the packet flow source and destination MAC
90 addresses as well as the number of active slaves available to the bonded
91 device to classify the packet to a specific slave to transmit on. Alternate
92 transmission policies supported are layer 2+3, this takes the IP source and
93 destination addresses into the calculation of the transmit slave port and
94 the final supported policy is layer 3+4, this uses IP source and
95 destination addresses as well as the TCP/UDP source and destination port.
98 The coloring differences of the packets are used to identify different flow
99 classification calculated by the selected transmit policy
102 * **Broadcast (Mode 3):**
106 This mode provides fault tolerance by transmission of packets on all slave
109 * **Link Aggregation 802.3AD (Mode 4):**
113 This mode provides dynamic link aggregation according to the 802.3ad
114 specification. It negotiates and monitors aggregation groups that share the
115 same speed and duplex settings using the selected balance transmit policy
116 for balancing outgoing traffic.
118 DPDK implementation of this mode provide some additional requirements of
121 #. It needs to call ``rte_eth_tx_burst`` and ``rte_eth_rx_burst`` with
122 intervals period of less than 100ms.
124 #. Calls to ``rte_eth_tx_burst`` must have a buffer size of at least 2xN,
125 where N is the number of slaves. This is a space required for LACP
126 frames. Additionally LACP packets are included in the statistics, but
127 they are not returned to the application.
129 * **Transmit Load Balancing (Mode 5):**
133 This mode provides an adaptive transmit load balancing. It dynamically
134 changes the transmitting slave, according to the computed load. Statistics
135 are collected in 100ms intervals and scheduled every 10ms.
138 Implementation Details
139 ----------------------
141 The librte_pmd_bond bonded device are compatible with the Ethernet device API
142 exported by the Ethernet PMDs described in the *DPDK API Reference*.
144 The Link Bonding Library supports the creation of bonded devices at application
145 startup time during EAL initialization using the ``--vdev`` option as well as
146 programmatically via the C API ``rte_eth_bond_create`` function.
148 Bonded devices support the dynamical addition and removal of slave devices using
149 the ``rte_eth_bond_slave_add`` / ``rte_eth_bond_slave_remove`` APIs.
151 After a slave device is added to a bonded device slave is stopped using
152 ``rte_eth_dev_stop`` and then reconfigured using ``rte_eth_dev_configure``
153 the RX and TX queues are also reconfigured using ``rte_eth_tx_queue_setup`` /
154 ``rte_eth_rx_queue_setup`` with the parameters use to configure the bonding
157 Link Status Change Interrupts / Polling
158 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
160 Link bonding devices support the registration of a link status change callback,
161 using the ``rte_eth_dev_callback_register`` API, this will be called when the
162 status of the bonding device changes. For example in the case of a bonding
163 device which has 3 slaves, the link status will change to up when one slave
164 becomes active or change to down when all slaves become inactive. There is no
165 callback notification when a single slave changes state and the previous
166 conditions are not met. If a user wishes to monitor individual slaves then they
167 must register callbacks with that slave directly.
169 The link bonding library also supports devices which do not implement link
170 status change interrupts, this is achieve by polling the devices link status at
171 a defined period which is set using the ``rte_eth_bond_link_monitoring_set``
172 API, the default polling interval is 10ms. When a device is added as a slave to
173 a bonding device it is determined using the ``RTE_PCI_DRV_INTR_LSC`` flag
174 whether the device supports interrupts or whether the link status should be
175 monitored by polling it.
177 Requirements / Limitations
178 ~~~~~~~~~~~~~~~~~~~~~~~~~~
180 The current implementation only supports devices that support the same speed
181 and duplex to be added as a slaves to the same bonded device. The bonded device
182 inherits these attributes from the first active slave added to the bonded
183 device and then all further slaves added to the bonded device must support
186 A bonding device must have a minimum of one slave before the bonding device
187 itself can be started.
189 Like all other PMD, all functions exported by a PMD are lock-free functions
190 that are assumed not to be invoked in parallel on different logical cores to
191 work on the same target object.
193 It should also be noted that the PMD receive function should not be invoked
194 directly on a slave devices after they have been to a bonded device since
195 packets read directly from the slave device will no longer be available to the
196 bonded device to read.
201 Link bonding devices are created using the ``rte_eth_bond_create`` API
202 which requires a unique device name, the bonding mode,
203 and the socket Id to allocate the bonding device's resources on.
204 The other configurable parameters for a bonded device are its slave devices,
205 its primary slave, a user defined MAC address and transmission policy to use if
206 the device is in balance XOR mode.
211 Bonding devices support up to a maximum of ``RTE_MAX_ETHPORTS`` slave devices
212 of the same speed and duplex. Ethernet devices can be added as a slave to a
213 maximum of one bonded device. Slave devices are reconfigured with the
214 configuration of the bonded device on being added to a bonded device.
216 The bonded also guarantees to return the MAC address of the slave device to its
217 original value of removal of a slave from it.
222 The primary slave is used to define the default port to use when a bonded
223 device is in active backup mode. A different port will only be used if, and
224 only if, the current primary port goes down. If the user does not specify a
225 primary port it will default to being the first port added to the bonded device.
230 The bonded device can be configured with a user specified MAC address, this
231 address will be inherited by the some/all slave devices depending on the
232 operating mode. If the device is in active backup mode then only the primary
233 device will have the user specified MAC, all other slaves will retain their
234 original MAC address. In mode 0, 2, 3, 4 all slaves devices are configure with
235 the bonded devices MAC address.
237 If a user defined MAC address is not defined then the bonded device will
238 default to using the primary slaves MAC address.
240 Balance XOR Transmit Policies
241 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
243 There are 3 supported transmission policies for bonded device running in
244 Balance XOR mode. Layer 2, Layer 2+3, Layer 3+4.
246 * **Layer 2:** Ethernet MAC address based balancing is the default
247 transmission policy for Balance XOR bonding mode. It uses a simple XOR
248 calculation on the source MAC address and destination MAC address of the
249 packet and then calculate the modulus of this value to calculate the slave
250 device to transmit the packet on.
252 * **Layer 2 + 3:** Ethernet MAC address & IP Address based balancing uses a
253 combination of source/destination MAC addresses and the source/destination
254 IP addresses of the data packet to decide which slave port the packet will
257 * **Layer 3 + 4:** IP Address & UDP Port based balancing uses a combination
258 of source/destination IP Address and the source/destination UDP ports of
259 the packet of the data packet to decide which slave port the packet will be
262 All these policies support 802.1Q VLAN Ethernet packets, as well as IPv4, IPv6
263 and UDP protocols for load balancing.
265 Using Link Bonding Devices
266 --------------------------
268 The librte_pmd_bond library support two modes of device creation, the libraries
269 export full C API or using the EAL command line to statically configure link
270 bonding devices at application startup. Using the EAL option it is possible to
271 use link bonding functionality transparently without specific knowledge of the
272 libraries API, this can be used, for example, to add bonding functionality,
273 such as active backup, to an existing application which has no knowledge of
274 the link bonding C API.
276 Using the Poll Mode Driver from an Application
277 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
279 Using the librte_pmd_bond libraries API it is possible to dynamically create
280 and manage link bonding device from within any application. Link bonding
281 device are created using the ``rte_eth_bond_create`` API which requires a
282 unique device name, the link bonding mode to initial the device in and finally
283 the socket Id which to allocate the devices resources onto. After successful
284 creation of a bonding device it must be configured using the generic Ethernet
285 device configure API ``rte_eth_dev_configure`` and then the RX and TX queues
286 which will be used must be setup using ``rte_eth_tx_queue_setup`` /
287 ``rte_eth_rx_queue_setup``.
289 Slave devices can be dynamically added and removed from a link bonding device
290 using the ``rte_eth_bond_slave_add`` / ``rte_eth_bond_slave_remove``
291 APIs but at least one slave device must be added to the link bonding device
292 before it can be started using ``rte_eth_dev_start``.
294 The link status of a bonded device is dictated by that of its slaves, if all
295 slave device link status are down or if all slaves are removed from the link
296 bonding device then the link status of the bonding device will go down.
298 It is also possible to configure / query the configuration of the control
299 parameters of a bonded device using the provided APIs
300 ``rte_eth_bond_mode_set/ get``, ``rte_eth_bond_primary_set/get``,
301 ``rte_eth_bond_mac_set/reset`` and ``rte_eth_bond_xmit_policy_set/get``.
303 Using Link Bonding Devices from the EAL Command Line
304 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
306 Link bonding devices can be created at application startup time using the
307 ``--vdev`` EAL command line option. The device name must start with the
308 eth_bond prefix followed by numbers or letters. The name must be unique for
309 each device. Each device can have multiple options arranged in a comma
310 separated list. Multiple devices definitions can be arranged by calling the
311 ``--vdev`` option multiple times.
313 Device names and bonding options must be separated by commas as shown below:
315 .. code-block:: console
317 $RTE_TARGET/app/testpmd -c f -n 4 --vdev 'eth_bond0,bond_opt0=..,bond opt1=..'--vdev 'eth_bond1,bond _opt0=..,bond_opt1=..'
319 Link Bonding EAL Options
320 ^^^^^^^^^^^^^^^^^^^^^^^^
322 There are multiple ways of definitions that can be assessed and combined as
323 long as the following two rules are respected:
325 * A unique device name, in the format of eth_bondX is provided,
326 where X can be any combination of numbers and/or letters,
327 and the name is no greater than 32 characters long.
329 * A least one slave device is provided with for each bonded device definition.
331 * The operation mode of the bonded device being created is provided.
333 The different options are:
335 * mode: Integer value defining the bonding mode of the device.
336 Currently supports modes 0,1,2,3,4,5 (round-robin, active backup, balance,
337 broadcast, link aggregation, transmit load balancing).
339 .. code-block:: console
343 * slave: Defines the PMD device which will be added as slave to the bonded
344 device. This option can be selected multiple time, for each device to be
345 added as a slave. Physical devices should be specified using their PCI
346 address, in the format domain:bus:devid.function
348 .. code-block:: console
350 slave=0000:0a:00.0,slave=0000:0a:00.1
352 * primary: Optional parameter which defines the primary slave port,
353 is used in active backup mode to select the primary slave for data TX/RX if
354 it is available. The primary port also is used to select the MAC address to
355 use when it is not defined by the user. This defaults to the first slave
356 added to the device if it is specified. The primary device must be a slave
357 of the bonded device.
359 .. code-block:: console
363 * socket_id: Optional parameter used to select which socket on a NUMA device
364 the bonded devices resources will be allocated on.
366 .. code-block:: console
370 * mac: Optional parameter to select a MAC address for link bonding device,
371 this overrides the value of the primary slave device.
373 .. code-block:: console
375 mac=00:1e:67:1d:fd:1d
377 * xmit_policy: Optional parameter which defines the transmission policy when
378 the bonded device is in balance mode. If not user specified this defaults
379 to l2 (layer 2) forwarding, the other transmission policies available are
380 l23 (layer 2+3) and l34 (layer 3+4)
382 .. code-block:: console
386 * lsc_poll_period_ms: Optional parameter which defines the polling interval
387 in milli-seconds at which devices which don't support lsc interrupts are
388 checked for a change in the devices link status
390 .. code-block:: console
392 lsc_poll_period_ms=100
394 * up_delay: Optional parameter which adds a delay in milli-seconds to the
395 propagation of a devices link status changing to up, by default this
398 .. code-block:: console
402 * down_delay: Optional parameter which adds a delay in milli-seconds to the
403 propagation of a devices link status changing to down, by default this
406 .. code-block:: console
413 Create a bonded device in round robin mode with two slaves specified by their PCI address:
415 .. code-block:: console
417 $RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=0, slave=0000:00a:00.01,slave=0000:004:00.00' -- --port-topology=chained
419 Create a bonded device in round robin mode with two slaves specified by their PCI address and an overriding MAC address:
421 .. code-block:: console
423 $RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=0, slave=0000:00a:00.01,slave=0000:004:00.00,mac=00:1e:67:1d:fd:1d' -- --port-topology=chained
425 Create a bonded device in active backup mode with two slaves specified, and a primary slave specified by their PCI addresses:
427 .. code-block:: console
429 $RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=1, slave=0000:00a:00.01,slave=0000:004:00.00,primary=0000:00a:00.01' -- --port-topology=chained
431 Create a bonded device in balance mode with two slaves specified by their PCI addresses, and a transmission policy of layer 3 + 4 forwarding:
433 .. code-block:: console
435 $RTE_TARGET/app/testpmd -c '0xf' -n 4 --vdev 'eth_bond0,mode=2, slave=0000:00a:00.01,slave=0000:004:00.00,xmit_policy=l34' -- --port-topology=chained
437 .. |bond-overview| image:: img/bond-overview.*
438 .. |bond-mode-0| image:: img/bond-mode-0.*
439 .. |bond-mode-1| image:: img/bond-mode-1.*
440 .. |bond-mode-2| image:: img/bond-mode-2.*
441 .. |bond-mode-3| image:: img/bond-mode-3.*
442 .. |bond-mode-4| image:: img/bond-mode-4.*
443 .. |bond-mode-5| image:: img/bond-mode-5.*