2 Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Intel Corporation nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31 Link Status Interrupt Sample Application
32 ========================================
34 The Link Status Interrupt sample application is a simple example of packet processing using
35 the Data Plane Development Kit (DPDK) that
36 demonstrates how network link status changes for a network port can be captured and
37 used by a DPDK application.
42 The Link Status Interrupt sample application registers a user space callback for the link status interrupt of each port
43 and performs L2 forwarding for each packet that is received on an RX_PORT.
44 The following operations are performed:
46 * RX_PORT and TX_PORT are paired with available ports one-by-one according to the core mask
48 * The source MAC address is replaced by the TX_PORT MAC address
50 * The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID
52 This application can be used to demonstrate the usage of link status interrupt and its user space callbacks
53 and the behavior of L2 forwarding each time the link status changes.
55 Compiling the Application
56 -------------------------
58 #. Go to the example directory:
60 .. code-block:: console
62 export RTE_SDK=/path/to/rte_sdk
63 cd ${RTE_SDK}/examples/link_status_interrupt
65 #. Set the target (a default target is used if not specified). For example:
67 .. code-block:: console
69 export RTE_TARGET=x86_64-native-linuxapp-gcc
71 See the *DPDK Getting Started Guide* for possible RTE_TARGET values.
73 #. Build the application:
75 .. code-block:: console
81 The compiled application is written to the build subdirectory.
82 To have the application written to a different location,
83 the O=/path/to/build/directory option may be specified on the make command line.
85 Running the Application
86 -----------------------
88 The application requires a number of command line options:
90 .. code-block:: console
92 ./build/link_status_interrupt [EAL options] -- -p PORTMASK [-q NQ][-T PERIOD]
96 * -p PORTMASK: A hexadecimal bitmask of the ports to configure
98 * -q NQ: A number of queues (=ports) per lcore (default is 1)
100 * -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default)
102 To run the application in a linuxapp environment with 4 lcores, 4 memory channels, 16 ports and 8 RX queues per lcore,
105 .. code-block:: console
107 $ ./build/link_status_interrupt -c f -n 4-- -q 8 -p ffff
109 Refer to the *DPDK Getting Started Guide* for general information on running applications
110 and the Environment Abstraction Layer (EAL) options.
115 The following sections provide some explanation of the code.
117 Command Line Arguments
118 ~~~~~~~~~~~~~~~~~~~~~~
120 The Link Status Interrupt sample application takes specific parameters,
121 in addition to Environment Abstraction Layer (EAL) arguments (see Section 13.3).
123 Command line parsing is done in the same way as it is done in the L2 Forwarding Sample Application.
124 See Section 9.4.1, "Command Line Arguments" for more information.
126 Mbuf Pool Initialization
127 ~~~~~~~~~~~~~~~~~~~~~~~~
129 Mbuf pool initialization is done in the same way as it is done in the L2 Forwarding Sample Application.
130 See Section 9.4.2, "Mbuf Pool Initialization" for more information.
132 Driver Initialization
133 ~~~~~~~~~~~~~~~~~~~~~
135 The main part of the code in the main() function relates to the initialization of the driver.
136 To fully understand this code, it is recommended to study the chapters that related to the Poll Mode Driver in the
137 *DPDK Programmer's Guide and the DPDK API Reference*.
141 if (rte_eal_pci_probe() < 0)
142 rte_exit(EXIT_FAILURE, "Cannot probe PCI\n");
144 nb_ports = rte_eth_dev_count();
146 rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
148 if (nb_ports > RTE_MAX_ETHPORTS)
149 nb_ports = RTE_MAX_ETHPORTS;
152 * Each logical core is assigned a dedicated TX queue on each port.
155 for (portid = 0; portid < nb_ports; portid++) {
156 /* skip ports that are not enabled */
158 if ((lsi_enabled_port_mask & (1 << portid)) == 0)
161 /* save the destination port id */
163 if (nb_ports_in_mask % 2) {
164 lsi_dst_ports[portid] = portid_last;
165 lsi_dst_ports[portid_last] = portid;
168 portid_last = portid;
172 rte_eth_dev_info_get((uint8_t) portid, &dev_info);
177 * rte_eal_pci_probe() parses the devices on the PCI bus and initializes recognized devices.
179 The next step is to configure the RX and TX queues.
180 For each port, there is only one RX queue (only one lcore is able to poll a given port).
181 The number of TX queues depends on the number of available lcores.
182 The rte_eth_dev_configure() function is used to configure the number of queues for a port:
186 ret = rte_eth_dev_configure((uint8_t) portid, 1, 1, &port_conf);
188 rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n", ret, portid);
190 The global configuration is stored in a static structure:
194 static const struct rte_eth_conf port_conf = {
197 .header_split = 0, /**< Header Split disabled */
198 .hw_ip_checksum = 0, /**< IP checksum offload disabled */
199 .hw_vlan_filter = 0, /**< VLAN filtering disabled */
200 .hw_strip_crc= 0, /**< CRC stripped by hardware */
204 .lsc = 1, /**< link status interrupt feature enabled */
208 Configuring lsc to 0 (the default) disables the generation of any link status change interrupts in kernel space
209 and no user space interrupt event is received.
210 The public interface rte_eth_link_get() accesses the NIC registers directly to update the link status.
211 Configuring lsc to non-zero enables the generation of link status change interrupts in kernel space
212 when a link status change is present and calls the user space callbacks registered by the application.
213 The public interface rte_eth_link_get() just reads the link status in a global structure
214 that would be updated in the interrupt host thread only.
216 Interrupt Callback Registration
217 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
219 The application can register one or more callbacks to a specific port and interrupt event.
220 An example callback function that has been written as indicated below.
225 lsi_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
227 struct rte_eth_link link;
231 printf("\n\nIn registered callback...\n");
233 printf("Event type: %s\n", type == RTE_ETH_EVENT_INTR_LSC ? "LSC interrupt" : "unknown event");
235 rte_eth_link_get_nowait(port_id, &link);
237 if (link.link_status) {
238 printf("Port %d Link Up - speed %u Mbps - %s\n\n", port_id, (unsigned)link.link_speed,
239 (link.link_duplex == ETH_LINK_FULL_DUPLEX) ? ("full-duplex") : ("half-duplex"));
241 printf("Port %d Link Down\n\n", port_id);
244 This function is called when a link status interrupt is present for the right port.
245 The port_id indicates which port the interrupt applies to.
246 The type parameter identifies the interrupt event type,
247 which currently can be RTE_ETH_EVENT_INTR_LSC only, but other types can be added in the future.
248 The param parameter is the address of the parameter for the callback.
249 This function should be implemented with care since it will be called in the interrupt host thread,
250 which is different from the main thread of its caller.
252 The application registers the lsi_event_callback and a NULL parameter to the link status interrupt event on each port:
256 rte_eth_dev_callback_register((uint8_t)portid, RTE_ETH_EVENT_INTR_LSC, lsi_event_callback, NULL);
258 This registration can be done only after calling the rte_eth_dev_configure() function and before calling any other function.
259 If lsc is initialized with 0, the callback is never called since no interrupt event would ever be present.
261 RX Queue Initialization
262 ~~~~~~~~~~~~~~~~~~~~~~~
264 The application uses one lcore to poll one or several ports, depending on the -q option,
265 which specifies the number of queues per lcore.
267 For example, if the user specifies -q 4, the application is able to poll four ports with one lcore.
268 If there are 16 ports on the target (and if the portmask argument is -p ffff),
269 the application will need four lcores to poll all the ports.
273 ret = rte_eth_rx_queue_setup((uint8_t) portid, 0, nb_rxd, SOCKET0, &rx_conf, lsi_pktmbuf_pool);
275 rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup: err=%d, port=%u\n", ret, portid);
277 The list of queues that must be polled for a given lcore is stored in a private structure called struct lcore_queue_conf.
281 struct lcore_queue_conf {
283 unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE]; unsigned tx_queue_id;
284 struct mbuf_table tx_mbufs[LSI_MAX_PORTS];
287 struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE];
289 The n_rx_port and rx_port_list[] fields are used in the main packet processing loop
290 (see Section 13.4.7, "Receive, Process and Transmit Packets" later in this chapter).
292 The global configuration for the RX queues is stored in a static structure:
296 static const struct rte_eth_rxconf rx_conf = {
298 .pthresh = RX_PTHRESH,
299 .hthresh = RX_HTHRESH,
300 .wthresh = RX_WTHRESH,
304 TX Queue Initialization
305 ~~~~~~~~~~~~~~~~~~~~~~~
307 Each lcore should be able to transmit on any port.
308 For every port, a single TX queue is initialized.
312 /* init one TX queue logical core on each port */
316 ret = rte_eth_tx_queue_setup(portid, 0, nb_txd, rte_eth_dev_socket_id(portid), &tx_conf);
318 rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup: err=%d,port=%u\n", ret, (unsigned) portid);
320 The global configuration for TX queues is stored in a static structure:
324 static const struct rte_eth_txconf tx_conf = {
326 .pthresh = TX_PTHRESH,
327 .hthresh = TX_HTHRESH,
328 .wthresh = TX_WTHRESH,
330 .tx_free_thresh = RTE_TEST_TX_DESC_DEFAULT + 1, /* disable feature */
333 Receive, Process and Transmit Packets
334 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
336 In the lsi_main_loop() function, the main task is to read ingress packets from the RX queues.
337 This is done using the following code:
342 * Read packet from RX queues
345 for (i = 0; i < qconf->n_rx_port; i++) {
346 portid = qconf->rx_port_list[i];
347 nb_rx = rte_eth_rx_burst((uint8_t) portid, 0, pkts_burst, MAX_PKT_BURST);
348 port_statistics[portid].rx += nb_rx;
350 for (j = 0; j < nb_rx; j++) {
352 rte_prefetch0(rte_pktmbuf_mtod(m, void *));
353 lsi_simple_forward(m, portid);
357 Packets are read in a burst of size MAX_PKT_BURST.
358 The rte_eth_rx_burst() function writes the mbuf pointers in a local table and returns the number of available mbufs in the table.
360 Then, each mbuf in the table is processed by the lsi_simple_forward() function.
361 The processing is very simple: processes the TX port from the RX port and then replaces the source and destination MAC addresses.
365 In the following code, the two lines for calculating the output port require some explanation.
366 If portId is even, the first line does nothing (as portid & 1 will be 0), and the second line adds 1.
367 If portId is odd, the first line subtracts one and the second line does nothing.
368 Therefore, 0 goes to 1, and 1 to 0, 2 goes to 3 and 3 to 2, and so on.
373 lsi_simple_forward(struct rte_mbuf *m, unsigned portid)
375 struct ether_hdr *eth;
377 unsigned dst_port = lsi_dst_ports[portid];
379 eth = rte_pktmbuf_mtod(m, struct ether_hdr *);
381 /* 02:00:00:00:00:xx */
383 tmp = ð->d_addr.addr_bytes[0];
385 *((uint64_t *)tmp) = 0x000000000002 + (dst_port << 40);
388 ether_addr_copy(&lsi_ports_eth_addr[dst_port], ð->s_addr);
390 lsi_send_packet(m, dst_port);
393 Then, the packet is sent using the lsi_send_packet(m, dst_port) function.
394 For this test application, the processing is exactly the same for all packets arriving on the same RX port.
395 Therefore, it would have been possible to call the lsi_send_burst() function directly from the main loop
396 to send all the received packets on the same TX port using
397 the burst-oriented send function, which is more efficient.
399 However, in real-life applications (such as, L3 routing),
400 packet N is not necessarily forwarded on the same port as packet N-1.
401 The application is implemented to illustrate that so the same approach can be reused in a more complex application.
403 The lsi_send_packet() function stores the packet in a per-lcore and per-txport table.
404 If the table is full, the whole packets table is transmitted using the lsi_send_burst() function:
408 /* Send the packet on an output interface */
411 lsi_send_packet(struct rte_mbuf *m, uint8_t port)
413 unsigned lcore_id, len;
414 struct lcore_queue_conf *qconf;
416 lcore_id = rte_lcore_id();
417 qconf = &lcore_queue_conf[lcore_id];
418 len = qconf->tx_mbufs[port].len;
419 qconf->tx_mbufs[port].m_table[len] = m;
422 /* enough pkts to be sent */
424 if (unlikely(len == MAX_PKT_BURST)) {
425 lsi_send_burst(qconf, MAX_PKT_BURST, port);
428 qconf->tx_mbufs[port].len = len;
433 To ensure that no packets remain in the tables, each lcore does a draining of the TX queue in its main loop.
434 This technique introduces some latency when there are not many packets to send.
435 However, it improves performance:
439 cur_tsc = rte_rdtsc();
442 * TX burst queue drain
445 diff_tsc = cur_tsc - prev_tsc;
447 if (unlikely(diff_tsc > drain_tsc)) {
448 /* this could be optimized (use queueid instead of * portid), but it is not called so often */
450 for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
451 if (qconf->tx_mbufs[portid].len == 0)
454 lsi_send_burst(&lcore_queue_conf[lcore_id],
455 qconf->tx_mbufs[portid].len, (uint8_t) portid);
456 qconf->tx_mbufs[portid].len = 0;
459 /* if timer is enabled */
461 if (timer_period > 0) {
462 /* advance the timer */
464 timer_tsc += diff_tsc;
466 /* if timer has reached its timeout */
468 if (unlikely(timer_tsc >= (uint64_t) timer_period)) {
469 /* do this only on master core */
471 if (lcore_id == rte_get_master_lcore()) {
474 /* reset the timer */