1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2020 Marvell International Ltd.
4 L3 Forwarding Graph Sample Application
5 ======================================
7 The L3 Forwarding Graph application is a simple example of packet processing
8 using the DPDK Graph framework. The application performs L3 forwarding using
9 Graph framework and nodes written for graph framework.
14 The application demonstrates the use of the graph framework and graph nodes
15 ``ethdev_rx``, ``ip4_lookup``, ``ip4_rewrite``, ``ethdev_tx`` and ``pkt_drop`` in DPDK to
16 implement packet forwarding.
18 The initialization is very similar to those of the :doc:`l3_forward`.
19 There is also additional initialization of graph for graph object creation
20 and configuration per lcore.
21 Run-time path is main thing that differs from L3 forwarding sample application.
22 Difference is that forwarding logic starting from Rx, followed by LPM lookup,
23 TTL update and finally Tx is implemented inside graph nodes. These nodes are
24 interconnected in graph framework. Application main loop needs to walk over
25 graph using ``rte_graph_walk()`` with graph objects created one per slave lcore.
27 The lookup method is as per implementation of ``ip4_lookup`` graph node.
28 The ID of the output interface for the input packet is the next hop returned by
29 the LPM lookup. The set of LPM rules used by the application is statically
30 configured and provided to ``ip4_lookup`` graph node and ``ip4_rewrite`` graph node
31 using node control API ``rte_node_ip4_route_add()`` and ``rte_node_ip4_rewrite_add()``.
33 In the sample application, only IPv4 forwarding is supported as of now.
35 Compiling the Application
36 -------------------------
38 To compile the sample application see :doc:`compiling`.
40 The application is located in the ``l3fwd-graph`` sub-directory.
42 Running the Application
43 -----------------------
45 The application has a number of command line options similar to l3fwd::
47 ./l3fwd-graph [EAL options] -- -p PORTMASK
49 --config(port,queue,lcore)[,(port,queue,lcore)]
50 [--eth-dest=X,MM:MM:MM:MM:MM:MM]
51 [--enable-jumbo [--max-pkt-len PKTLEN]]
57 * ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure
59 * ``-P:`` Optional, sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
60 Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
62 * ``--config (port,queue,lcore)[,(port,queue,lcore)]:`` Determines which queues from which ports are mapped to which cores.
64 * ``--eth-dest=X,MM:MM:MM:MM:MM:MM:`` Optional, ethernet destination for port X.
66 * ``--enable-jumbo:`` Optional, enables jumbo frames.
68 * ``--max-pkt-len:`` Optional, under the premise of enabling jumbo, maximum packet length in decimal (64-9600).
70 * ``--no-numa:`` Optional, disables numa awareness.
72 * ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports.
74 For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0,
75 while cores 8-15 and 24-31 appear on socket 1.
77 To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2,
78 (which are in the same socket too), use the following command:
80 .. code-block:: console
82 ./build/l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)"
86 * The -l option enables cores 1, 2
88 * The -p option enables ports 0 and 1
90 * The --config option enables one queue on each port and maps each (port,queue) pair to a specific core.
91 The following table shows the mapping in this example:
93 +----------+-----------+-----------+-------------------------------------+
94 | **Port** | **Queue** | **lcore** | **Description** |
96 +----------+-----------+-----------+-------------------------------------+
97 | 0 | 0 | 1 | Map queue 0 from port 0 to lcore 1. |
99 +----------+-----------+-----------+-------------------------------------+
100 | 1 | 0 | 2 | Map queue 0 from port 1 to lcore 2. |
102 +----------+-----------+-----------+-------------------------------------+
104 Refer to the *DPDK Getting Started Guide* for general information on running applications and
105 the Environment Abstraction Layer (EAL) options.
107 .. _l3_fwd_graph_explanation:
112 The following sections provide some explanation of the sample application code.
113 As mentioned in the overview section, the initialization is similar to that of
114 the :doc:`l3_forward`. Run-time path though similar in functionality to that of
115 :doc:`l3_forward`, major part of the implementation is in graph nodes via used
116 via ``librte_node`` library.
117 The following sections describe aspects that are specific to the L3 Forwarding
118 Graph sample application.
120 Graph Node Pre-Init Configuration
121 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
123 After device configuration and device Rx, Tx queue setup is complete,
124 a minimal config of port id, num_rx_queues, num_tx_queues, mempools etc will
125 be passed to *ethdev_** node ctrl API ``rte_node_eth_config()``. This will be
126 lead to the clone of ``ethdev_rx`` and ``ethdev_tx`` nodes as ``ethdev_rx-X-Y`` and
127 ``ethdev_tx-X`` where X, Y represent port id and queue id associated with them.
128 In case of ``ethdev_tx-X`` nodes, tx queue id assigned per instance of the node
131 These cloned nodes along with existing static nodes such as ``ip4_lookup`` and
132 ``ip4_rewrite`` will be used in graph creation to associate node's to lcore
133 specific graph object.
137 RTE_ETH_FOREACH_DEV(portid)
141 ret = rte_eth_dev_configure(portid, nb_rx_queue,
142 n_tx_queue, &local_port_conf);
145 /* Init one TX queue per couple (lcore,port) */
147 for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
149 ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd,
155 /* Setup ethdev node config */
156 ethdev_conf[nb_conf].port_id = portid;
157 ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue;
158 ethdev_conf[nb_conf].num_tx_queues = n_tx_queue;
160 ethdev_conf[nb_conf].mp = pktmbuf_pool[0];
162 ethdev_conf[nb_conf].mp = pktmbuf_pool[portid];
163 ethdev_conf[nb_conf].mp_count = NB_SOCKETS;
169 for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
171 for (queue = 0; queue < qconf->n_rx_queue; ++queue) {
174 ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, socketid,
175 &rxq_conf, pktmbuf_pool[0][socketid]);
177 ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, socketid,
178 &rxq_conf, pktmbuf_pool[portid][socketid]);
183 /* Ethdev node config, skip rx queue mapping */
184 ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs);
189 Now a graph needs to be created with a specific set of nodes for every lcore.
190 A graph object returned after graph creation is a per lcore object and
191 cannot be shared between lcores. Since ``ethdev_tx-X`` node is per port node,
192 it can be associated with all the graphs created as all the lcores should have
193 Tx capability for every port. But ``ethdev_rx-X-Y`` node is created per
194 (port, rx_queue_id), so they should be associated with a graph based on
195 the application argument ``--config`` specifying rx queue mapping to lcore.
199 The Graph creation will fail if the passed set of shell node pattern's
200 are not sufficient to meet their inter-dependency or even one node is not
201 found with a given regex node pattern.
205 static const char *const default_patterns[] = {
210 const char **node_patterns;
215 /* Create a graph object per lcore with common nodes and
216 * lcore specific nodes based on application arguments
218 nb_patterns = RTE_DIM(default_patterns);
219 node_patterns = malloc((MAX_RX_QUEUE_PER_LCORE + nb_patterns) *
220 sizeof(*node_patterns));
221 memcpy(node_patterns, default_patterns,
222 nb_patterns * sizeof(*node_patterns));
224 memset(&graph_conf, 0, sizeof(graph_conf));
226 /* Common set of nodes in every lcore's graph object */
227 graph_conf.node_patterns = node_patterns;
229 for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
232 /* Skip graph creation if no source exists */
233 if (!qconf->n_rx_queue)
236 /* Add rx node patterns of this lcore based on --config */
237 for (i = 0; i < qconf->n_rx_queue; i++) {
238 graph_conf.node_patterns[nb_patterns + i] =
239 qconf->rx_queue_list[i].node_name;
242 graph_conf.nb_node_patterns = nb_patterns + i;
243 graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
245 snprintf(qconf->name, sizeof(qconf->name), "worker_%u", lcore_id);
247 graph_id = rte_graph_create(qconf->name, &graph_conf);
251 qconf->graph = rte_graph_lookup(qconf->name);
256 Forwarding data(Route, Next-Hop) addition
257 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
259 Once graph objects are created, node specific info like routes and rewrite
260 headers will be provided run-time using ``rte_node_ip4_route_add()`` and
261 ``rte_node_ip4_rewrite_add()`` API.
265 Since currently ``ip4_lookup`` and ``ip4_rewrite`` nodes don't support
266 lock-less mechanisms(RCU, etc) to add run-time forwarding data like route and
267 rewrite data, forwarding data is added before packet processing loop is
268 launched on slave lcore.
272 /* Add route to ip4 graph infra */
273 for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
276 dst_port = ipv4_l3fwd_lpm_route_array[i].if_out;
280 ret = rte_node_ip4_route_add(ipv4_l3fwd_lpm_route_array[i].ip,
281 ipv4_l3fwd_lpm_route_array[i].depth, next_hop,
282 RTE_NODE_IP4_LOOKUP_NEXT_REWRITE);
286 memcpy(rewrite_data, val_eth + dst_port, rewrite_len);
288 /* Add next hop for a given destination */
289 ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data,
290 rewrite_len, dst_port);
292 RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n",
293 route_str, next_hop);
296 Packet Forwarding using Graph Walk
297 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
299 Now that all the device configurations are done, graph creations are done and
300 forwarding data is updated with nodes, slave lcores will be launched with graph
301 main loop. Graph main loop is very simple in the sense that it needs to
302 continuously call a non-blocking API ``rte_graph_walk()`` with it's lcore
303 specific graph object that was already created.
307 rte_graph_walk() will walk over all the sources nodes i.e ``ethdev_rx-X-Y``
308 associated with a given graph and Rx the available packets and enqueue them
309 to the following node ``ip4_lookup`` which then will enqueue them to ``ip4_rewrite``
310 node if LPM lookup succeeds. ``ip4_rewrite`` node then will update Ethernet header
311 as per next-hop data and transmit the packet via port 'Z' by enqueuing
312 to ``ethdev_tx-Z`` node instance in its graph object.
316 /* Main processing loop */
318 graph_main_loop(void *conf)
322 lcore_id = rte_lcore_id();
323 qconf = &lcore_conf[lcore_id];
324 graph = qconf->graph;
326 RTE_LOG(INFO, L3FWD_GRAPH,
327 "Entering main loop on lcore %u, graph %s(%p)\n", lcore_id,
330 /* Walk over graph until signal to quit */
331 while (likely(!force_quit))
332 rte_graph_walk(graph);