2 Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Intel Corporation nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
31 Server-Node EFD Sample Application
32 ==================================
34 This sample application demonstrates the use of EFD library as a flow-level
35 load balancer, for more information about the EFD Library please refer to the
36 DPDK programmer's guide.
38 This sample application is a variant of the
39 :ref:`client-server sample application <multi_process_app>`
40 where a specific target node is specified for every and each flow
41 (not in a round-robin fashion as the original load balancing sample application).
46 The architecture of the EFD flow-based load balancer sample application is
47 presented in the following figure.
49 .. _figure_efd_sample_app_overview:
51 .. figure:: img/server_node_efd.*
53 Using EFD as a Flow-Level Load Balancer
55 As shown in :numref:`figure_efd_sample_app_overview`,
56 the sample application consists of a front-end node (server)
57 using the EFD library to create a load-balancing table for flows,
58 for each flow a target backend worker node is specified. The EFD table does not
59 store the flow key (unlike a regular hash table), and hence, it can
60 individually load-balance millions of flows (number of targets * maximum number
61 of flows fit in a flow table per target) while still fitting in CPU cache.
63 It should be noted that although they are referred to as nodes, the frontend
64 server and worker nodes are processes running on the same platform.
69 Upon initializing, the frontend server node (process) creates a flow
70 distributor table (based on the EFD library) which is populated with flow
71 information and its intended target node.
73 The sample application assigns a specific target node_id (process) for each of
74 the IP destination addresses as follows:
78 node_id = i % num_nodes; /* Target node id is generated */
79 ip_dst = rte_cpu_to_be_32(i); /* Specific ip destination address is
80 assigned to this target node */
82 then the pair of <key,target> is inserted into the flow distribution table.
84 The main loop of the server process receives a burst of packets, then for
85 each packet, a flow key (IP destination address) is extracted. The flow
86 distributor table is looked up and the target node id is returned. Packets are
87 then enqueued to the specified target node id.
89 It should be noted that flow distributor table is not a membership test table.
90 I.e. if the key has already been inserted the target node id will be correct,
91 but for new keys the flow distributor table will return a value (which can be
97 Upon initializing, the worker node (process) creates a flow table (a regular
98 hash table that stores the key default size 1M flows) which is populated with
99 only the flow information that is serviced at this node. This flow key is
100 essential to point out new keys that have not been inserted before.
102 The worker node's main loop is simply receiving packets then doing a hash table
103 lookup. If a match occurs then statistics are updated for flows serviced by
104 this node. If no match is found in the local hash table then this indicates
105 that this is a new flow, which is dropped.
108 Compiling the Application
109 -------------------------
111 The sequence of steps used to build the application is:
113 #. Export the required environment variables:
115 .. code-block:: console
117 export RTE_SDK=/path/to/rte_sdk
118 export RTE_TARGET=x86_64-native-linuxapp-gcc
120 #. Build the application executable file:
122 .. code-block:: console
124 cd ${RTE_SDK}/examples/server_node_efd/
127 For more details on how to build the DPDK libraries and sample
129 please refer to the *DPDK Getting Started Guide.*
132 Running the Application
133 -----------------------
135 The application has two binaries to be run: the front-end server
136 and the back-end node.
138 The frontend server (server) has the following command line options::
140 ./server [EAL options] -- -p PORTMASK -n NUM_NODES -f NUM_FLOWS
144 * ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure
145 * ``-n NUM_NODES:`` Number of back-end nodes that will be used
146 * ``-f NUM_FLOWS:`` Number of flows to be added in the EFD table (1 million, by default)
148 The back-end node (node) has the following command line options::
150 ./node [EAL options] -- -n NODE_ID
154 * ``-n NODE_ID:`` Node ID, which cannot be equal or higher than NUM_MODES
157 First, the server app must be launched, with the number of nodes that will be run.
158 Once it has been started, the node instances can be run, with different NODE_ID.
159 These instances have to be run as secondary processes, with ``--proc-type=secondary``
160 in the EAL options, which will attach to the primary process memory, and therefore,
161 they can access the queues created by the primary process to distribute packets.
163 To successfully run the application, the command line used to start the
164 application has to be in sync with the traffic flows configured on the traffic
167 For examples of application command lines and traffic generator flows, please
168 refer to the DPDK Test Report. For more details on how to set up and run the
169 sample applications provided with DPDK package, please refer to the
170 :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` and
171 :ref:`DPDK Getting Started Guide for FreeBSD <freebsd_gsg>`.
177 As described in previous sections, there are two processes in this example.
179 The first process, the front-end server, creates and populates the EFD table,
180 which is used to distribute packets to nodes, which the number of flows
181 specified in the command line (1 million, by default).
187 create_efd_table(void)
189 uint8_t socket_id = rte_socket_id();
192 efd_table = rte_efd_create("flow table", num_flows * 2, sizeof(uint32_t),
193 1 << socket_id, socket_id);
195 if (efd_table == NULL)
196 rte_exit(EXIT_FAILURE, "Problem creating the flow table\n");
200 populate_efd_table(void)
205 uint8_t socket_id = rte_socket_id();
208 /* Add flows in table */
209 for (i = 0; i < num_flows; i++) {
210 node_id = i % num_nodes;
212 ip_dst = rte_cpu_to_be_32(i);
213 ret = rte_efd_update(efd_table, socket_id,
214 (void *)&ip_dst, (efd_value_t)node_id);
216 rte_exit(EXIT_FAILURE, "Unable to add entry %u in "
220 printf("EFD table: Adding 0x%x keys\n", num_flows);
223 After initialization, packets are received from the enabled ports, and the IPv4
224 address from the packets is used as a key to look up in the EFD table,
225 which tells the node where the packet has to be distributed.
230 process_packets(uint32_t port_num __rte_unused, struct rte_mbuf *pkts[],
231 uint16_t rx_count, unsigned int socket_id)
235 efd_value_t data[EFD_BURST_MAX];
236 const void *key_ptrs[EFD_BURST_MAX];
238 struct ipv4_hdr *ipv4_hdr;
239 uint32_t ipv4_dst_ip[EFD_BURST_MAX];
241 for (i = 0; i < rx_count; i++) {
242 /* Handle IPv4 header.*/
243 ipv4_hdr = rte_pktmbuf_mtod_offset(pkts[i], struct ipv4_hdr *,
244 sizeof(struct ether_hdr));
245 ipv4_dst_ip[i] = ipv4_hdr->dst_addr;
246 key_ptrs[i] = (void *)&ipv4_dst_ip[i];
249 rte_efd_lookup_bulk(efd_table, socket_id, rx_count,
250 (const void **) key_ptrs, data);
251 for (i = 0; i < rx_count; i++) {
252 node = (uint8_t) ((uintptr_t)data[i]);
254 if (node >= num_nodes) {
256 * Node is out of range, which means that
257 * flow has not been inserted
259 flow_dist_stats.drop++;
260 rte_pktmbuf_free(pkts[i]);
262 flow_dist_stats.distributed++;
263 enqueue_rx_packet(node, pkts[i]);
267 for (i = 0; i < num_nodes; i++)
271 The burst of packets received is enqueued in temporary buffers (per node),
272 and enqueued in the shared ring between the server and the node.
273 After this, a new burst of packets is received and this process is
279 flush_rx_queue(uint16_t node)
284 if (cl_rx_buf[node].count == 0)
288 if (rte_ring_enqueue_bulk(cl->rx_q, (void **)cl_rx_buf[node].buffer,
289 cl_rx_buf[node].count, NULL) != cl_rx_buf[node].count){
290 for (j = 0; j < cl_rx_buf[node].count; j++)
291 rte_pktmbuf_free(cl_rx_buf[node].buffer[j]);
292 cl->stats.rx_drop += cl_rx_buf[node].count;
294 cl->stats.rx += cl_rx_buf[node].count;
296 cl_rx_buf[node].count = 0;
299 The second process, the back-end node, receives the packets from the shared
300 ring with the server and send them out, if they belong to the node.
302 At initialization, it attaches to the server process memory, to have
303 access to the shared ring, parameters and statistics.
307 rx_ring = rte_ring_lookup(get_rx_queue_name(node_id));
309 rte_exit(EXIT_FAILURE, "Cannot get RX ring - "
310 "is server process running?\n");
312 mp = rte_mempool_lookup(PKTMBUF_POOL_NAME);
314 rte_exit(EXIT_FAILURE, "Cannot get mempool for mbufs\n");
316 mz = rte_memzone_lookup(MZ_SHARED_INFO);
318 rte_exit(EXIT_FAILURE, "Cannot get port info structure\n");
320 tx_stats = &(info->tx_stats[node_id]);
321 filter_stats = &(info->filter_stats[node_id]);
323 Then, the hash table that contains the flows that will be handled
324 by the node is created and populated.
328 static struct rte_hash *
329 create_hash_table(const struct shared_info *info)
331 uint32_t num_flows_node = info->num_flows / info->num_nodes;
332 char name[RTE_HASH_NAMESIZE];
336 struct rte_hash_parameters hash_params = {
337 .entries = num_flows_node * 2, /* table load = 50% */
338 .key_len = sizeof(uint32_t), /* Store IPv4 dest IP address */
339 .socket_id = rte_socket_id(),
340 .hash_func_init_val = 0,
343 snprintf(name, sizeof(name), "hash_table_%d", node_id);
344 hash_params.name = name;
345 h = rte_hash_create(&hash_params);
348 rte_exit(EXIT_FAILURE,
349 "Problem creating the hash table for node %d\n",
355 populate_hash_table(const struct rte_hash *h, const struct shared_info *info)
360 uint32_t num_flows_node = 0;
361 uint64_t target_node;
363 /* Add flows in table */
364 for (i = 0; i < info->num_flows; i++) {
365 target_node = i % info->num_nodes;
366 if (target_node != node_id)
369 ip_dst = rte_cpu_to_be_32(i);
371 ret = rte_hash_add_key(h, (void *) &ip_dst);
373 rte_exit(EXIT_FAILURE, "Unable to add entry %u "
374 "in hash table\n", i);
380 printf("Hash table: Adding 0x%x keys\n", num_flows_node);
383 After initialization, packets are dequeued from the shared ring
384 (from the server) and, like in the server process,
385 the IPv4 address from the packets is used as a key to look up in the hash table.
386 If there is a hit, packet is stored in a buffer, to be eventually transmitted
387 in one of the enabled ports. If key is not there, packet is dropped, since the
388 flow is not handled by the node.
393 handle_packets(struct rte_hash *h, struct rte_mbuf **bufs, uint16_t num_packets)
395 struct ipv4_hdr *ipv4_hdr;
396 uint32_t ipv4_dst_ip[PKT_READ_SIZE];
397 const void *key_ptrs[PKT_READ_SIZE];
399 int32_t positions[PKT_READ_SIZE] = {0};
401 for (i = 0; i < num_packets; i++) {
402 /* Handle IPv4 header.*/
403 ipv4_hdr = rte_pktmbuf_mtod_offset(bufs[i], struct ipv4_hdr *,
404 sizeof(struct ether_hdr));
405 ipv4_dst_ip[i] = ipv4_hdr->dst_addr;
406 key_ptrs[i] = &ipv4_dst_ip[i];
408 /* Check if packets belongs to any flows handled by this node */
409 rte_hash_lookup_bulk(h, key_ptrs, num_packets, positions);
411 for (i = 0; i < num_packets; i++) {
412 if (likely(positions[i] >= 0)) {
413 filter_stats->passed++;
414 transmit_packet(bufs[i]);
416 filter_stats->drop++;
417 /* Drop packet, as flow is not handled by this node */
418 rte_pktmbuf_free(bufs[i]);
423 Finally, note that both processes updates statistics, such as transmitted, received
424 and dropped packets, which are shown and refreshed by the server app.
429 do_stats_display(void)
432 const char clr[] = {27, '[', '2', 'J', '\0'};
433 const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
434 uint64_t port_tx[RTE_MAX_ETHPORTS], port_tx_drop[RTE_MAX_ETHPORTS];
435 uint64_t node_tx[MAX_NODES], node_tx_drop[MAX_NODES];
437 /* to get TX stats, we need to do some summing calculations */
438 memset(port_tx, 0, sizeof(port_tx));
439 memset(port_tx_drop, 0, sizeof(port_tx_drop));
440 memset(node_tx, 0, sizeof(node_tx));
441 memset(node_tx_drop, 0, sizeof(node_tx_drop));
443 for (i = 0; i < num_nodes; i++) {
444 const struct tx_stats *tx = &info->tx_stats[i];
446 for (j = 0; j < info->num_ports; j++) {
447 const uint64_t tx_val = tx->tx[info->id[j]];
448 const uint64_t drop_val = tx->tx_drop[info->id[j]];
450 port_tx[j] += tx_val;
451 port_tx_drop[j] += drop_val;
452 node_tx[i] += tx_val;
453 node_tx_drop[i] += drop_val;
457 /* Clear screen and move to top left */
458 printf("%s%s", clr, topLeft);
462 for (i = 0; i < info->num_ports; i++)
463 printf("Port %u: '%s'\t", (unsigned int)info->id[i],
464 get_printable_mac_addr(info->id[i]));
466 for (i = 0; i < info->num_ports; i++) {
467 printf("Port %u - rx: %9"PRIu64"\t"
469 (unsigned int)info->id[i], info->rx_stats.rx[i],
473 printf("\nSERVER\n");
475 printf("distributed: %9"PRIu64", drop: %9"PRIu64"\n",
476 flow_dist_stats.distributed, flow_dist_stats.drop);
480 for (i = 0; i < num_nodes; i++) {
481 const unsigned long long rx = nodes[i].stats.rx;
482 const unsigned long long rx_drop = nodes[i].stats.rx_drop;
483 const struct filter_stats *filter = &info->filter_stats[i];
485 printf("Node %2u - rx: %9llu, rx_drop: %9llu\n"
486 " tx: %9"PRIu64", tx_drop: %9"PRIu64"\n"
487 " filter_passed: %9"PRIu64", "
488 "filter_drop: %9"PRIu64"\n",
489 i, rx, rx_drop, node_tx[i], node_tx_drop[i],
490 filter->passed, filter->drop);