.. BSD LICENSE
- Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+ Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
All rights reserved.
Redistribution and use in source and binary forms, with or without
Avoiding lock contention is a key issue in a multi-core environment.
To address this issue, PMDs are designed to work with per-core private resources as much as possible.
-For example, a PMD maintains a separate transmit queue per-core, per-port.
+For example, a PMD maintains a separate transmit queue per-core, per-port, if the PMD is not ``DEV_TX_OFFLOAD_MT_LOCKFREE`` capable.
In the same way, every receive queue of a port is assigned to and polled by a single logical core (lcore).
To comply with Non-Uniform Memory Access (NUMA), memory management is designed to assign to each logical core
To achieve optimal performance, overall software design choices and pure software optimization techniques must be considered and
balanced against available low-level hardware-based optimization features (CPU cache properties, bus speed, NIC PCI bandwidth, and so on).
-The case of packet transmission is an example of this software/ hardware tradeoff issue when optimizing burst-oriented network packet processing engines.
+The case of packet transmission is an example of this software/hardware tradeoff issue when optimizing burst-oriented network packet processing engines.
In the initial case, the PMD could export only an rte_eth_tx_one function to transmit one packet at a time on a given queue.
On top of that, one can easily build an rte_eth_tx_burst function that loops invoking the rte_eth_tx_one function to transmit several packets at a time.
However, an rte_eth_tx_burst function is effectively implemented by the PMD to minimize the driver-level transmit cost per packet through the following optimizations:
Multiple logical cores should never share receive or transmit queues for interfaces since this would require global locks and hinder performance.
+If the PMD is ``DEV_TX_OFFLOAD_MT_LOCKFREE`` capable, multiple threads can invoke ``rte_eth_tx_burst()``
+concurrently on the same tx queue without SW lock. This PMD feature found in some NICs and useful in the following use cases:
+
+* Remove explicit spinlock in some applications where lcores are not mapped to Tx queues with 1:1 relation.
+
+* In the eventdev use case, avoid dedicating a separate TX core for transmitting and thus
+ enables more scaling as all workers can send the packets.
+
+See `Hardware Offload`_ for ``DEV_TX_OFFLOAD_MT_LOCKFREE`` capability probing details.
+
Device Identification and Configuration
---------------------------------------
Other features such as the L3/L4 5-Tuple packet filtering feature of a port can be configured in the same way.
Ethernet* flow control (pause frame) can be configured on the individual port.
Refer to the testpmd source code for details.
-Also, L4 (UDP/TCP/ SCTP) checksum offload by the NIC can be enabled for an individual packet as long as the packet mbuf is set up correctly.
-Refer to the testpmd source code (specifically the csumonly.c file) for details.
+Also, L4 (UDP/TCP/ SCTP) checksum offload by the NIC can be enabled for an individual packet as long as the packet mbuf is set up correctly. See `Hardware Offload`_ for details.
-That being said, the support of some offload features implies the addition of dedicated status bit(s) and value field(s) into the rte_mbuf
-data structure, along with their appropriate handling by the receive/transmit functions exported by each PMD.
-
-For instance, this is the case for the IEEE1588 packet timestamp mechanism, the VLAN tagging and the IP checksum computation, as described in
-the Section 7.6 "Meta Information".
-
-Configuration of Transmit and Receive Queues
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Configuration of Transmit Queues
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each transmit queue is independently configured with the following information:
When configuring for DCB operation, at port initialization, both the number of transmit queues and the number of receive queues must be set to 128.
+Free Tx mbuf on Demand
+~~~~~~~~~~~~~~~~~~~~~~
+
+Many of the drivers do not release the mbuf back to the mempool, or local cache,
+immediately after the packet has been transmitted.
+Instead, they leave the mbuf in their Tx ring and
+either perform a bulk release when the ``tx_rs_thresh`` has been crossed
+or free the mbuf when a slot in the Tx ring is needed.
+
+An application can request the driver to release used mbufs with the ``rte_eth_tx_done_cleanup()`` API.
+This API requests the driver to release mbufs that are no longer in use,
+independent of whether or not the ``tx_rs_thresh`` has been crossed.
+There are two scenarios when an application may want the mbuf released immediately:
+
+* When a given packet needs to be sent to multiple destination interfaces
+ (either for Layer 2 flooding or Layer 3 multi-cast).
+ One option is to make a copy of the packet or a copy of the header portion that needs to be manipulated.
+ A second option is to transmit the packet and then poll the ``rte_eth_tx_done_cleanup()`` API
+ until the reference count on the packet is decremented.
+ Then the same packet can be transmitted to the next destination interface.
+ The application is still responsible for managing any packet manipulations needed
+ between the different destination interfaces, but a packet copy can be avoided.
+ This API is independent of whether the packet was transmitted or dropped,
+ only that the mbuf is no longer in use by the interface.
+
+* Some applications are designed to make multiple runs, like a packet generator.
+ For performance reasons and consistency between runs,
+ the application may want to reset back to an initial state
+ between each run, where all mbufs are returned to the mempool.
+ In this case, it can call the ``rte_eth_tx_done_cleanup()`` API
+ for each destination interface it has been using
+ to request it to release of all its used mbufs.
+
+To determine if a driver supports this API, check for the *Free Tx mbuf on demand* feature
+in the *Network Interface Controller Drivers* document.
+
+Hardware Offload
+~~~~~~~~~~~~~~~~
+
+Depending on driver capabilities advertised by
+``rte_eth_dev_info_get()``, the PMD may support hardware offloading
+feature like checksumming, TCP segmentation, VLAN insertion or
+lockfree multithreaded TX burst on the same TX queue.
+
+The support of these offload features implies the addition of dedicated
+status bit(s) and value field(s) into the rte_mbuf data structure, along
+with their appropriate handling by the receive/transmit functions
+exported by each PMD. The list of flags and their precise meaning is
+described in the mbuf API documentation and in the in :ref:`Mbuf Library
+<Mbuf_Library>`, section "Meta Information".
+
Poll Mode Driver API
--------------------
The Ethernet device API exported by the Ethernet PMDs is described in the *DPDK API Reference*.
-Vector PMD for IXGBE
---------------------
-
-Vector PMD uses Intel® SIMD instructions to optimize packet I/O.
-It improves load/store bandwidth efficiency of L1 data cache by using a wider SSE/AVX register 1 (1).
-The wider register gives space to hold multiple packet buffers so as to save instruction number when processing bulk of packets.
-
-There is no change to PMD API. The RX/TX handler are the only two entries for vPMD packet I/O.
-They are transparently registered at runtime RX/TX execution if all condition checks pass.
-
-1. To date, only an SSE version of IX GBE vPMD is available.
- To ensure that vPMD is in the binary code, ensure that the option CONFIG_RTE_IXGBE_INC_VECTOR=y is in the configure file.
-
-Some constraints apply as pre-conditions for specific optimizations on bulk packet transfers.
-The following sections explain RX and TX constraints in the vPMD.
-
-RX Constraints
-~~~~~~~~~~~~~~
+Extended Statistics API
+~~~~~~~~~~~~~~~~~~~~~~~
-Prerequisites and Pre-conditions
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The extended statistics API allows a PMD to expose all statistics that are
+available to it, including statistics that are unique to the device.
+Each statistic has three properties ``name``, ``id`` and ``value``:
-The following prerequisites apply:
+* ``name``: A human readable string formatted by the scheme detailed below.
+* ``id``: An integer that represents only that statistic.
+* ``value``: A unsigned 64-bit integer that is the value of the statistic.
-* To enable vPMD to work for RX, bulk allocation for Rx must be allowed.
+Note that extended statistic identifiers are
+driver-specific, and hence might not be the same for different ports.
+The API consists of various ``rte_eth_xstats_*()`` functions, and allows an
+application to be flexible in how it retrieves statistics.
-* The RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC=y configuration MACRO must be set before compiling the code.
+Scheme for Human Readable Names
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Ensure that the following pre-conditions are satisfied:
+A naming scheme exists for the strings exposed to clients of the API. This is
+to allow scraping of the API for statistics of interest. The naming scheme uses
+strings split by a single underscore ``_``. The scheme is as follows:
-* rxq->rx_free_thresh >= RTE_PMD_IXGBE_RX_MAX_BURST
+* direction
+* detail 1
+* detail 2
+* detail n
+* unit
-* rxq->rx_free_thresh < rxq->nb_rx_desc
+Examples of common statistics xstats strings, formatted to comply to the scheme
+proposed above:
-* (rxq->nb_rx_desc % rxq->rx_free_thresh) == 0
+* ``rx_bytes``
+* ``rx_crc_errors``
+* ``tx_multicast_packets``
-* rxq->nb_rx_desc < (IXGBE_MAX_RING_DESC - RTE_PMD_IXGBE_RX_MAX_BURST)
+The scheme, although quite simple, allows flexibility in presenting and reading
+information from the statistic strings. The following example illustrates the
+naming scheme:``rx_packets``. In this example, the string is split into two
+components. The first component ``rx`` indicates that the statistic is
+associated with the receive side of the NIC. The second component ``packets``
+indicates that the unit of measure is packets.
-These conditions are checked in the code.
+A more complicated example: ``tx_size_128_to_255_packets``. In this example,
+``tx`` indicates transmission, ``size`` is the first detail, ``128`` etc are
+more details, and ``packets`` indicates that this is a packet counter.
-Scattered packets are not supported in this mode.
-If an incoming packet is greater than the maximum acceptable length of one "mbuf" data size (by default, the size is 2 KB),
-vPMD for RX would be disabled.
+Some additions in the metadata scheme are as follows:
-By default, IXGBE_MAX_RING_DESC is set to 4096 and RTE_PMD_IXGBE_RX_MAX_BURST is set to 32.
+* If the first part does not match ``rx`` or ``tx``, the statistic does not
+ have an affinity with either receive of transmit.
-Feature not Supported by RX Vector PMD
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* If the first letter of the second part is ``q`` and this ``q`` is followed
+ by a number, this statistic is part of a specific queue.
-Some features are not supported when trying to increase the throughput in vPMD.
-They are:
+An example where queue numbers are used is as follows: ``tx_q7_bytes`` which
+indicates this statistic applies to queue number 7, and represents the number
+of transmitted bytes on that queue.
-* IEEE1588
+API Design
+^^^^^^^^^^
-* FDIR
+The xstats API uses the ``name``, ``id``, and ``value`` to allow performant
+lookup of specific statistics. Performant lookup means two things;
-* Header split
+* No string comparisons with the ``name`` of the statistic in fast-path
+* Allow requesting of only the statistics of interest
-* RX checksum off load
+The API ensures these requirements are met by mapping the ``name`` of the
+statistic to a unique ``id``, which is used as a key for lookup in the fast-path.
+The API allows applications to request an array of ``id`` values, so that the
+PMD only performs the required calculations. Expected usage is that the
+application scans the ``name`` of each statistic, and caches the ``id``
+if it has an interest in that statistic. On the fast-path, the integer can be used
+to retrieve the actual ``value`` of the statistic that the ``id`` represents.
-Other features are supported using optional MACRO configuration. They include:
-
-* HW VLAN strip
-
-* HW extend dual VLAN
-
-* Enabled by RX_OLFLAGS (RTE_IXGBE_RX_OLFLAGS_DISABLE=n)
-
-
-To guarantee the constraint, configuration flags in dev_conf.rxmode will be checked:
-
-* hw_vlan_strip
-
-* hw_vlan_extend
-
-* hw_ip_checksum
-
-* header_split
-
-* dev_conf
-
-fdir_conf->mode will also be checked.
-
-RX Burst Size
+API Functions
^^^^^^^^^^^^^
-As vPMD is focused on high throughput, it assumes that the RX burst size is equal to or greater than 32 per burst.
-It returns zero if using nb_pkt < 32 as the expected packet number in the receive handler.
-
-TX Constraint
-~~~~~~~~~~~~~
-
-Prerequisite
-^^^^^^^^^^^^
-
-The only prerequisite is related to tx_rs_thresh.
-The tx_rs_thresh value must be greater than or equal to RTE_PMD_IXGBE_TX_MAX_BURST,
-but less or equal to RTE_IXGBE_TX_MAX_FREE_BUF_SZ.
-Consequently, by default the tx_rs_thresh value is in the range 32 to 64.
-
-Feature not Supported by RX Vector PMD
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-TX vPMD only works when txq_flags is set to IXGBE_SIMPLE_FLAGS.
-
-This means that it does not support TX multi-segment, VLAN offload and TX csum offload.
-The following MACROs are used for these three features:
-
-* ETH_TXQ_FLAGS_NOMULTSEGS
-
-* ETH_TXQ_FLAGS_NOVLANOFFL
-
-* ETH_TXQ_FLAGS_NOXSUMSCTP
-
-* ETH_TXQ_FLAGS_NOXSUMUDP
-
-* ETH_TXQ_FLAGS_NOXSUMTCP
-
-
-Sample Application Notes
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-testpmd
-^^^^^^^
-
-By default, using CONFIG_RTE_IXGBE_RX_OLFLAGS_DISABLE=n:
-
-.. code-block:: console
-
- ./x86_64-native-linuxapp-gcc/app/testpmd -c 300 -n 4 -- -i --burst=32 --rxfreet=32 --mbcache=250 --txpt=32 --rxht=8 --rxwt=0 --txfreet=32 --txrst=32 --txqflags=0xf01
-
-When CONFIG_RTE_IXGBE_RX_OLFLAGS_DISABLE=y, better performance can be achieved:
-
-.. code-block:: console
-
- ./x86_64-native-linuxapp-gcc/app/testpmd -c 300 -n 4 -- -i --burst=32 --rxfreet=32 --mbcache=250 --txpt=32 --rxht=8 --rxwt=0 --txfreet=32 --txrst=32 --txqflags=0xf01 --disable-hw-vlan
-
-If scatter gather lists are not required, set CONFIG_RTE_MBUF_SCATTER_GATHER=n for better throughput.
-
-l3fwd
-^^^^^
-
-When running l3fwd with vPMD, there is one thing to note.
-In the configuration, ensure that port_conf.rxmode.hw_ip_checksum=0.
-Otherwise, by default, RX vPMD is disabled.
-
-load_balancer
-^^^^^^^^^^^^^
+The API is built out of a small number of functions, which can be used to
+retrieve the number of statistics and the names, IDs and values of those
+statistics.
+
+* ``rte_eth_xstats_get_names_by_id()``: returns the names of the statistics. When given a
+ ``NULL`` parameter the function returns the number of statistics that are available.
+
+* ``rte_eth_xstats_get_id_by_name()``: Searches for the statistic ID that matches
+ ``xstat_name``. If found, the ``id`` integer is set.
-As in the case of l3fwd, set configure port_conf.rxmode.hw_ip_checksum=0 to enable vPMD.
-In addition, for improved performance, use -bsz "(32,32),(64,64),(32,32)" in load_balancer to avoid using the default burst size of 144.
+* ``rte_eth_xstats_get_by_id()``: Fills in an array of ``uint64_t`` values
+ with matching the provided ``ids`` array. If the ``ids`` array is NULL, it
+ returns all statistics that are available.
+
+
+Application Usage
+^^^^^^^^^^^^^^^^^
+
+Imagine an application that wants to view the dropped packet count. If no
+packets are dropped, the application does not read any other metrics for
+performance reasons. If packets are dropped, the application has a particular
+set of statistics that it requests. This "set" of statistics allows the app to
+decide what next steps to perform. The following code-snippets show how the
+xstats API can be used to achieve this goal.
+
+First step is to get all statistics names and list them:
+
+.. code-block:: c
+
+ struct rte_eth_xstat_name *xstats_names;
+ uint64_t *values;
+ int len, i;
+
+ /* Get number of stats */
+ len = rte_eth_xstats_get_names_by_id(port_id, NULL, NULL, 0);
+ if (len < 0) {
+ printf("Cannot get xstats count\n");
+ goto err;
+ }
+
+ xstats_names = malloc(sizeof(struct rte_eth_xstat_name) * len);
+ if (xstats_names == NULL) {
+ printf("Cannot allocate memory for xstat names\n");
+ goto err;
+ }
+
+ /* Retrieve xstats names, passing NULL for IDs to return all statistics */
+ if (len != rte_eth_xstats_get_names_by_id(port_id, xstats_names, NULL, len)) {
+ printf("Cannot get xstat names\n");
+ goto err;
+ }
+
+ values = malloc(sizeof(values) * len);
+ if (values == NULL) {
+ printf("Cannot allocate memory for xstats\n");
+ goto err;
+ }
+
+ /* Getting xstats values */
+ if (len != rte_eth_xstats_get_by_id(port_id, NULL, values, len)) {
+ printf("Cannot get xstat values\n");
+ goto err;
+ }
+
+ /* Print all xstats names and values */
+ for (i = 0; i < len; i++) {
+ printf("%s: %"PRIu64"\n", xstats_names[i].name, values[i]);
+ }
+
+The application has access to the names of all of the statistics that the PMD
+exposes. The application can decide which statistics are of interest, cache the
+ids of those statistics by looking up the name as follows:
+
+.. code-block:: c
+
+ uint64_t id;
+ uint64_t value;
+ const char *xstat_name = "rx_errors";
+
+ if(!rte_eth_xstats_get_id_by_name(port_id, xstat_name, &id)) {
+ rte_eth_xstats_get_by_id(port_id, &id, &value, 1);
+ printf("%s: %"PRIu64"\n", xstat_name, value);
+ }
+ else {
+ printf("Cannot find xstats with a given name\n");
+ goto err;
+ }
+
+The API provides flexibility to the application so that it can look up multiple
+statistics using an array containing multiple ``id`` numbers. This reduces the
+function call overhead of retrieving statistics, and makes lookup of multiple
+statistics simpler for the application.
+
+.. code-block:: c
+
+ #define APP_NUM_STATS 4
+ /* application cached these ids previously; see above */
+ uint64_t ids_array[APP_NUM_STATS] = {3,4,7,21};
+ uint64_t value_array[APP_NUM_STATS];
+
+ /* Getting multiple xstats values from array of IDs */
+ rte_eth_xstats_get_by_id(port_id, ids_array, value_array, APP_NUM_STATS);
+
+ uint32_t i;
+ for(i = 0; i < APP_NUM_STATS; i++) {
+ printf("%d: %"PRIu64"\n", ids_array[i], value_array[i]);
+ }
+
+
+This array lookup API for xstats allows the application create multiple
+"groups" of statistics, and look up the values of those IDs using a single API
+call. As an end result, the application is able to achieve its goal of
+monitoring a single statistic ("rx_errors" in this case), and if that shows
+packets being dropped, it can easily retrieve a "set" of statistics using the
+IDs array parameter to ``rte_eth_xstats_get_by_id`` function.
git.droids-corp.org / dpdk.git / blobdiff