2 Copyright(c) 2016 Intel Corporation. All rights reserved.
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions
9 * Redistributions of source code must retain the above copyright
10 notice, this list of conditions and the following disclaimer.
11 * Redistributions in binary form must reproduce the above copyright
12 notice, this list of conditions and the following disclaimer in
13 the documentation and/or other materials provided with the
15 * Neither the name of Intel Corporation nor the names of its
16 contributors may be used to endorse or promote products derived
17 from this software without specific prior written permission.
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
32 L2 Forwarding Sample Application with Cache Allocation Technology (CAT)
33 =======================================================================
35 Basic Forwarding sample application is a simple *skeleton* example of
36 a forwarding application. It has been extended to make use of CAT via extended
37 command line options and linking against the libpqos library.
39 It is intended as a demonstration of the basic components of a DPDK forwarding
40 application and use of the libpqos library to program CAT.
41 For more detailed implementations see the L2 and L3 forwarding
44 CAT and Code Data Prioritization (CDP) features allow management of the CPU's
45 last level cache. CAT introduces classes of service (COS) that are essentially
46 bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to
47 one cache way in last level cache.
48 A CPU core is always assigned to one of the CAT classes.
49 By programming CPU core assignment and COS bitmasks, applications can be given
50 exclusive, shared, or mixed access to the CPU's last level cache.
51 CDP extends CAT so that there are two bitmasks per COS,
52 one for data and one for code.
53 The number of classes and number of valid bits in a COS bitmask is CPU model
54 specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
55 ``cbm`` or capacity bitmask.
56 By default, after reset, all CPU cores are assigned to COS 0 and all classes
57 are programmed to allow fill into all cache ways.
58 CDP is off by default.
60 For more information about CAT please see:
62 * https://github.com/01org/intel-cmt-cat
64 White paper demonstrating example use case:
66 * `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_
68 Compiling the Application
69 -------------------------
71 Requires ``libpqos`` from Intel's
72 `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
73 hosted on GitHub repository. For installation notes, please see ``README`` file.
77 * https://github.com/01org/intel-cmt-cat
79 To compile the application export the path to PQoS lib
80 and the DPDK source tree and go to the example directory:
82 .. code-block:: console
84 export PQOS_INSTALL_PATH=/path/to/libpqos
85 export RTE_SDK=/path/to/rte_sdk
87 cd ${RTE_SDK}/examples/l2fwd-cat
89 Set the target, for example:
91 .. code-block:: console
93 export RTE_TARGET=x86_64-native-linuxapp-gcc
95 See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
97 Build the application as follows:
99 .. code-block:: console
104 Running the Application
105 -----------------------
107 To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
109 .. code-block:: console
111 ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)"
113 or to enable CAT and CDP on cpus 1,3:
115 .. code-block:: console
117 ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)"
119 If CDP is not supported it will fail with following error message:
121 .. code-block:: console
123 PQOS: CDP requested but not supported.
124 PQOS: Requested CAT configuration is not valid!
125 PQOS: Shutting down PQoS library...
126 EAL: Error - exiting with code: 1
127 Cause: PQOS: L3CA init failed!
129 The option to enable CAT is:
131 * ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
133 where ``cbm`` stands for capacity bitmask and must be expressed in
136 ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
137 masks (``code_cbm`` and ``data_cbm``) is used.
139 ``(`` and ``)`` are necessary if it's a group.
141 ``cpus`` could be a single digit/range or a group and must be expressed in
144 ``(`` and ``)`` are necessary if it's a group.
146 e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
148 * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
150 * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
152 * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
154 * cpu 7 has exclusive access to all of its 4 ways;
156 e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
158 * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
159 data ways are not overlapping.
162 Refer to *DPDK Getting Started Guide* for general information on running
163 applications and the Environment Abstraction Layer (EAL) options.
166 To reset or list CAT configuration and control CDP please use ``pqos`` tool
168 `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
170 To enabled or disable CDP:
172 .. code-block:: console
174 sudo ./pqos -S cdp-on
176 sudo ./pqos -S cdp-off
178 to reset CAT configuration:
180 .. code-block:: console
186 .. code-block:: console
190 For more info about ``pqos`` tool please see its man page or
191 `intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
197 The following sections provide an explanation of the main components of the
200 All DPDK library functions used in the sample code are prefixed with ``rte_``
201 and are explained in detail in the *DPDK API Documentation*.
207 The ``main()`` function performs the initialization and calls the execution
208 threads for each lcore.
210 The first task is to initialize the Environment Abstraction Layer (EAL). The
211 ``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
212 function. The value returned is the number of parsed arguments:
216 int ret = rte_eal_init(argc, argv);
218 rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
220 The next task is to initialize the PQoS library and configure CAT. The
221 ``argc`` and ``argv`` arguments are provided to the ``cat_init()``
222 function. The value returned is the number of parsed arguments:
226 int ret = cat_init(argc, argv);
228 rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
230 ``cat_init()`` is a wrapper function which parses the command, validates
231 the requested parameters and configures CAT accordingly.
233 Parsing of command line arguments is done in ``parse_args(...)``.
234 libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
235 queried for system CPU information and L3CA capabilities via
236 ``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
237 calls. When all capability and topology information is collected, the requested
238 CAT configuration is validated. A check is then performed (on per socket basis)
239 for a sufficient number of un-associated COS. COS are selected and
240 configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
241 relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
243 ``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
244 a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
245 COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.