1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright 2021 The DPDK contributors
4 DPDK Unit Testing Guidelines
5 ============================
7 This document outlines the guidelines for running and adding new
8 tests to the in-tree DPDK test suites.
10 The DPDK test suite model is loosely based on the xUnit model,
11 where tests are grouped into test suites, and suites are run by runners.
12 For a basic overview, see the basic Wikipedia article on `xUnit
13 <https://en.wikipedia.org/wiki/XUnit>`_.
19 The in-tree testing infrastructure for DPDK consists of
20 multiple applications and support tools.
21 The primary tools are the `dpdk-test` application,
22 and the ``meson test`` infrastructure.
23 These two are the primary ways through which
24 a user will interact with the DPDK testing infrastructure.
26 There exists a bit of confusion with the test suite and test case separation
27 with respect to `dpdk-test` and ``meson test``.
28 Both have a concept of test suite and test case.
29 In both, the concept is similar.
30 A test suite is a group of test cases,
31 and a test case represents the steps needed to test a particular set of code.
32 Where needed, they will be disambiguated by the word `Meson`
33 to denote a Meson test suite / case.
39 DPDK tests are run via the main test runner, the `dpdk-test` app.
40 The `dpdk-test` app is a command-line interface that facilitates
41 running various tests or test suites.
43 There are three modes of operation.
44 The first mode is as an interactive command shell
45 that allows launching specific test suites.
46 This is the default operating mode of `dpdk-test` and can be done by::
48 $ ./build/app/test/dpdk-test --dpdk-options-here
49 EAL: Detected 4 lcore(s)
50 EAL: Detected 1 NUMA nodes
51 EAL: Static memory layout is selected, amount of reserved memory...
52 EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
53 EAL: Selected IOVA mode 'VA'
54 EAL: Probing VFIO support...
55 EAL: PCI device 0000:00:1f.6 on NUMA socket -1
56 EAL: Invalid NUMA socket, default to 0
57 EAL: probe driver: 8086:15d7 net_e1000_em
58 APP: HPET is not enabled, using TSC as default timer
61 At the prompt, simply type the name of the test suite you wish to run
64 The second form is useful for a scripting environment,
65 and is used by the DPDK Meson build system.
66 This mode is invoked by
67 assigning a specific test suite name to the environment variable ``DPDK_TEST``
68 before invoking the `dpdk-test` command, such as::
70 $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
71 EAL: Detected 4 lcore(s)
72 EAL: Detected 1 NUMA nodes
73 EAL: Static memory layout is selected, amount of reserved memory can be...
74 EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
75 EAL: Selected IOVA mode 'VA'
76 EAL: Probing VFIO support...
77 EAL: PCI device 0000:00:1f.6 on NUMA socket -1
78 EAL: Invalid NUMA socket, default to 0
79 EAL: probe driver: 8086:15d7 net_e1000_em
80 APP: HPET is not enabled, using TSC as default timer
82 Version string: 'DPDK 20.02.0-rc0'
86 The above shows running a specific test case.
87 On success, the return code will be '0',
88 otherwise it will be set to some error value (such as '255', or a negative value).
90 The third form is an alternative
91 to providing the test suite name in an environment variable.
92 The unit test app can accept test suite names via command line arguments::
94 $ ./build/app/test/dpdk-test --dpdk-options-here version_autotest version_autotest
95 EAL: Detected 8 lcore(s)
96 EAL: Detected 1 NUMA nodes
97 EAL: Static memory layout is selected, amount of reserved memory can be...
98 EAL: Detected static linkage of DPDK
99 EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
100 EAL: Selected IOVA mode 'VA'
101 TELEMETRY: No legacy callbacks, legacy socket not created
102 APP: HPET is not enabled, using TSC as default timer
103 RTE>>version_autotest
104 Version string: 'DPDK 21.08.0-rc0'
106 RTE>>version_autotest
107 Version string: 'DPDK 21.08.0-rc0'
111 The primary benefit here is specifying multiple test names,
112 which is not possible with the ``DPDK_TEST`` environment variable.
114 Additionally, it is possible to specify additional test parameters
115 via the ``DPDK_TEST_PARAMS`` argument,
116 in case some tests need additional configuration.
117 This isn't currently used in the Meson test suites.
120 Running test cases via Meson
121 ----------------------------
123 In order to allow developers to quickly execute all the standard internal tests
124 without needing to remember or look up each test suite name,
125 the build system includes a standard way of executing the Meson test suites.
126 After building via ``ninja``, the ``meson test`` command
127 with no arguments will execute the Meson test suites.
129 There are five pre-configured Meson test suites.
130 The first is the **fast** test suite, which is the largest group of test cases.
131 These are the bulk of the unit tests to validate functional blocks.
132 The second is the **perf** tests.
133 These test suites can take longer to run and do performance evaluations.
134 The third is the **driver** test suite,
135 which is mostly for special hardware related testing (such as `cryptodev`).
136 The fourth suite is the **debug** suite.
137 These tests mostly are used to dump system information.
138 The last suite is the **extra** suite for tests having some known issues.
140 The Meson test suites can be selected by adding the ``--suite`` option
141 to the ``meson test`` command.
142 Ex: ``meson test --suite fast-tests``::
144 $ meson test -C build --suite fast-tests
145 ninja: Entering directory `/home/aconole/git/dpdk/build'
146 [2543/2543] Linking target app/test/dpdk-test.
147 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s
148 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s
149 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s
150 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s
151 5/60 DPDK:fast-tests / common_autotest OK 0.57 s
152 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s
155 The ``meson test`` command can also execute individual Meson test cases
156 via the command line by adding the test names as an argument::
158 $ meson test -C build version_autotest
159 ninja: Entering directory `/home/aconole/git/dpdk/build'
160 [2543/2543] Linking target app/test/dpdk-test.
161 1/1 DPDK:fast-tests / version_autotest OK 0.17s
164 Note that these test cases must be known to Meson
165 for the ``meson test`` command to run them.
166 Simply adding a new test to the `dpdk-test` application isn't enough.
167 See the section `Adding a suite or test case to Meson`_ for more details.
170 Adding tests to dpdk-test application
171 -------------------------------------
173 Unit tests should be added to the system
174 whenever we introduce new functionality to DPDK,
175 as well as whenever a bug is resolved.
176 This helps the DPDK project to catch regressions as they are introduced.
178 The DPDK test application supports two layers of tests:
179 #. *test cases* which are individual tests
180 #. *test suites* which are groups of test cases
182 To add a new test suite to the DPDK test application,
183 create a new test file for that suite
184 (ex: see *app/test/test_version.c* for the ``version_autotest`` test suite).
185 There are two important functions for interacting with the test harness:
187 ``REGISTER_TEST_COMMAND(command_name, function_to_execute)``
188 Registers a test command with the name `command_name`
189 and which runs the function `function_to_execute`
190 when `command_name` is invoked.
192 ``unit_test_suite_runner(struct unit_test_suite *)``
193 Returns a runner for a full test suite object,
194 which contains a test suite name, setup, tear down,
195 and vector of unit test cases.
197 Each test suite has a setup and tear down function
198 that runs at the beginning and end of the test suite execution.
199 Each unit test has a similar function for test case setup and tear down.
201 Test cases are added to the ``.unit_test_cases`` element
202 of the appropriate unit test suite structure.
203 An example of both a test suite and a case:
210 #include <rte_common.h>
211 #include <rte_cycles.h>
212 #include <rte_hexdump.h>
213 #include <rte_random.h>
217 static int testsuite_setup(void) { return TEST_SUCCESS; }
218 static void testsuite_teardown(void) { }
220 static int ut_setup(void) { return TEST_SUCCESS; }
221 static void ut_teardown(void) { }
223 static int test_case_first(void) { return TEST_SUCCESS; }
225 static struct unit_test_suite example_testsuite = {
226 .suite_name = "EXAMPLE TEST SUITE",
227 .setup = testsuite_setup,
228 .teardown = testsuite_teardown,
230 TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
232 TEST_CASES_END(), /**< NULL terminate unit test array */
236 static int example_tests()
238 return unit_test_suite_runner(&example_testsuite);
241 REGISTER_TEST_COMMAND(example_autotest, example_tests);
243 The above code block is a small example
244 that can be used to create a complete test suite with test case.
250 Test cases have multiple ways of indicating an error has occurred,
251 in order to reflect failure state back to the runner.
252 Using the various methods of indicating errors can assist
253 in not only validating the requisite functionality is working,
254 but also to help debug when a change in environment or code
255 has caused things to go wrong.
257 The first way to indicate a generic error is
258 by returning a test result failure, using the ``TEST_FAILED`` error code.
259 This is the most basic way of indicating that an error
260 has occurred in a test routine.
261 It isn't very informative to the user, so it should really be used in cases
262 where the test has catastrophically failed.
264 The preferred method of indicating an error is
265 via the ``RTE_TEST_ASSERT`` family of macros,
266 which will immediately return ``TEST_FAILED`` error condition,
267 but will also log details about the failure.
272 RTE_TEST_ASSERT(cond, msg, ...)
274 In the above macro, *cond* is the condition to evaluate to **true**.
275 Any generic condition can go here.
276 The *msg* parameter will be a message to display if *cond* evaluates to **false**.
277 Some specialized macros already exist.
278 See `lib/librte_eal/include/rte_test.h` for a list of defined test assertions.
280 Sometimes it is important to indicate that a test needs to be skipped,
281 either because the environment isn't able to support running the test,
282 or because some requisite functionality isn't available.
283 The test suite supports returning a result of ``TEST_SKIPPED``
284 during test case setup, or during test case execution
285 to indicate that the preconditions of the test aren't available.
288 $ meson test -C build --suite fast-tests
289 ninja: Entering directory `/home/aconole/git/dpdk/build
290 [2543/2543] Linking target app/test/dpdk-test.
291 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s
292 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s
293 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s
295 46/60 DPDK:fast-tests / ipsec_autotest SKIP 0.22 s
299 Checking code coverage
300 ----------------------
302 The Meson build system supports generating a code coverage report
303 via the ``-Db_coverage=true`` option,
304 in conjunction with a package like **lcov**,
305 to generate an HTML code coverage report.
308 $ meson setup build -Db_coverage=true
309 $ meson test -C build --suite fast-tests
310 $ ninja coverage-html -C build
312 The above will generate an HTML report
313 in the `build/meson-logs/coveragereport/` directory
314 that can be explored for detailed code covered information.
315 This can be used to assist in test development.
318 Adding a suite or test case to Meson
319 ------------------------------------
321 Adding to one of the Meson test suites involves
322 editing the appropriate Meson build file `app/test/meson.build`
323 and adding the command to the correct test suite class.
324 Once added, the new test will be run
325 as part of the appropriate class (fast, perf, driver, etc.).
327 A user or developer can confirm that a test is known to Meson
328 by using the ``--list`` option::
330 $ meson test -C build --list
331 DPDK:fast-tests / acl_autotest
332 DPDK:fast-tests / bitops_autotest
335 Some of these test suites are run during continuous integration tests,
336 making regression checking automatic for new patches submitted to the project.
338 In general, when a test is added to the `dpdk-test` application,
339 it probably should be added to a Meson test suite,
340 but the choice is left to maintainers and individual developers.
341 Preference is to add tests to the Meson test suites.