1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(C) 2020 Marvell International Ltd.
10 *Tracing* is a technique used to understand what goes on in a running software
11 system. The software used for tracing is called a *tracer*, which is
12 conceptually similar to a tape recorder.
13 When recording, specific instrumentation points placed in the software source
14 code generate events that are saved on a giant tape: a trace file.
15 The trace file then later can be opened in *trace viewers* to visualize and
16 analyze the trace events with timestamps and multi-core views.
17 Such a mechanism will be useful for resolving a wide range of problems such as
18 multi-core synchronization issues, latency measurements, finding out the
19 post analysis information like CPU idle time, etc that would otherwise be
20 extremely challenging to get.
22 Tracing is often compared to *logging*. However, tracers and loggers are two
23 different tools, serving two different purposes.
24 Tracers are designed to record much lower-level events that occur much more
25 frequently than log messages, often in the range of thousands per second, with
26 very little execution overhead.
27 Logging is more appropriate for a very high-level analysis of less frequent
28 events: user accesses, exceptional conditions (errors and warnings, for
29 example), database transactions, instant messaging communications, and such.
30 Simply put, logging is one of the many use cases that can be satisfied with
33 DPDK tracing library features
34 -----------------------------
36 - A framework to add tracepoints in control and fast path APIs with minimum
37 impact on performance.
38 Typical trace overhead is ~20 cycles and instrumentation overhead is 1 cycle.
39 - Enable and disable the tracepoints at runtime.
40 - Save the trace buffer to the filesystem at any point in time.
41 - Support ``overwrite`` and ``discard`` trace mode operations.
42 - String-based tracepoint object lookup.
43 - Enable and disable a set of tracepoints based on regular expression and/or
45 - Generate trace in ``Common Trace Format (CTF)``. ``CTF`` is an open-source
46 trace format and is compatible with ``LTTng``.
47 For detailed information, refer to
48 `Common Trace Format <https://diamon.org/ctf/>`_.
50 How to add a tracepoint?
51 ------------------------
53 This section steps you through the details of adding a simple tracepoint.
55 .. _create_provider_header_file:
57 Create the tracepoint provider header file
58 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
62 #include <rte_trace_point.h>
66 RTE_TRACE_POINT_ARGS(const char *str),
67 rte_trace_point_emit_string(str);
70 The above macro creates ``app_trace_string`` tracepoint.
71 The user can choose any name for the tracepoint.
72 However, when adding a tracepoint in the DPDK library, the
73 ``rte_<library_name>_trace_[<domain>_]<name>`` naming convention must be
75 The examples are ``rte_eal_trace_generic_str``, ``rte_mempool_trace_create``.
77 The ``RTE_TRACE_POINT`` macro expands from above definition as the following
82 static __rte_always_inline void
83 app_trace_string(const char *str)
85 /* Trace subsystem hooks */
87 rte_trace_point_emit_string(str);
90 The consumer of this tracepoint can invoke
91 ``app_trace_string(const char *str)`` to emit the trace event to the trace
94 Register the tracepoint
95 ~~~~~~~~~~~~~~~~~~~~~~~
99 /* Select tracepoint register macros */
100 #define RTE_TRACE_POINT_REGISTER_SELECT
102 #include <my_tracepoint_provider.h>
104 RTE_TRACE_POINT_DEFINE(app_trace_string);
106 RTE_INIT(app_trace_init)
108 RTE_TRACE_POINT_REGISTER(app_trace_string, app.trace.string);
111 The above code snippet registers the ``app_trace_string`` tracepoint to
112 trace library. Here, the ``my_tracepoint_provider.h`` is the header file
113 that the user created in the first step :ref:`create_provider_header_file`.
115 The second argument for the ``RTE_TRACE_POINT_REGISTER`` is the name for the
116 tracepoint. This string will be used for tracepoint lookup or regular
117 expression and/or glob based tracepoint operations.
118 There is no requirement for the tracepoint function and its name to be similar.
119 However, it is recommended to have a similar name for a better naming
122 The user must register the tracepoint before the ``rte_eal_init`` invocation.
123 The user can use the ``RTE_INIT`` construction scheme to achieve this.
127 The ``RTE_TRACE_POINT_REGISTER_SELECT`` must be defined before including the
128 header for the tracepoint registration to work properly.
132 The ``RTE_TRACE_POINT_DEFINE`` defines the placeholder for the
133 ``rte_trace_point_t`` tracepoint object. The user must export a
134 ``__<trace_function_name>`` symbol in the library ``.map`` file for this
135 tracepoint to be used out of the library, in shared builds.
136 For example, ``__app_trace_string`` will be the exported symbol in the
142 In order to avoid performance impact in fast path code, the library introduced
143 ``RTE_TRACE_POINT_FP``. When adding the tracepoint in fast path code,
144 the user must use ``RTE_TRACE_POINT_FP`` instead of ``RTE_TRACE_POINT``.
146 ``RTE_TRACE_POINT_FP`` is compiled out by default and it can be enabled using
147 ``CONFIG_RTE_ENABLE_TRACE_FP`` configuration parameter.
148 The ``enable_trace_fp`` option shall be used for the same for meson build.
153 Event record mode is an attribute of trace buffers. Trace library exposes the
157 When the trace buffer is full, new trace events overwrites the existing
158 captured events in the trace buffer.
160 When the trace buffer is full, new trace events will be discarded.
162 The mode can be configured either using EAL command line parameter
163 ``--trace-mode`` on application boot up or use ``rte_trace_mode_set()`` API to
164 configure at runtime.
169 On ``rte_trace_save()`` or ``rte_eal_cleanup()`` invocation, the library saves
170 the trace buffers to the filesystem. By default, the trace files are stored in
171 ``$HOME/dpdk-traces/rte-yyyy-mm-dd-[AP]M-hh-mm-ss/``.
172 It can be overridden by the ``--trace-dir=<directory path>`` EAL command line
175 For more information, refer to :doc:`../linux_gsg/linux_eal_parameters` for
176 trace EAL command line options.
178 View and analyze the recorded events
179 ------------------------------------
181 Once the trace directory is available, the user can view/inspect the recorded
184 There are many tools you can use to read DPDK traces:
186 1. ``babeltrace`` is a command-line utility that converts trace formats; it
187 supports the format that DPDK trace library produces, CTF, as well as a
188 basic text output that can be grep'ed.
189 The babeltrace command is part of the Open Source Babeltrace project.
191 2. ``Trace Compass`` is a graphical user interface for viewing and analyzing
192 any type of logs or traces, including DPDK traces.
194 Use the babeltrace command-line tool
195 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
197 The simplest way to list all the recorded events of a trace is to pass its path
198 to babeltrace with no options::
200 babeltrace </path-to-trace-events/rte-yyyy-mm-dd-[AP]M-hh-mm-ss/>
202 ``babeltrace`` finds all traces recursively within the given path and prints
203 all their events, merging them in chronological order.
205 You can pipe the output of the babeltrace into a tool like grep(1) for further
206 filtering. Below example grep the events for ``ethdev`` only::
208 babeltrace /tmp/my-dpdk-trace | grep ethdev
210 You can pipe the output of babeltrace into a tool like wc(1) to count the
211 recorded events. Below example count the number of ``ethdev`` events::
213 babeltrace /tmp/my-dpdk-trace | grep ethdev | wc --lines
215 Use the tracecompass GUI tool
216 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
218 ``Tracecompass`` is another tool to view/analyze the DPDK traces which gives
219 a graphical view of events. Like ``babeltrace``, tracecompass also provides
220 an interface to search for a particular event.
221 To use ``tracecompass``, following are the minimum required steps:
223 - Install ``tracecompass`` to the localhost. Variants are available for Linux,
225 - Launch ``tracecompass`` which will open a graphical window with trace
226 management interfaces.
227 - Open a trace using ``File->Open Trace`` option and select metadata file which
228 is to be viewed/analyzed.
230 For more details, refer
231 `Trace Compass <https://www.eclipse.org/tracecompass/>`_.
236 This section steps you through the details of generating trace and viewing it.
238 - Start the dpdk-test::
240 echo "quit" | ./build/app/test/dpdk-test --no-huge --trace=.*
242 - View the traces with babeltrace viewer::
244 babeltrace $HOME/dpdk-traces/rte-yyyy-mm-dd-[AP]M-hh-mm-ss/
246 Implementation details
247 ----------------------
249 As DPDK trace library is designed to generate traces that uses ``Common Trace
250 Format (CTF)``. ``CTF`` specification consists of the following units to create
253 - ``Stream`` Sequence of packets.
254 - ``Packet`` Header and one or more events.
255 - ``Event`` Header and payload.
257 For detailed information, refer to
258 `Common Trace Format <https://diamon.org/ctf/>`_.
260 The implementation details broadly divided into the following areas:
262 Trace metadata creation
263 ~~~~~~~~~~~~~~~~~~~~~~~
265 Based on the ``CTF`` specification, one of a CTF trace's streams is mandatory:
266 the metadata stream. It contains exactly what you would expect: data about the
267 trace itself. The metadata stream contains a textual description of the binary
268 layouts of all the other streams.
270 This description is written using the Trace Stream Description Language (TSDL),
271 a declarative language that exists only in the realm of CTF.
272 The purpose of the metadata stream is to make CTF readers know how to parse a
273 trace's binary streams of events without CTF specifying any fixed layout.
274 The only stream layout known in advance is, in fact, the metadata stream's one.
276 The internal ``trace_metadata_create()`` function generates the metadata.
281 The trace memory will be allocated through an internal function
282 ``__rte_trace_mem_per_thread_alloc()``. The trace memory will be allocated
283 per thread to enable lock less trace-emit function.
284 The memory for the trace memory for DPDK lcores will be allocated on
285 ``rte_eal_init()`` if the trace is enabled through a EAL option.
286 For non DPDK threads, on the first trace emission, the memory will be
292 .. _table_trace_mem_layout:
294 .. table:: Trace memory layout.
296 +-------------------+
298 +-------------------+
300 +-------------------+
302 +-------------------+
304 +-------------------+
306 +-------------------+
308 +-------------------+
310 +-------------------+
312 +-------------------+
317 .. _table_packet_header:
319 .. table:: Packet header layout.
321 +-------------------+
323 +-------------------+
325 +-------------------+
330 .. _table_packet_context:
332 .. table:: Packet context layout.
334 +----------------------+
335 | uint32_t thread_id |
336 +----------------------+
337 | char thread_name[32] |
338 +----------------------+
343 .. _table_trace_header:
345 .. table:: Trace header layout.
347 +----------------------+
349 +----------------------+
351 +----------------------+
353 The trace header is 64 bits, it consists of 48 bits of timestamp and 16 bits
356 The ``packet.header`` and ``packet.context`` will be written in the slow path
357 at the time of trace memory creation. The ``trace.header`` and trace payload
358 will be emitted when the tracepoint function is invoked.