1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright(c) 2010-2015 Intel Corporation.
9 The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
11 The benefits of using the DPDK KNI are:
13 * Faster than existing Linux TUN/TAP interfaces
14 (by eliminating system calls and copy_to_user()/copy_from_user() operations.
16 * Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
18 * Allows an interface with the kernel network stack.
20 The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
22 .. _figure_kernel_nic_intf:
24 .. figure:: img/kernel_nic_intf.*
26 Components of a DPDK KNI Application
29 The DPDK KNI Kernel Module
30 --------------------------
32 The KNI kernel loadable module ``rte_kni`` provides the kernel interface
33 for DPDK applications.
35 When the ``rte_kni`` module is loaded, it will create a device ``/dev/kni``
36 that is used by the DPDK KNI API functions to control and communicate with
39 The ``rte_kni`` kernel module contains several optional parameters which
40 can be specified when the module is loaded to control its behavior:
42 .. code-block:: console
46 parm: lo_mode: KNI loopback mode (default=lo_mode_none):
47 lo_mode_none Kernel loopback disabled
48 lo_mode_fifo Enable kernel loopback with fifo
49 lo_mode_fifo_skb Enable kernel loopback with fifo and skb buffer
51 parm: kthread_mode: Kernel thread mode (default=single):
52 single Single kernel thread mode enabled.
53 multiple Multiple kernel thread mode enabled.
55 parm: carrier: Default carrier state for KNI interface (default=off):
56 off Interfaces will be created with carrier state set to off.
57 on Interfaces will be created with carrier state set to on.
59 parm: enable_bifurcated: Enable request processing support for
60 bifurcated drivers, which means releasing rtnl_lock before calling
61 userspace callback and supporting async requests (default=off):
62 on Enable request processing support for bifurcated drivers.
66 Loading the ``rte_kni`` kernel module without any optional parameters is
67 the typical way a DPDK application gets packets into and out of the kernel
68 network stack. Without any parameters, only one kernel thread is created
69 for all KNI devices for packet receiving in kernel side, loopback mode is
70 disabled, and the default carrier state of KNI interfaces is set to *off*.
72 .. code-block:: console
74 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko
76 .. _kni_loopback_mode:
81 For testing, the ``rte_kni`` kernel module can be loaded in loopback mode
82 by specifying the ``lo_mode`` parameter:
84 .. code-block:: console
86 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko lo_mode=lo_mode_fifo
88 The ``lo_mode_fifo`` loopback option will loop back ring enqueue/dequeue
89 operations in kernel space.
91 .. code-block:: console
93 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko lo_mode=lo_mode_fifo_skb
95 The ``lo_mode_fifo_skb`` loopback option will loop back ring enqueue/dequeue
96 operations and sk buffer copies in kernel space.
98 If the ``lo_mode`` parameter is not specified, loopback mode is disabled.
100 .. _kni_kernel_thread_mode:
105 To provide flexibility of performance, the ``rte_kni`` KNI kernel module
106 can be loaded with the ``kthread_mode`` parameter. The ``rte_kni`` kernel
107 module supports two options: "single kernel thread" mode and "multiple
110 Single kernel thread mode is enabled as follows:
112 .. code-block:: console
114 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko kthread_mode=single
116 This mode will create only one kernel thread for all KNI interfaces to
117 receive data on the kernel side. By default, this kernel thread is not
118 bound to any particular core, but the user can set the core affinity for
119 this kernel thread by setting the ``core_id`` and ``force_bind`` parameters
120 in ``struct rte_kni_conf`` when the first KNI interface is created:
122 For optimum performance, the kernel thread should be bound to a core in
123 on the same socket as the DPDK lcores used in the application.
125 The KNI kernel module can also be configured to start a separate kernel
126 thread for each KNI interface created by the DPDK application. Multiple
127 kernel thread mode is enabled as follows:
129 .. code-block:: console
131 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko kthread_mode=multiple
133 This mode will create a separate kernel thread for each KNI interface to
134 receive data on the kernel side. The core affinity of each ``kni_thread``
135 kernel thread can be specified by setting the ``core_id`` and ``force_bind``
136 parameters in ``struct rte_kni_conf`` when each KNI interface is created.
138 Multiple kernel thread mode can provide scalable higher performance if
139 sufficient unused cores are available on the host system.
141 If the ``kthread_mode`` parameter is not specified, the "single kernel
142 thread" mode is used.
144 .. _kni_default_carrier_state:
146 Default Carrier State
147 ~~~~~~~~~~~~~~~~~~~~~
149 The default carrier state of KNI interfaces created by the ``rte_kni``
150 kernel module is controlled via the ``carrier`` option when the module
153 If ``carrier=off`` is specified, the kernel module will leave the carrier
154 state of the interface *down* when the interface is management enabled.
155 The DPDK application can set the carrier state of the KNI interface using the
156 ``rte_kni_update_link()`` function. This is useful for DPDK applications
157 which require that the carrier state of the KNI interface reflect the
158 actual link state of the corresponding physical NIC port.
160 If ``carrier=on`` is specified, the kernel module will automatically set
161 the carrier state of the interface to *up* when the interface is management
162 enabled. This is useful for DPDK applications which use the KNI interface as
163 a purely virtual interface that does not correspond to any physical hardware
164 and do not wish to explicitly set the carrier state of the interface with
165 ``rte_kni_update_link()``. It is also useful for testing in loopback mode
166 where the NIC port may not be physically connected to anything.
168 To set the default carrier state to *on*:
170 .. code-block:: console
172 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko carrier=on
174 To set the default carrier state to *off*:
176 .. code-block:: console
178 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko carrier=off
180 If the ``carrier`` parameter is not specified, the default carrier state
181 of KNI interfaces will be set to *off*.
183 .. _kni_bifurcated_device_support:
185 Bifurcated Device Support
186 ~~~~~~~~~~~~~~~~~~~~~~~~~
188 User callbacks are executed while kernel module holds the ``rtnl`` lock, this
189 causes a deadlock when callbacks run control commands on another Linux kernel
192 Bifurcated devices has kernel network driver part and to prevent deadlock for
193 them ``enable_bifurcated`` is used.
195 To enable bifurcated device support:
197 .. code-block:: console
199 # insmod <build_dir>/kernel/linux/kni/rte_kni.ko enable_bifurcated=on
201 Enabling bifurcated device support releases ``rtnl`` lock before calling
202 callback and locks it back after callback. Also enables asynchronous request to
203 support callbacks that requires rtnl lock to work (interface down).
205 KNI Creation and Deletion
206 -------------------------
208 Before any KNI interfaces can be created, the ``rte_kni`` kernel module must
209 be loaded into the kernel and configured with the ``rte_kni_init()`` function.
211 The KNI interfaces are created by a DPDK application dynamically via the
212 ``rte_kni_alloc()`` function.
214 The ``struct rte_kni_conf`` structure contains fields which allow the
215 user to specify the interface name, set the MTU size, set an explicit or
216 random MAC address and control the affinity of the kernel Rx thread(s)
217 (both single and multi-threaded modes).
218 By default the KNI sample example gets the MTU from the matching device,
219 and in case of KNI PMD it is derived from mbuf buffer length.
221 The ``struct rte_kni_ops`` structure contains pointers to functions to
222 handle requests from the ``rte_kni`` kernel module. These functions
223 allow DPDK applications to perform actions when the KNI interfaces are
224 manipulated by control commands or functions external to the application.
226 For example, the DPDK application may wish to enabled/disable a physical
227 NIC port when a user enabled/disables a KNI interface with ``ip link set
228 [up|down] dev <ifaceX>``. The DPDK application can register a callback for
229 ``config_network_if`` which will be called when the interface management
232 There are currently four callbacks for which the user can register
233 application functions:
235 ``config_network_if``:
237 Called when the management state of the KNI interface changes.
238 For example, when the user runs ``ip link set [up|down] dev <ifaceX>``.
242 Called when the user changes the MTU size of the KNI
243 interface. For example, when the user runs ``ip link set mtu <size>
246 ``config_mac_address``:
248 Called when the user changes the MAC address of the KNI interface.
249 For example, when the user runs ``ip link set address <MAC>
250 dev <ifaceX>``. If the user sets this callback function to NULL,
251 but sets the ``port_id`` field to a value other than -1, a default
252 callback handler in the rte_kni library ``kni_config_mac_address()``
253 will be called which calls ``rte_eth_dev_default_mac_addr_set()``
254 on the specified ``port_id``.
256 ``config_promiscusity``:
258 Called when the user changes the promiscuity state of the KNI
259 interface. For example, when the user runs ``ip link set promisc
260 [on|off] dev <ifaceX>``. If the user sets this callback function to
261 NULL, but sets the ``port_id`` field to a value other than -1, a default
262 callback handler in the rte_kni library ``kni_config_promiscusity()``
263 will be called which calls ``rte_eth_promiscuous_<enable|disable>()``
264 on the specified ``port_id``.
266 ``config_allmulticast``:
268 Called when the user changes the allmulticast state of the KNI interface.
269 For example, when the user runs ``ifconfig <ifaceX> [-]allmulti``. If the
270 user sets this callback function to NULL, but sets the ``port_id`` field to
271 a value other than -1, a default callback handler in the rte_kni library
272 ``kni_config_allmulticast()`` will be called which calls
273 ``rte_eth_allmulticast_<enable|disable>()`` on the specified ``port_id``.
275 In order to run these callbacks, the application must periodically call
276 the ``rte_kni_handle_request()`` function. Any user callback function
277 registered will be called directly from ``rte_kni_handle_request()`` so
278 care must be taken to prevent deadlock and to not block any DPDK fastpath
279 tasks. Typically DPDK applications which use these callbacks will need
280 to create a separate thread or secondary process to periodically call
281 ``rte_kni_handle_request()``.
283 The KNI interfaces can be deleted by a DPDK application with
284 ``rte_kni_release()``. All KNI interfaces not explicitly deleted will be
285 deleted when the ``/dev/kni`` device is closed, either explicitly with
286 ``rte_kni_close()`` or when the DPDK application is closed.
291 To minimize the amount of DPDK code running in kernel space, the mbuf mempool is managed in userspace only.
292 The kernel module will be aware of mbufs,
293 but all mbuf allocation and free operations will be handled by the DPDK application only.
295 :numref:`figure_pkt_flow_kni` shows a typical scenario with packets sent in both directions.
297 .. _figure_pkt_flow_kni:
299 .. figure:: img/pkt_flow_kni.*
301 Packet Flow via mbufs in the DPDK KNI
307 On the DPDK RX side, the mbuf is allocated by the PMD in the RX thread context.
308 This thread will enqueue the mbuf in the rx_q FIFO,
309 and the next pointers in mbuf-chain will convert to physical address.
310 The KNI thread will poll all KNI active devices for the rx_q.
311 If an mbuf is dequeued, it will be converted to a sk_buff and sent to the net stack via netif_rx().
312 The dequeued mbuf must be freed, so the same pointer is sent back in the free_q FIFO,
313 and next pointers must convert back to virtual address if exists before put in the free_q FIFO.
315 The RX thread, in the same main loop, polls this FIFO and frees the mbuf after dequeuing it.
316 The address conversion of the next pointer is to prevent the chained mbuf
317 in different hugepage segments from causing kernel crash.
322 For packet egress the DPDK application must first enqueue several mbufs to create an mbuf cache on the kernel side.
324 The packet is received from the Linux net stack, by calling the kni_net_tx() callback.
325 The mbuf is dequeued (without waiting due the cache) and filled with data from sk_buff.
326 The sk_buff is then freed and the mbuf sent in the tx_q FIFO.
328 The DPDK TX thread dequeues the mbuf and sends it to the PMD via ``rte_eth_tx_burst()``.
329 It then puts the mbuf back in the cache.
334 KNI operates in IOVA_VA scheme when
336 - LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0) and
337 - EAL option `iova-mode=va` is passed or bus IOVA scheme in the DPDK is selected
340 Due to IOVA to KVA address translations, based on the KNI use case there
341 can be a performance impact. For mitigation, forcing IOVA to PA via EAL
342 "--iova-mode=pa" option can be used, IOVA_DC bus iommu scheme can also
343 result in IOVA as PA.
348 Ethtool is a Linux-specific tool with corresponding support in the kernel.
349 The current version of kni provides minimal ethtool functionality
350 including querying version and link state. It does not support link
351 control, statistics, or dumping device registers.