app/bbdev: add explicit check for counters

Adding explicit check in ut that the stats counters
have the expect values.
This was identified as a gap from code coverage.

Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com>
Acked-by: Aidan Goddard <aidan.goddard@accelercomm.com>
Acked-by: Dave Burley <dave.burley@accelercomm.com>

app/bbdev: add explicit test for latency vs validation

Adding explicit different ut when testing for validation
or latency (early termination enabled or not).

Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com>
Acked-by: Aidan Goddard <aidan.goddard@accelercomm.com>
Acked-by: Dave Burley <dave.burley@accelercomm.com>

doc: recommend latest OpenSSL version

Add recommendation to update to latest OpenSSL version when
using the OpenSSL PMD and to at least version 1.1.1g to avoid
known CVEs.

Signed-off-by: John McNamara <john.mcnamara@intel.com>

common/qat: add missing kmod dependency info

Dependency on kmod needed to manage crypto devices is missing
in qat crypto pmd.

Fixes: 0880c40113ef ("drivers: advertise kmod dependencies in pmdinfo")
Cc: stable@dpdk.org
Signed-off-by: Didier Pallard <didier.pallard@6wind.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

crypto/dpaa2_sec: remove dead code

RTE_LIBRTE_SECURITY_TEST never existed, the variable under this check is
never used.

Fixes: e117c18a1dbe ("crypto/dpaa2_sec: restructure session management")
Cc: stable@dpdk.org
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

fips_validation: fix GCM test

This patch fixes FIPS GCM test of the redundant plaintext string
write and insufficient test case support check for some corner
cases.

Fixes: d09abf2d1007 ("examples/fips_validation: update GCM test")
Cc: stable@dpdk.org
Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>

examples/fips_validation: fix missed version line

This patch fixes the missing version line in the response file.

Fixes: 793650184099 ("examples/fips_validation: fix version compatibility")
Cc: stable@dpdk.org
Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>

crypto/octeontx2: fix multi-process

During crypto device probe few functions should be called only
for the primary process. This patch fixes this issue.

Fixes: 818d138bcce9 ("crypto/octeontx2: add init sequence in probe")
Cc: stable@dpdk.org
Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
Reviewed-by: Anoob Joseph <anoobj@marvell.com>

baseband/acc100: remove useless checks

Coverity reported dead code for a few error
checks which are indeed not reachable.

Coverity issue: 363451, 363454, 363455
Fixes: 5ad5060f8f7a ("baseband/acc100: add LDPC processing functions")
Fixes: f404dfe35cc3 ("baseband/acc100: support 4G processing")

Signed-off-by: Nicolas Chautru <nicolas.chautru@intel.com>
Reviewed-by: Tom Rix <trix@redhat.com>

baseband/turbo_sw: fix memory leak in error path

In q_setup() allocated memory for the queue data, we should free
it when error happens, otherwise it will lead to memory leak.

Fixes: b8cfe2c9aed2 ("bb/turbo_sw: add software turbo driver")
Cc: stable@dpdk.org
Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
Reviewed-by: Nicolas Chautru <nicolas.chautru@intel.com>

doc: remove notice about AES-GCM IV and J0

This patch removes information about deprecation of AES-GCM/GMAC
API for IV without J0.

Fixes: fac52fb26a54 ("cryptodev: add option to support both IV and J0 for GCM")
Cc: stable@dpdk.org
Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>

doc: update VFIO usage in qat crypto guide

This patch marks the old igb-uio driver as unsecure when used
with the QAT PMD and updates all examples to recommend using
VFIO-PCI instead.
It also mentions security issues with the QAT CPM and provides
information about the new VFIO-PCI parameter 'disable_denylist'
available in Linux kernels 5.9 and later.

Signed-off-by: Adam Dybkowski <adamx.dybkowski@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

doc: fix driver names

Since the built driver filenames have changed in DPDK 20.11, we need to
update the driver doc to match.

Most drivers start their section with the driver filename highlighted in
bold, while a number were missing the highlight. When updating the names,
add the markers for bold text to any missing it, so as to have things more
consistent.

Fixes: a20b2c01a7a1 ("build: standardize component names and defines")

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Rosen Xu <rosen.xu@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

ethdev: include mbuf registration in Tx timestamp API

Previously, the Tx timestamp field and flag were registered in testpmd,
as described in mlx5 guide.
For consistency between Rx and Tx timestamps,
managing mbuf registrations inside the driver, as properly documented,
is a simpler expectation.

The only driver to support this feature (mlx5) is updated
as well as the testpmd application.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

mbuf: add Tx timestamp registration helper

The function rte_mbuf_dyn_tx_timestamp_register()
can be used to register the required field and flag.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

mbuf: remove deprecated timestamp field

As announced in the deprecation note, the field timestamp
is removed to give more space to the dynamic fields.
The related offload flag PKT_RX_TIMESTAMP is also removed.

This is how the mbuf layout looks like (pahole-style):

word  type                              name                byte  size
 0    void *                            buf_addr;         /*   0 +  8 */
 1    rte_iova_t                        buf_iova          /*   8 +  8 */
      /* --- RTE_MARKER64               rearm_data;                   */
 2    uint16_t                          data_off;         /*  16 +  2 */
      uint16_t                          refcnt;           /*  18 +  2 */
      uint16_t                          nb_segs;          /*  20 +  2 */
      uint16_t                          port;             /*  22 +  2 */
 3    uint64_t                          ol_flags;         /*  24 +  8 */
      /* --- RTE_MARKER                 rx_descriptor_fields1;        */
 4    uint32_t             union        packet_type;      /*  32 +  4 */
      uint32_t                          pkt_len;          /*  36 +  4 */
 5    uint16_t                          data_len;         /*  40 +  2 */
      uint16_t                          vlan_tci;         /*  42 +  2 */
 5.5  uint64_t             union        hash;             /*  44 +  8 */
 6.5  uint16_t                          vlan_tci_outer;   /*  52 +  2 */
      uint16_t                          buf_len;          /*  54 +  2 */
 7    uint64_t                          dynfield0[1];     /*  56 +  8 */
      /* --- RTE_MARKER                 cacheline1;                   */
 8    struct rte_mempool *              pool;             /*  64 +  8 */
 9    struct rte_mbuf *                 next;             /*  72 +  8 */
10    uint64_t             union        tx_offload;       /*  80 +  8 */
11    struct rte_mbuf_ext_shared_info * shinfo;           /*  88 +  8 */
12    uint16_t                          priv_size;        /*  96 +  2 */
      uint16_t                          timesync;         /*  98 +  2 */
12.5  uint32_t                          dynfield1[7];     /* 100 + 28 */
16    /* --- END                                             128      */

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

ethdev: add doxygen comment for Rx timestamp API

The offload flag DEV_RX_OFFLOAD_TIMESTAMP had no documentation.
After switching to dynamic mbuf flag and field,
it becomes even more important to explicit the feature behaviour.

A doxygen comment for the timesync API was mentioning
the deprecated timestamp field, so it is also updated.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

examples/rxtx_callbacks: switch timestamp to dynamic field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

app/testpmd: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/pcap: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/octeontx2: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced.

The registration of field and flag is done in both
otx2_nix_dev_start() and otx2_nix_timesync_enable().

The dynamic offset and flag are stored in struct otx2_timesync_info
to favor cache locality.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/nfb: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/mlx5: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced.

The dynamic offset and flag are stored in struct mlx5_rxq_data
to favor cache locality.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/mlx5: fix dynamic mbuf offset lookup check

The functions rte_mbuf_dynfield_lookup() and rte_mbuf_dynflag_lookup()
can return an offset starting with 0 or a negative error code.

In reality the first offsets are probably reserved forever,
but for the sake of strict API compliance,
the checks which considered 0 as an error are fixed.

Fixes: efa79e68c8cd ("net/mlx5: support fine grain dynamic flag")
Fixes: 3172c471b86f ("net/mlx5: prepare Tx queue structures to support timestamp")
Fixes: 0febfcce3693 ("net/mlx5: prepare Tx to support scheduling")
Cc: stable@dpdk.org
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/dpaa2: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

net/ark: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related dynamic mbuf flag is set, although was missing previously.

The timestamp is set if configured for at least one device.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

latency: switch Rx timestamp to dynamic mbuf field

The mbuf timestamp is moved to a dynamic field
in order to allow removal of the deprecated static field.
The related mbuf flag is also replaced with the dynamic one.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

mbuf: add Rx timestamp flag and helpers

There is already a dynamic field for timestamp,
used only for Tx scheduling with the dedicated Tx offload flag.
The same field can be used for Rx timestamp filled by drivers.

A new dynamic flag is defined for Rx usage.
A new function wraps the registration of both field and Rx flag.
The type rte_mbuf_timestamp_t is defined for the API users.

After migrating all Rx timestamp usages, it will be possible
to remove the deprecated timestamp field.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

eventdev: remove software Rx timestamp

This a revert of the commit 569758758dcd ("eventdev: add Rx timestamp").
If the Rx timestamp is not configured on the ethdev port,
there is no reason to set one.
Also the accuracy  of the timestamp was bad because set at a late stage.
Anyway there is no trace of the usage of this timestamp.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

regex/octeontx2: fix driver name

Following the recent alignment of all driver names,
this new driver get unaligned:
librte_regex_octeontx2_regex.so

The 'fmt_name' must be "octeontx2_regex", and if not provided,
is taken from the 'name' variable.
But the variable 'name' should not be overwritten,
to keep the automatic value from the directory name.

The library name will be composed of the class directory
and the driver directory name:
librte_regex_octeontx2.so

Reported-by: Christian Ehrhardt <christian.ehrhardt@canonical.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: David Marchand <david.marchand@redhat.com>

event/dlb: add timeout ticks entry point

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add queue and port release

These entry points are NO-OPS. DLB does not support
reconfiguring individual queues or ports. The entire device
must be reconfigured.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add self-tests

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add token pop API

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
     are released. Supported on load-balanced ports
     only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add eventdev stop and close

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add dequeue and its burst variants

Add support for dequeue, dequeue_burst, ...

DLB does not currently support interrupts, but instead uses
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add enqueue and its burst variants

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add eventdev start

Add support for the eventdev start entry point.
DLB delays setting up single link resources until
eventdev start, because it is only then that it can
ascertain which ports have just one linked queue.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add port unlink and unlinks in progress

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread. Port QE and memzone
memory is freed here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add port link

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add port setup

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add queue setup

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add queue and port default conf

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add infos get and configure

Add support for configuring the DLB hardware.
In particular, this patch configures the DLB
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add xstats

Add support for DLB xstats.  Perform initialization and add
standard xstats entry points

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add probe-time hardware init

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add flexible interface

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF modei,
but bifurcated mode will be added in a future patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add eventdev probe

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add inline functions

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add definitions shared with LKM or shared code

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add private data structures and constants

Add headers used internally by the PMD.  They include constants,
macros for device resources, structure definitions for hardware interfaces
and software state, and various forward-declarations.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add dynamic logging

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb: add documentation and build infrastructure

Note that config/rte_config.h contains several configuration
switches, providing for fine control of the PMD's
runtime behaviour.

The meson infrastructure is expanded as additional files are
added to this patchset.

Adds announcement of availability of the new driver
for Intel Dynamic Load Balancer 1.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add timeout ticks entry point

Adds the timeout ticks conversion function.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add queue and port release

DLB does not support reconfiguring individual queues
or ports on the fly. The entire device must be reconfigured.
Previously allocated port QE and memzone memory
is freed in this patch.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add self-tests

Add a variety of self-tests for both ldb and directed
ports/queues, as well as configure, start, stop, link, etc...

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add token pop API

The PMD uses a public interface to allow applications to
control the token pop mode. Supported token pop modes are
as follows, and they impact core scheduling affinity for
ldb ports.

AUTO_POP: Pop the CQ tokens immediately after dequeueing.
DELAYED_POP: Pop CQ tokens after (dequeue_depth - 1) events
             are released. Supported on load-balanced ports
             only.
DEFERRED_POP: Pop the CQ tokens during next dequeue operation.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add eventdev stop and close

Add support for eventdev stop and close entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add dequeue and its burst variants

Add support for dequeue, dequeue_burst, ...

DLB2 does not currently support interrupts, but instead use
umonitor/umwait if supported by the processor. This allows
the software to monitor and wait on writes to a cache-line.

DLB2 supports normal and sparse cq mode. In normal mode the
hardware will pack 4 QEs into each cache line. In sparse cq
mode, the hardware will only populate one QE per cache line.
Software must be aware of the cq mode, and take the appropriate
actions, based on the mode.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add enqueue and its burst variants

Add support for enqueue and its variants.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add eventdev start

Add support for the eventdev start entry point.
We delay initializing some resources until
eventdev start, since the number of linked queues can be
used to determine if we are dealing with a ldb or dir resource.
If this is a device restart, then the previous configuration
will be reapplied.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add port unlink and unlinks in progress

Add supports for the port unlink(s) eventdev entry points.
The unlink operation is an asynchronous operation executed by
a control thread, and the unlinks-in-progress function reads
a counter shared with the control thread.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add port link

Add port link entry point. Directed queues are identified and created
at this stage. Their setup deferred until link-time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add port setup

Configure the load balanced (ldb) or directed (dir) port.
The consumer queue (CQ) and producer port (PP) are also
set up here.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add queue setup

Load balanced (ldb) queues are setup here.
Directed queues are not set up until link time, at which
point we know the directed port ID. Directed queue setup
will only fail if this queue is already setup or there are
no directed queues left to configure.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add queue and port default conf

Add support for getting the queue and port default configuration.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add infos get and configure

Add support for configuring the DLB2 hardware.
In particular, this patch configures the DLB2
hardware's scheduling domain, such that it is provisioned with
the requested number of ports and queues, provided sufficient
resources are available. Individual queues and ports are
configured later in port setup and eventdev start.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add xstats

Add support for DLB2 xstats.  Perform initialization and add
standard xstats entry points.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Mike Ximing Chen <mike.ximing.chen@intel.com>

event/dlb2: add probe-time hardware init

This commit adds probe-time low level hardware
initialization.  It also adds probe-time init for both
primary and secondary DPDK processes.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add flexible interface

This commit introduces the flexible interface. This
interface allows the core code to operate in PF mode (direct
hardware access) or bifurcated mode (hardware configured via
kernel driver). This driver currently only supports PF mode
but bifurcated mode will be added in a future DPDK patch-set.
Note that the flexible interface is not used for data path
operations, and thus there are no performance concerns
related to the use of function pointers.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add eventdev probe

Add the eventdev portion of probe, and parse command line
options, but do not initialize hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add inline functions

Add miscellaneous inline functions that may be called
from multiple files.  These functions include inline
assembly of new x86 instructions, such as movdir64b,
since they are not available as builtin functions in
the minimum supported GCC version.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add definitions shared with LKM or shared code

Add headers containing structs and constants shared between
the PMD and the shared code.  The term shared code refers to
the code that implements the hardware interface. The shared code
is introduced in the probe patch, and then is extended as
additional eventdev PMD entry points are added to the patchset.
In the case of the bifurcated PMD (to be introduced in the
future), the shared code is contained in the Linux kernel
module itself.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add private data structures and constants

The header file dlb2_priv.h is used internally by the PMD.
It include constants, macros for device resources,
structure definitions for hardware interfaces and
software state, and various forward-declarations.
The header file rte_pmd_dlb2.h will be exported in a
subsequent patch, but is included here due to a data
structure dependency.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add dynamic logging

This commit adds base support for dynamic logging.
The default log level is NOTICE. Dynamic logging
is used exclusively throughout this patchset.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

event/dlb2: add documentation and build infrastructure

Adds the meson build infrastructure, which includes
compile-time constants in rte_config.h. DLB2 is
only supported on Linux 64 bit X86 platforms at this time.

Adds announcement of availability for the new driver
for Intel Dynamic Load Balancer 2.0 hardware.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>

eventdev: increase max queues configuration

DLB supports a total of 256 queues, 128 load balanced queues
and 128 directed queues. Increase RTE_EVENT_MAX_QUEUES_PER_DEV
to max possible uint_8_t max value.

Signed-off-by: Timothy McDaniel <timothy.mcdaniel@intel.com>
Reviewed-by: Gage Eads <gage.eads@intel.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

eventdev: check input parameter for dump op

Rather than have drivers check for this, let's ensure the passed FILE *
is not NULL.

Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Timothy McDaniel <timothy.mcdaniel@intel.com>

app/regex: add job context

Store mbuf pointer associated with that job.

Signed-off-by: Guy Kaneti <guyk@marvell.com>
Acked-by: Ori Kam <orika@nvidia.com>

app/regex: configure queue according to capabilities

configure qp with OOS according to device capabilities
returned from rte_regexdev_info_get.

Signed-off-by: Guy Kaneti <guyk@marvell.com>
Acked-by: Ori Kam <orika@nvidia.com>

regex/mlx5: add out-of-order scan capability

Add missing out of order scan capability
RTE_REGEXDEV_CAPA_QUEUE_PAIR_OOS_F to mlx5 regex PMD.

Signed-off-by: Ophir Munk <ophirmu@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

regexdev: add out-of-order scan capability

Add out of order scan capability to check PMD
support for OOS.

Signed-off-by: Guy Kaneti <guyk@marvell.com>
Acked-by: Ori Kam <orika@nvidia.com>

pipeline: increase SWX immediate operand size

This patch increases the immediate operand size from 32 to 64 bits.

Signed-off-by: Venkata Suresh Kumar P <venkata.suresh.kumar.p@intel.com>
Acked-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

examples/l3fwd-power: fix race on interrupt wakeup log

Currently, the interrupt status notification prevents log spam by
remembering whether previous interrupt wakeup was due to traffic or due
to timeout expiring. However, it is a single variable that can
potentially be accessed from multiple threads, so it is not thread-safe.

Fix it by having per-lcore interrupt status.

Fixes: f4d1e19c293d ("examples/l3fwd-power: add Rx interrupt timeout")
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
Tested-by: Xi Zhang <xix.zhang@intel.com>

mbuf: remove seqn field

As announced in the deprecation note, the field seqn is removed to give
more space to the dynamic fields.

This is how the mbuf layout looks like (pahole-style):

word  type                              name                byte  size
 0    void *                            buf_addr;         /*   0 +  8 */
 1    rte_iova_t                        buf_iova          /*   8 +  8 */
      /* --- RTE_MARKER64               rearm_data;                   */
 2    uint16_t                          data_off;         /*  16 +  2 */
      uint16_t                          refcnt;           /*  18 +  2 */
      uint16_t                          nb_segs;          /*  20 +  2 */
      uint16_t                          port;             /*  22 +  2 */
 3    uint64_t                          ol_flags;         /*  24 +  8 */
      /* --- RTE_MARKER                 rx_descriptor_fields1;        */
 4    uint32_t             union        packet_type;      /*  32 +  4 */
      uint32_t                          pkt_len;          /*  36 +  4 */
 5    uint16_t                          data_len;         /*  40 +  2 */
      uint16_t                          vlan_tci;         /*  42 +  2 */
 5.5  uint64_t             union        hash;             /*  44 +  8 */
 6.5  uint16_t                          vlan_tci_outer;   /*  52 +  2 */
      uint16_t                          buf_len;          /*  54 +  2 */
 7    uint64_t                          timestamp;        /*  56 +  8 */
      /* --- RTE_MARKER                 cacheline1;                   */
 8    struct rte_mempool *              pool;             /*  64 +  8 */
 9    struct rte_mbuf *                 next;             /*  72 +  8 */
10    uint64_t             union        tx_offload;       /*  80 +  8 */
11    struct rte_mbuf_ext_shared_info * shinfo;           /*  88 +  8 */
12    uint16_t                          priv_size;        /*  96 +  2 */
      uint16_t                          timesync;         /*  98 +  2 */
12.5  uint32_t                          dynfield1[7];     /* 100 + 28 */
16    /* --- END                                             128      */

Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Thomas Monjalon <thomas@monjalon.net>

app/eventdev: switch sequence number to dynamic mbuf field

The order test stored a sequence number in the deprecated mbuf field
seqn.
It is moved to a dynamic field in order to allow removal of seqn.

Signed-off-by: David Marchand <david.marchand@redhat.com>

eventdev: switch sequence number to dynamic mbuf field

The eventdev drivers have been hacking the deprecated field seqn for
internal test usage.
It is moved to a dynamic mbuf field in order to allow removal of seqn.

Signed-off-by: David Marchand <david.marchand@redhat.com>

bus/fslmc: switch sequence number to dynamic mbuf field

The dpaa2 drivers have been hacking the deprecated field seqn for
internal features.
It is moved to a dynamic mbuf field in order to allow removal of seqn.

Signed-off-by: David Marchand <david.marchand@redhat.com>

bus/dpaa: switch sequence number to dynamic mbuf field

The dpaa drivers have been hacking the deprecated field seqn for
internal features.
It is moved to a dynamic mbuf field in order to allow removal of seqn.

Signed-off-by: David Marchand <david.marchand@redhat.com>

reorder: switch sequence number to dynamic mbuf field

The reorder library used sequence numbers stored in the deprecated field
seqn.
It is moved to a dynamic mbuf field in order to allow removal of seqn.

Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

net/ark: remove use of seqn for debug

The seqn mbuf field is deprecated.
It is currently hacked for debug purpose, it could be changed to a
dynamic field but I see little value in the debug info it offers.

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Ed Czeck <ed.czeck@atomicrules.com>

crypto/scheduler: remove unused internal seqn

This field has been left behind after dropping its use.

Fixes: 8a48e039432b ("crypto/scheduler: optimize crypto op ordering")

Signed-off-by: David Marchand <david.marchand@redhat.com>

event/dpaa2: remove dead code from self test

This code has never been used since introduction.

Fixes: 653242c3375a ("event/dpaa2: add self test")

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Nipun Gupta <nipun.gupta@nxp.com>

examples/fips_validation: fix leak on failure

The wb_data variable went out of scope on failure in the
get_writeback_data function. This is now freed before returning -1.

Coverity issue: 363453
Fixes: 952e10cdad5e ("examples/fips_validation: support scatter gather list")

Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>

examples/fips_validation: fix crash on allocation failure

The return value was not being checked when calling the
get_writeback_data function in the AES test case. On failure, this led
to a NULL dereference when using memcpy later. The return value is now
checked to avoid this NULL dereference.

Coverity issue: 363463
Fixes: 952e10cdad5e ("examples/fips_validation: support scatter gather list")

Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>

mbuf: remove userdata field

As announced in the deprecation note, the field userdata / udata64
is removed to give more space to the dynamic fields.

This is how the mbuf layout looks like (pahole-style):

word  type                              name                byte  size
 0    void *                            buf_addr;         /*   0 +  8 */
 1    rte_iova_t                        buf_iova          /*   8 +  8 */
      /* --- RTE_MARKER64               rearm_data;                   */
 2    uint16_t                          data_off;         /*  16 +  2 */
      uint16_t                          refcnt;           /*  18 +  2 */
      uint16_t                          nb_segs;          /*  20 +  2 */
      uint16_t                          port;             /*  22 +  2 */
 3    uint64_t                          ol_flags;         /*  24 +  8 */
      /* --- RTE_MARKER                 rx_descriptor_fields1;        */
 4    uint32_t             union        packet_type;      /*  32 +  4 */
      uint32_t                          pkt_len;          /*  36 +  4 */
 5    uint16_t                          data_len;         /*  40 +  2 */
      uint16_t                          vlan_tci;         /*  42 +  2 */
 5.5  uint64_t             union        hash;             /*  44 +  8 */
 6.5  uint16_t                          vlan_tci_outer;   /*  52 +  2 */
      uint16_t                          buf_len;          /*  54 +  2 */
 7    uint64_t                          timestamp;        /*  56 +  8 */
      /* --- RTE_MARKER                 cacheline1;                   */
 8    struct rte_mempool *              pool;             /*  64 +  8 */
 9    struct rte_mbuf *                 next;             /*  72 +  8 */
10    uint64_t             union        tx_offload;       /*  80 +  8 */
11    uint16_t                          priv_size;        /*  88 +  2 */
      uint16_t                          timesync;         /*  90 +  2 */
      uint32_t                          seqn;             /*  92 +  4 */
12    struct rte_mbuf_ext_shared_info * shinfo;           /*  96 +  8 */
13    uint64_t                          dynfield1[3];     /* 104 + 24 */
16    /* --- END                                             128      */

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

examples/rxtx_callbacks: switch TSC to dynamic field

The example used the deprecated mbuf field udata64.
It is moved to a dynamic field in order to allow removal of udata64.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

examples/bbdev: switch to dynamic mbuf field

The example used the deprecated mbuf field udata64 as input mbuf pointer.
It is moved to a dynamic field in order to allow removal of udata64.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

app/eventdev: switch flow ID to dynamic mbuf field

The order test stored the flow ID in the deprecated mbuf field udata64.
It is moved to a dynamic field in order to allow removal of udata64.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

test/graph: switch user data to dynamic mbuf field

The test used the deprecated mbuf field udata64.
It is moved to a dynamic field in order to allow removal of udata64.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>