common/cnxk: support UDP encapsulation

Adds support for UDP encapsulation in crypto_cn10k
PMD.

Signed-off-by: Srujana Challa <schalla@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

test/crypto: fix mempool size for session-less

Currently, private_data_offset for the sessionless is computed
wrongly which includes extra bytes added by
sizeof(struct rte_crypto_sym_xform) * 2. This causes buffer
overflow which leads to test application crash while freeing the
ops mempool. This patch provides fix for the same and also takes
care of increasing the length of ops to accommodate space for
rte_event_crypto_metadata while creating the crypto ops mempool.

Fixes: 3c2c535ecfc0 ("test: add event crypto adapter auto-test")
Cc: stable@dpdk.org
Signed-off-by: Abhinandan Gujjar <abhinandan.gujjar@intel.com>
Reported-by: Ciara Power <ciara.power@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/cnxk: reset feature flags on reconfigure

Feature flag in dev would be updated during config.
On reconfigure, the field need to be set again to
original value.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

compress/isal: support Arm platform

Isal compress PMD has build failures on Arm platform.

As dependent library ISA-L is supported on Arm platform,
support of the PMD is expanded to Arm architecture.
Fixed build failure caused by architecture specific code,
and made the PMD multi architecture compatible.

Bugzilla ID: 755
Signed-off-by: Ruifeng Wang <ruifeng.wang@arm.com>

compress/mlx5: fix memory region unregistration

The issue can cause illegal physical address access while a huge-page A
is released and huge-page B is allocated on the same virtual address.
The old MR can be matched using the virtual address of huge-page B but
the HW will access the physical address of huge-page A which is no more
part of the DPDK process.

Register a driver callback for memory event in order to free out all the
MRs of memory that is going to be freed from the dpdk process.

Fixes: f8c97babc9f4 ("compress/mlx5: add data-path functions")
Cc: stable@dpdk.org
Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

common/cpt: allocate auth key dynamically

Reduce session private data size by allocating
auth_key dynamically as required. Added auth_key_iova
to eliminate any impact on fastpath.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/cnxk: allocate auth key dynamically

Reduce session private data size by allocating
auth_key dynamically as required.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/octeontx2: fix lookaside IPsec IV pointer

In case of AES-GCM/CCM, nonce/salt comes along
with IV, hence can be copied in a single memcpy.
This patch fixes the IV copy in lookaside IPsec
outbound instruction.

Fixes: fab634eb87ca ("crypto/octeontx2: support security session data path")
Cc: stable@dpdk.org
Signed-off-by: Tejasree Kondoj <ktejasree@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

net/octeontx2: clear SA valid during session destroy

SA table entry would be reserved for inline inbound operations. Clear
valid bit of the SA so that CPT would treat SA entry as invalid. Also,
move setting of valid bit to the end in case of session_create() to
eliminate possibility of hardware seeing partial data.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

net/octeontx2: add lock for inline IPsec tables

Add locking for IPsec table updates.

Fixed error handling to clear SA entry if the SA
population functions encounters any error.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/octeontx2: fix IPsec session member overlap

The member 'dir' should not overlap with 'ip'. Usage of union for all
members would mean dir would get corrupt.

Fixes: e91b4f45ff54 ("net/octeontx2: support anti-replay for security session")
Cc: stable@dpdk.org
Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/cnxk: update instruction queue in start/stop

The instruction queue is enabled in dev start and
is disabled in dev stop.

Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/cnxk: move instruction queue enable to ROC

The code for enabling instruction queue is moved to ROC API.

Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: update raw data path

This commit updates the QAT raw data-path API to support the
changes made to device and sessions. The QAT RAW data-path API
now works on Generation 1-3 devices and is disabled on GEN4.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
Signed-off-by: Adam Dybkowski <adamx.dybkowski@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/qat: add service discovery

This commit adds service discovery to generation four
of Intel QuickAssist Technology devices.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/qat: reset ring pairs before setting GEN4

This commit resets ring pairs of particular vf before
setting PMD.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/qat: add PF to VF communication

Add communication between physical device and virtual function
in Intel QucikAssist Technology PMD.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: support GMAC in GEN4 legacy mode

Add AES-GMAC algorithm in legacy mode to generation 4 devices.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: add Chacha-Poly in UCS-SPC mode

This commit adds Chacha20-Poly1305 aglorithm that works
in UCS (Unified crypto slice) SPC(Single-Pass) mode.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: add AES-GCM in UCS-SPC mode

This commit adds AES-GCM algorithm that works
in UCS (Unified crypto slice) SPC(Single-Pass) mode.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: rework init common header

Rework init common header function for request
descriptor so it can be called only once.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: support legacy GCM and CCM

Add AES-GCM, AES-CCM algorithms in legacy mode.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: rename content descriptor functions

Content descriptor functions are incorrectly named,
having them with proper name will improve readability and
facilitate further work.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: support GEN4 unified cipher slice

This commit adds unified cipher slice(UCS) to Intel QuickAssist
Technology PMD and enables AES-CTR algorithm.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

crypto/qat: enable GEN4 legacy algorithms

This commit enables algorithms labeled as 'legacy'
on QAT generation 4 devices.
Following algorithms were enabled:
* AES-CBC
* AES-CMAC
* AES-XCBC MAC
* NULL (auth, cipher)
* SHA1-HMAC
* SHA2-HMAC (224, 256, 384, 512)

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/qat: support GEN4 devices

This commit adds support for fourth generation (GEN4) of
Intel QuickAssist (QAT) Technology devices.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

common/qat: rework queue pair per service

Different generations of Intel QuickAssist Technology devices may
differ in approach to allocate queues. Queue pair number function
therefore needs to be more generic.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

examples/pipeline: fix out of bounds array access

Fix the incorrect array out of bounds check within the function
pipeline_selector_group_member_read().

Coverity issue: 371911
Fixes: 598fe0dd0d8e3 ("examples/pipeline: support selector table")
Signed-off-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

pipeline: fix selector freeing

Due to a typo, the selector_free() function incorrectly takes an early
return when the selectors array is non-NULL, as opposed to the other
way around.

Coverity issue: 371912
Fixes: cdaa937d3eaab ("pipeline: support selector table")
Signed-off-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

test/power: fix CPU frequency check for intel_pstate

For acpi_cpufreq and cppc_cpufreq, both cpuinfo_cur_freq and
scaling_cur_freq exist. For pstate, only scaling_cur_freq exists.
And value in scaling_cur_freq and cpuinfo_cur_freq may not be the
same. For acpi_cpufreq and cppc_cpufreq, we should check
cpuinfo_cur_freq but not scaling_cur_freq. So here change the
check sequence to make sure it works for all cpufreq drivers.
Besides, add rounding for pstate driver.

Fixes: ff6dfb8e492f ("test/power: fix CPU frequency check")
Cc: stable@dpdk.org
Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>
Acked-by: David Hunt <david.hunt@intel.com>

power: fix crash on error for intel_pstate

Currently, the error paths can lead to attempts at dereferencing NULL
pointers. Add the check to avoid attempts at dereferencing NULL
pointers.

Coverity issue: 371895
Coverity issue: 371889
Fixes: 06cffd468fdd ("power: refactor ACPI and intel_pstate support")
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>

raw/ioat: fix termination descriptor for batch

When batch_size == 1, idxd has to add a dummy termination descriptor
to satisfy HW requirements.
Right now it uses NOP descriptor with FENCE flag.
This is excessive and fencing can slowdown things quite significantly.
The patch removes FENCE flag from termination dummy descriptor.
That helps to improve performance for no-burst scenarios.

Fixes: 245efe544d8e ("raw/ioat: report status of completed jobs")
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

raw/ioat: fix config script queue size calculation

The queue size calculation is currently based on "max_tokens" rather than
"max_work_queues_size". This is resulting in the queue size being
incorrectly configured when using the script to configure devices bound to
the IDXD kernel driver.
This patch fixes this miscalculation so devices are configured with
appropriate queue size.

Fixes: 01863b9d2354 ("raw/ioat: include example configuration script")
Cc: stable@dpdk.org
Reported-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Signed-off-by: Kevin Laatz <kevin.laatz@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

distributor: fix 128-bit write alignment

When the distributor sample app is built as a 32-bit app,
the data buffer passed to find_match_vec can be unaligned,
causing a segmentation fault due to writing a 128-bit value
using _mm_store_si128().  128-bit align the data being
passed in so this does not happen.

Fixes: 775003ad2f96 ("distributor: add new burst-capable library")
Cc: stable@dpdk.org
Signed-off-by: David Hunt <david.hunt@intel.com>

net/sfc: support count action in flow query

The query reports the number of hits for a counter associated
with a flow rule.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: support flow action count in transfer rules

For now, a rule may have only one dedicated counter, shared counters
are not supported.

HW delivers (or "streams") counter readings using special packets.
The driver creates a dedicated Rx queue to receive such packets
and requests that HW start "streaming" the readings to it.

The counter queue is polled periodically, and the first available
service core is used for that. Hence, the user has to specify at least
one service core for counters to work. Such a core is shared by all
MAE-capable devices managed by sfc driver.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

common/sfc_efx/base: add packetiser packet format definition

Packetiser composes packets with MAE counters update.

Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>

common/sfc_efx/base: add max MAE counters to limits

The information about the maximum number of MAE counters is
crucial to the counter support in the driver.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: add Rx datapath method to get pushed buffers count

The information about the number of pushed Rx buffers is required
for counter Rx queue to know when to give credits to counter
stream.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

common/sfc_efx/base: support counter in action set

User will be able to associate counter with MAE action set to
collect counter packets and bytes for a specific action set.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

common/sfc_efx/base: add counter stream MCDI wrappers

The MCDIs will be used to control counter Rx queue packet flow.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

common/sfc_efx/base: add counter creation MCDI wrappers

User will be able to create and free MAE counters. Support for
associating counters with action set will be added in upcoming
patches.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: reserve internal Rx queue for counters

MAE delivers counters data as special packets via dedicated Rx queue.
Reserve an RxQ so that it does not interfere with ethdev Rx queues.
A routine will be added later to handle these packets.

There is no point to reserve the queue if no service cores are
available and counters cannot be used.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: add NUMA-aware registry of service logical cores

The driver requires service cores for housekeeping. Share these
cores for many adapters and various purposes to avoid extra CPU
overhead.

Since housekeeping services will talk to NIC, it should be possible
to choose logical core on matching NUMA node.

Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: support initialising different Rx queue types

Add extra EFX flags to RxQ info initialization API to support
choosing different RxQ types and make the API public to use
it in for counter queues.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>

net/sfc: add abstractions for the management EVQ identity

Add a function returning management event queue software index.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>

common/sfc_efx/base: add user mark RxQ flag

Add a flag to request support for user mark field on an RxQ.
The field is required to retrieve generation count value from
counter RxQ.

Implement it only for Riverhead and EF10 ESSB since they support
the field in the Rx prefix.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

common/sfc_efx/base: add ingress m-port RxQ flag

Add a flag to request support for ingress m-port on an RxQ.
Implement it only for Riverhead, other families will return an error
if the flag is set.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: prepare for internal Tx queue

Make software index of a Tx queue and ethdev index separate.
When an ethdev TxQ is accessed in ethdev callbacks, an explicit ethdev
queue index is used.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

net/sfc: explicitly control IRQ used for Rx queues

Interrupts support has assumptions on interrupt numbers used
for LSC and Rx queues. The first interrupt is used for LSC,
subsequent interrupts are used for Rx queues.

Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>

common/sfc_efx/base: support custom EvQ to IRQ mapping

Custom mapping is actually supported for EF10 and EF100 families only.

A driver (e.g. DPDK PMD) may require to customize mapping of EvQ
to interrupts if, for example, extra EvQ are used for house-keeping
in polling or wake up (via another EvQ) mode.

Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>

common/sfc_efx/base: separate target EvQ and IRQ config

Target EvQ and IRQ number are specified in the same location
in MCDI request. The value is treated as IRQ number if the
event queue is interrupting (corresponding flag is set) and
as target event queue otherwise.

However it is better to separate it on helper API level to
make it more clear.

Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>

net/sfc: do not enable interrupts on internal Rx queues

rxq_intr flag requests support for interrupt mode for ethdev Rx queues.
There is no internal Rx queues yet.

Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

net/sfc: prepare for internal Rx queue

Make software index of an Rx queue and ethdev index separate.
When an ethdev RxQ is accessed in ethdev callbacks, an explicit ethdev
queue index is used.

Signed-off-by: Igor Romanov <igor.romanov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Andy Moreton <amoreton@xilinx.com>
Reviewed-by: Ivan Malov <ivan.malov@oktetlabs.ru>

examples/pipeline: fix build

This patch fixes the following build failures seen on Ubuntu 16.04
with gcc 5.4.0 because of uninitialized variables:
...
examples/pipeline/cli.c:1559:11: error: 'weight_val' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:1545:13: error: 'member_id_val' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:1538:12: error: 'group_id_val' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:2189:2: error: 'idx1' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:2179:43: error: 'idx0' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:2265:2: error: 'idx1' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:2248:43: error: 'idx0' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:2358:2: error: 'idx1' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]
...
examples/pipeline/cli.c:2325:43: error: 'idx0' may be used
  uninitialized in this function [-Werror=maybe-uninitialized]

Fixes: 598fe0dd0d8e ("examples/pipeline: support selector table")
Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
Acked-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

event/cnxk: support vectorized Tx event fast path

Add Tx event vector fastpath, integrate event vector Tx routine
into Tx burst.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

event/cnxk: support vectorized Rx event fast path

Add Rx event vector fastpath to convert HW defined metadata into
rte_mbuf and rte_event_vector.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

event/cnxk: support vectorized Rx adapter

Add event vector support for cnxk event Rx adapter, add control path
APIs to get vector limits and ability to configure event vectorization
on a given Rx queue.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

event/cnxk: support Tx adapter fast path

Add support for event eth Tx adapter fastpath operations.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

event/cnxk: support Tx adapter

Add support for event eth Tx adapter.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Nithin Dabilpuram <ndabilpuram@marvell.com>

event/cnxk: support Rx adapter fast path

Add support for event eth Rx adapter fastpath operations.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

event/cnxk: support Rx adapter

Add support for event eth Rx adapter.
Resize cn10k workslot fastpath structure to fit in 64B cacheline size.

Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

net/iavf: fix bandwidth unit in TM capability query

In IAVF node TM capability querying, the unit of bandwidth is Kbps,
which is not correct according to TM specification. Change the unit to
Byte per second. Refine some unclear comments as well.

Fixes: 44d0a720a538 ("net/iavf: query QoS capabilities and set queue TC mapping")
Signed-off-by: Ting Xu <ting.xu@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/ice/base: support MPLS ethertype switch filter

Add MPLS training packet and offsets.
Add check to identify MPLS ethertype filters.

For example:
testpmd> flow create 0 ingress pattern eth dst is 00:11:22:33:44:55 \
         type is 0x8847 / end actions queue index 2 / end

This flow will result in all the matched ingress packets be
forwarded to queue 2.

Signed-off-by: Alvin Zhang <alvinx.zhang@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/ice: fix ESP flow director with SPI as input set

FDIR can't work when SPI as inputset for both ESP over IP and ESP
over UDP flow. This patch fixes this issue by adding the corresponding
input set for ESP over IP and ESP over UDP when parsing input set. Also,
it adds input set bit for NAT_T_ESP to distinguish ESP over IP and ESP
over UDP.

Fixes: 70feafc1a3f2 ("net/ice: support ESP/NATT flow director to match outer IP")
Signed-off-by: Simei Su <simei.su@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/ice/base: revert change of first profile mask

Segmentation fault mentioned in below commit is related to
other root cause under investigation.
This reverts patch below since it may have potential
risk and side effect if the first profile mask is set to 0.

Fixes: 148fdf2d3537 ("net/ice/base: fix first profile mask")
Cc: stable@dpdk.org
Signed-off-by: Wenjun Wu <wenjun1.wu@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/i40e: replace SMP barrier with thread fence in Rx

Simply replace the SMP barrier with atomic thread fence for
i40e hw ring scan, if there is no synchronization point.

Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/iavf: support default RSS for IP fragment

This patch adds default RSS support for IPv4 and IPv6 fragment packet.

Signed-off-by: Wenjun Wu <wenjun1.wu@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/iavf: support RSS for GTPoGRE

Support AVF RSS for inner most header of GTPoGRE packet. It supports
RSS based on inner most IP src + dst address and TCP/UDP src + dst
port.

Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/iavf: support flow director for GTPoGRE

Support AVF FDIR for inner header of GTPoGRE tunnel packet.
Only patterns without inner most L3,L4 header support outer L3 src/dst
and TEID,QFI FDIR.

+------------------------------------+-------------------------------+
|                Pattern             |            Input Set          |
+------------------------------------+-------------------------------+
|eth/ipv4/gre/ipv4/gtpu/(eh/)ipv4    |inner: src/dst ip              |
|eth/ipv4/gre/ipv4/gtpu/(eh/)ipv4/udp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv4/gtpu/(eh/)ipv4/tcp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv4/gtpu/(eh/)ipv6    |inner: src/dst ip              |
|eth/ipv4/gre/ipv4/gtpu/(eh/)ipv6/udp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv4/gtpu/(eh/)ipv6/tcp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv6/gtpu/(eh/)ipv4    |inner: src/dst ip              |
|eth/ipv4/gre/ipv6/gtpu/(eh/)ipv4/udp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv6/gtpu/(eh/)ipv4/tcp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv6/gtpu/(eh/)ipv6    |inner: src/dst ip              |
|eth/ipv4/gre/ipv6/gtpu/(eh/)ipv6/udp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv6/gtpu/(eh/)ipv6/tcp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv4/gtpu/(eh/)ipv4    |inner: src/dst ip              |
|eth/ipv6/gre/ipv4/gtpu/(eh/)ipv4/udp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv4/gtpu/(eh/)ipv4/tcp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv4/gtpu/(eh/)ipv6    |inner: src/dst ip              |
|eth/ipv6/gre/ipv4/gtpu/(eh/)ipv6/udp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv4/gtpu/(eh/)ipv6/tcp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv6/gtpu/(eh/)ipv4    |inner: src/dst ip              |
|eth/ipv6/gre/ipv6/gtpu/(eh/)ipv4/udp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv6/gtpu/(eh/)ipv4/tcp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv6/gtpu/(eh/)ipv6    |inner: src/dst ip              |
|eth/ipv6/gre/ipv6/gtpu/(eh/)ipv6/udp|inner: src/dst ip, src/dst port|
|eth/ipv6/gre/ipv6/gtpu/(eh/)ipv6/tcp|inner: src/dst ip, src/dst port|
|eth/ipv4/gre/ipv4/gtpu(/eh)         |outer: src/dst ip, teid(,qfi)  |
|eth/ipv4/gre/ipv6/gtpu(/eh)         |outer: src/dst ip, teid(,qfi)  |
|eth/ipv6/gre/ipv4/gtpu(/eh)         |outer: src/dst ip, teid(,qfi)  |
|eth/ipv6/gre/ipv6/gtpu(/eh)         |outer: src/dst ip, teid(,qfi)  |
+------------------------------------+-------------------------------+

Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/iavf: support flow pattern for GTPoGRE

Add GTPoGRE pattern support for AVF FDIR and RSS.

Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
Acked-by: Qi Zhang <qi.z.zhang@intel.com>

net/bnxt: update CFA resource types

Update cfa_resource_types.h to add a new entry for compatibility with FW.

Signed-off-by: Shuanglin Wang <shuanglin.wang@broadcom.com>
Reviewed-by: Randy Schacher <stuart.schacher@broadcom.com>
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

net/bnxt: clear cached statistics

As part of the workaround put in the commit "219842b9990c",
driver caches the last read stats values from the hardware.
But this is not cleared during the clear stats operation. This
results in showing up stale stats values while reading the stats
after the clear operation.

Fixes: 219842b9990c ("net/bnxt: workaround spurious zero stats in Thor")
Cc: stable@dpdk.org
Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Lance Richardson <lance.richardson@broadcom.com>
Reviewed-by: Somnath Kotur <somnath.kotur@broadcom.com>

net/bnxt: handle pause storm event

FW has been modified to send a new async event when it detects
a pause storm. Register for this new event and log it upon receipt.

Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

net/bnxt: refactor async event handling

Store the async event completion data1 and data2 in separate variables
at the start of the function before the switch case for the different
events so they can be used by any of the event handlers.

Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

net/bnxt: inform firmware about host MTU

This enables device firmware to respond appropriately to BMC queries
about the driver's configured MTU.

Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Lance Richardson <lance.richardson@broadcom.com>

net/bnxt: update HSI structure

- HWRM version updated to 1.10.2.44
- Added corresponding driver changes for the Admin MTU field name change.

Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Somnath Kotur <somnath.kotur@broadcom.com>

net/bnxt: fix nested lock during bonding

Bnxt PMD registers LSC callback (bond_ethdev_lsc_event_callback) when
working at bond mode. This callback will dead lock when LSC
interrupt triggered.

lsc interrupt ->
bnxt_handle_async_event ->
bnxt_link_update_op ->
bond_ethdev_lsc_event_callback (lsc_lock) ->
bnxt_link_update_op ->
bond_ethdev_lsc_event_callback (lsc_lock dead lock)

Fixes: c2faa1d1969e ("net/bnxt: add support for LSC interrupt event")
Cc: stable@dpdk.org
Signed-off-by: Weifeng Li <liweifeng96@126.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

net/bnxt: fix missing barriers in completion handling

Ensure that Rx/Tx/Async completion entry fields are accessed
only after the completion's valid flag has been loaded and
verified. This is needed for correct operation on systems that
use relaxed memory consistency models.

Fixes: 2eb53b134aae ("net/bnxt: add initial Rx code")
Fixes: 6eb3cc2294fd ("net/bnxt: add initial Tx code")
Cc: stable@dpdk.org
Signed-off-by: Lance Richardson <lance.richardson@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>

net/cnxk: support raw flow pattern

Add support for rte_flow_item_raw to parse custom L2 and L3
protocols.

Signed-off-by: Satheesh Paul <psatheesh@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

common/cnxk: support custom L2/L3 protocols parsing

Add roc API for parsing custom L2 and L3 protocols.

Signed-off-by: Satheesh Paul <psatheesh@marvell.com>
Reviewed-by: Kiran Kumar K <kirankumark@marvell.com>

net/cnxk: update link status when device stopped

Set link status to down and don't fetch link status from kernel
when device in stopped state.

Signed-off-by: Satha Rao <skoteshwar@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

net/octeontx2: fix TM node statistics query

Until hierarchy committed TM hardware resources are not allocated
for node.
This patch check for status of HW resources before reading statistics.

Fixes: 1e25d57fae38 ("net/octeontx2: add TM stats and shaper profile")
Cc: stable@dpdk.org
Signed-off-by: Satha Rao <skoteshwar@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

net/octeontx2: handle link status when device stopped

Set link status to down and don't fetch link status from kernel
when device in stopped state.

Signed-off-by: Satha Rao <skoteshwar@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

net/cnxk: fix default MCAM allocation size

Preallocation of MCAM entries is not valid anymore since the
AF side MCAM allocation scheme has changed. This patch disables
preallocation by changing the default MCAM preallocation size
from 8 to 1.

Fixes: 168c59cfe42 ("net/octeontx2: add flow MCAM utility functions")
Cc: stable@dpdk.org
Signed-off-by: Satheesh Paul <psatheesh@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

net/octeontx2: support non-ethernet L2 header

In the inline inound path, a custom header would be present at L3 which
has sequence number & SPI. L2 need to be adjusted such that the eventual
packet would have L3 after L2. Remove assumption of L2 type in this
handling.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

net/mvpp2: fix not supported VLAN operations status

vlan_strip and vlan_extend features need to return "unsupported"
error value.

Fixes: ff0b8b10dc4 ("net/mvpp2: support VLAN offload")
Cc: stable@dpdk.org
Signed-off-by: Meir Levi <mlevi4@marvell.com>
Reviewed-by: Liron Himi <lironh@marvell.com>

net/mvpp2: fix configured state dependency

Need to set configure flag to allow create and commit mrvl tm
hierarchy tree. tm configuration depends on parameters that are
being set in port configure stage, e.g. nb_tx_queues.
This also aligned with the tm api description.

Fixes: 429c394417 ("net/mvpp2: support traffic manager")
Cc: stable@dpdk.org
Signed-off-by: Dana Vardi <danat@marvell.com>
Reviewed-by: Liron Himi <lironh@marvell.com>

net/mvpp2: fix port speed overflow

ethtool_cmd_speed return uint32 and after the arithmetic
operation in mrvl_get_max_rate func the result is out of range.

Fixes: 429c394417 ("net/mvpp2: support traffic manager")
Cc: stable@dpdk.org
Signed-off-by: Dana Vardi <danat@marvell.com>
Reviewed-by: Liron Himi <lironh@marvell.com>

net/mlx5: fix typo in vectorized Rx comments

Change "returing" to "returning".

Fixes: 2e542da70937 ("net/mlx5: add Altivec Rx")
Fixes: 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")
Fixes: 3c2ddbd413e3 ("net/mlx5: separate shareable vector functions")
Cc: stable@dpdk.org
Signed-off-by: Sarosh Arif <sarosh.arif@emumba.com>

net/mlx5: fix threshold for mbuf replenishment in MPRQ

The replenishment scheme for the vectorized MPRQ Rx burst aims
to improve the cache locality by allocating new mbufs only when
there are almost no mbufs left: one burst gap between allocated
and consumed indexes.

This gap is not big enough to accommodate a corner case when we
have a very aggressive CQE compression with multiple regular CQEs
at the beginning and 64 zipped CQEs at the end.

Need to keep in mind this case and extend the replenishment
threshold by MLX5_VPMD_RX_MAX_BURST (64) to avoid mbuf overflow.

Fixes: 5fc2e5c27d6 ("net/mlx5: fix mbuf overflow in vectorized MPRQ")
Cc: stable@dpdk.org
Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

net/mlx5: fix missing RSS expansion of IPv6 frag

IPV6_FRAG_EXT item is missed for RSS expansion which causes wrongly
expanded flows:
flow create 0 ingress pattern eth / ipv6 / udp dst is 250 / vxlan-gpe /
ipv6 / ipv6_frag_ext / end actions rss level 2 types ip end / end

Different from other items, IPV6_FRAG_EXT hasn't next field because HW
only support to do hash of UDP/TCP for non-fragment.

This MLX5_EXPANSION_IPV6_FRAG_EXT node in RSS expansion graph only helps
RSS expansion function to locate right node in graph from which start
to expand.

Fixes: 0e5a0d8f7556 ("net/mlx5: support match on IPv6 fragment extension")
Cc: stable@dpdk.org
Signed-off-by: Xiaoyu Min <jackmin@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: fix missing RSS expandable items

Some RSS expandable items are missing which leads to the expanded
rte flow rules with wrong patterns.

Fix by adding missed items.

Fixes: d91093b9a2af ("net/mlx5: fix RSS pattern expansion")
Cc: stable@dpdk.org
Signed-off-by: Xiaoyu Min <jackmin@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: support flow matchng on IPv4 IHL

Query MLX5 port hardware if it is capable to offload IPv4
IHL field.

Provide flow rules capability to match on IPv4 IHL field.
Minimal HCA firmware version required to offload IPv4 IHL is
xx_30_2000.

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

doc: add multi-thread flow rate optimizations for mlx5

This commit adds the multiple-thread flow insertion optimization
description.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: optimize Rx queue match

As hrxq struct has the indirect table pointer, while matching the
hrxq, better to use the hrxq indirect table instead of searching
from the list.

This commit optimizes the hrxq indirect table matching.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: change memory release configuration

This commit changes the index pool memory release configuration
to 0 when memory reclaim mode is not required.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: optimize hash list table allocate on demand

Currently, all the hash list tables are allocated during start up.
Since different applications may only use dedicated limited actions,
optimized the hash list table allocate on demand will save initial
memory.

This commit optimizes hash list table allocate on demand.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: enable indexed pool per-core cache

This commit enables the tag and header modify action indexed
pool per-core cache in non-reclaim memory mode.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

net/mlx5: adjust hash bucket size

With the new per core optimization to the list, the hash bucket size
can be tuned to a more accurate number.

This commit adjusts the hash bucket size.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>