test/ring: add unit tests for exact size rings

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

ring: allow non power-of-2 sizes

The rte_rings traditionally have only supported having ring sizes as powers
of 2, with the actual usable space being the size - 1. In some cases, for
example, with an eventdev where we want to precisely control queue depths
for latency, we need to allow ring sizes which are not powers of two so we
add in an additional ring capacity value to allow that. For existing rings,
this value will be size-1, i.e. the same as the mask, but if the new
EXACT_SZ flag is passed on ring creation, the ring will have exactly the
usable space requested, although the underlying memory size may be bigger.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>

event/octeontx: add enqueue fwd op variant

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Acked-by: Gage Eads <gage.eads@intel.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

event/octeontx: add enqueue new op variant

OCTEONTX can have optimized handling of events if the PMD
knows it is a producer pattern in advance and it can support
burst mode if all the events has op == RTE_EVENT_OP_NEW.

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Acked-by: Gage Eads <gage.eads@intel.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

eventdev: introduce specialized enqueue forward op variant

Introducing the rte_event_enqueue_new_burst() for enabling the
PMD, an optimization opportunity to optimize if all the events in
the enqueue burst has the op type of RTE_EVENT_OP_FORWARD.

If a PMD does not have any optimization opportunity
for this operation then the PMD can choose the generic enqueue
burst PMD callback as the fallback.

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Acked-by: Gage Eads <gage.eads@intel.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

eventdev: introduce specialized enqueue new op variant

Introducing the rte_event_enqueue_new_burst() for enabling the
PMD, an optimization opportunity to optimize if all the events in
the enqueue burst has the op type of RTE_EVENT_OP_NEW.

If a PMD does not have any optimization opportunity
for this operation then the PMD can choose the generic enqueue
burst PMD callback as the fallback.

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Acked-by: Gage Eads <gage.eads@intel.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

eventdev: introduce helper function for enqueue burst

Introducing a helper function to avoid duplicating
common enqueue burst code when introducing
enqueue burst variants.

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Acked-by: Gage Eads <gage.eads@intel.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>

doc: add NXP DPAA2 eventdev guide

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>
Acked-by: John McNamara <john.mcnamara@intel.com>

event/dpaa2: handle timeout using interrupts in dequeue

This patch adds support for interrupt handling on the event port.
These interrupts facilitates managing of timeout ticks in the
event dequeue functions.

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: enable portal interrupt handling

Eventdev requires portal interrupts to handle timeout in the
event dequeue. This patch provides mechanism to enable the
portal interrupts.

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: add interrupt enabling routine

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

event/dpaa2: add enqueue and dequeue functionality

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: change argument to const to avoid warning

qbman_get_dqrr_idx() API is required with constant dqrr entry
in the eventdev driver. Also, this routine is not updating the
dqrr. So, this patch updates its input argument to a const type.

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: add callback per queue to enable

Dequeue from event device needs to process the event on
the basis of the hardware queue from which it is dequeued.
A callback is added into dpaa2_queue structure, to enable
event dequeue functionality to call that processing routine.

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: support enqueue with multiple descriptors

This patch adds the QBMAN API which support multiple enqueue
descriptors.

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

event/dpaa2: add configuration functions

This patch adds all the configuration API's for DPAA2 eventdev
including device config, start, stop & port and queue
related API's

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: support static dequeue from portal

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

event/dpaa2: initialize device

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: adding cpu support in stashing config

Stashing can also be configured by other drivers (for instance
event driver) passing cpu_id as an argument. This change
facilitates the same.

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: register dpci as dpaa2 device for bus scan

Registering dpci as dpaa2 type device handling initialization,
allocation and freeing of the device

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: adding basic dpci support

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

event/dpaa2: register dpcon as dpaa2 device for bus scan

Registering dpcon as dpaa2 type device handling initialization,
allocation and freeing of the device

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: export qbman dqrr funcs for eventdev usages

Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>

bus/fslmc: add basic dpcon support

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: integrate dpio and dpbp to object framework

This patch removes the existing static call for dpio and dpbp
create and add them to object registration framework.

This patch also changes the vfio mc object processing to use
the framework.

Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

bus/fslmc: generic framework for mc object creation

There are muliple help mc object, which are not an independent
device, but they are required for dpaa2 based devices.
This framework allows registration and handling of all such
mc devices.

Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>
Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>

event/dpaa2: add basic build infrastructure

Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>
Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>

drivers: add bus dependency for eventdev

Signed-off-by: Hemant Agrawal <hemant.agrawal@nxp.com>

app/crypto-perf: wait for cores launched by app

After test is finished the main thread waits only for lcores
where runners were started because, in case of using the
multicore scheduler, more cores are launched that do not need
to be waited for.

Signed-off-by: Kirill Rybalchenko <kirill.rybalchenko@intel.com>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

crypto/qat: fix possible out-of-bounds

Out-of-bounds access possible if ctx.qat_cipher_alg has invalid value.
This should never happen at this point on data path, but fix for safety.

Coverity issue: 143458, 143465

Fixes: d18ab45f7654 ("crypto/qat: support DOCSIS BPI mode")

Signed-off-by: Fiona Trahe <fiona.trahe@intel.com>

doc: add new crypto session information

Modified cryptodev library section in Programmer's Guide,
with the recent changes in the crypto sessions.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: John McNamara <john.mcnamara@intel.com>

cryptodev: remove session init internal function

Since now the private session data is initialized after
the session pool is created, there is no need to keep
this PMD function.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: add mempool pointer in queue pair setup

The session mempool pointer is needed in each queue pair,
if session-less operations are being handled.
Therefore, the API is changed to accept this parameter,
as the session mempool is created outside the
device configuration function, similar to what ethdev
does with the rx queues.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: support device independent sessions

Change crypto device's session management to make it
device independent and simplify architecture when session
is intended to be used on more than one device.

Sessions private data is agnostic to underlying device
by adding an indirection in the sessions private data
using the crypto driver identifier.
A single session can contain indirections to multiple device types.

New function rte_cryptodev_sym_session_init has been created,
to initialize the driver private session data per driver to be
used on a same session, and rte_cryptodev_sym_session_clear
to clear this data before calling rte_cryptodev_sym_session_free.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: remove mempool from session

Mempool pointer can be obtained from the object itself,
which means that it is not required to actually store the pointer
in the session.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: remove driver id from session

Since crypto session will not be attached to a specific
device or driver, the field driver_id is not required
anymore (only used to check that a session was being
handled by the right device).

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: remove device id from session

Device id is necessary in the crypto session,
as it was only used for the functions that attach/detach
a session to a queue pair.

Since the session is not going to be attached to a device
anymore, this is field is no longer necessary.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: change attach session to queue pair API

Device id is going to be removed from session,
as the session will be device independent.
Therefore, the functions that attach/dettach a session
to a queue pair need to be updated, to accept the device id
as a parameter, apart from the queue pair id and the session.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: do not create session mempool internally

Instead of creating the session mempool while configuring
the crypto device, apps will create the mempool themselves.
This way, it gives flexibility to the user to have a single
mempool for all devices (as long as the objects are big
enough to contain the biggest private session size) or
separate mempools for different drivers.

Also, since the mempool is now created outside the
device configuration function, now it needs to be passed
through this function, which will be eventually passed
when setting up the queue pairs, as ethernet devices do.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: add private session size retrieval function

Provide a function to get the private session size
of any crypto device (specifically, to its crypto driver).

This will be useful once the session mempool is created
outside the library.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: move session init out of pool creation

Prior to removing the session pool creation from cryptodev
configure function, session init function needs to be
separated from the pool creation.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: remove unused cryptodev session structure

Cryptodev session structure was a duplication of the
cryptodev symmetric structure.
It was used by some PMDs that should use the symmetric
structure instead.

Since this structure was internal, there is no deprecation
notice required.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>

crypto/scheduler: add multicore scheduling mode

Multi-core scheduling mode is a mode where scheduler distributes
crypto operations in a round-robin base, between several core
assigned as workers.

Signed-off-by: Kirill Rybalchenko <kirill.rybalchenko@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

cryptodev: remove crypto device driver name

Remove crypto device driver name string definitions from librte_cryptodev,
which avoid to library changes every time a new crypto driver was added.

The driver name is predefined internaly in the each PMD.
The applications could use the crypto device driver names based on
options with the driver name string provided in command line.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>

cryptodev: remove crypto device type enumeration

Changes device type identification to be based on a unique
driver id replacing the current device type enumeration, which needed
library changes every time a new crypto driver was added.

The driver id is assigned dynamically during driver registration using
the new macro RTE_PMD_REGISTER_CRYPTO_DRIVER which returns a unique
uint8_t identifier for that driver. New APIs are also introduced
to allow retrieval of the driver id using the driver name.

Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>

crypto/dpaa2_sec: update driver name string

while registering driver to dpaa2, hard coded string is used.
It is now updated as per the latest changes in string name.

Signed-off-by: Akhil Goyal <akhil.goyal@nxp.com>

test/crypto: extend AES-GCM 192/256 to other PMDs

QAT, OpenSSL and DPAA2 PMDs support 192-bit and 256-bit key sizes
for AES-GCM, apart from 128-bit keys. Therefore, these tests that were
only used by AES-NI MB PMD are extended to these other two PMDs.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

test/crypto: add AES-GCM 192

Added AES GCM tests for 192-bit keys, now that AES GCM PMD
supports it.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

test/crypto: rename some tests

AES-GCM tests are used for AES-NI MB, OpenSSL,
QAT and DPAA2_SEC PMDs, so they are not specific to the first one.
Therefore, _mb_ prefix can be removed.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

crypto/aesni_gcm: migrate to Multi-buffer library

Since Intel Multi Buffer library for IPSec has been updated to
support Scatter Gather List, the AESNI GCM PMD can link
to this library, instead of the ISA-L library.

This move eases the maintenance of the driver, as it will
use the same library as the AESNI MB PMD.
It also adds support for 192-bit keys.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Signed-off-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>

test/crypto: add 12-byte IV AES-CTR test cases

AESNI MB PMD supports now 12-byte IVs, so some tests
are added to check that, apart from the ones with 16-byte IVs,
as this is still compatible.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>

crypto/aesni_mb: support IPSec Multi-buffer lib v0.46

IPSec Multi-buffer library v0.46 has been released,
which includes, among othe features, support for 12-byte IV,
for AES-CTR, keeping also the previous 16-byte IV,
for backward compatibility reasons.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>

crypto/dpaa2_sec: support AES-GCM and CTR

AES-GCM support is added as per the AEAD type of crypto
operations. Support for AES-CTR is also added.

test/crypto and documentation is also updated for
dpaa2_sec to add supported algorithms.

Signed-off-by: Akhil Goyal <akhil.goyal@nxp.com>

crypto/dpaa2_sec: add HW desc support for AES-GCM

Signed-off-by: Akhil Goyal <akhil.goyal@nxp.com>

crypto/dpaa2_sec: add HW desc support for CTR

Signed-off-by: Akhil Goyal <akhil.goyal@nxp.com>

crypto/dpaa2_sec: add per dev mempool to store FLE

rte_malloc uses common memory area for all cores.

Now rte_malloc are replaced by per device mempool to allocate
space for FLE. This removes contention and improves performance.

Signed-off-by: Akhil Goyal <akhil.goyal@nxp.com>

bus/fslmc: add macros to get/set FLE context

Signed-off-by: Akhil Goyal <akhil.goyal@nxp.com>

cryptodev: remove AAD from authentication structure

Now that AAD is only used in AEAD algorithms,
there is no need to keep AAD in the authentication
structure.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: use AES-GCM/CCM as AEAD algorithms

Now that all the structures/functions for AEAD algorithms
are in place, migrate the two supported algorithms
AES-GCM and AES-CCM to these, instead of using
cipher and authentication parameters.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

examples/l2fwd-crypto: add AEAD parameters

Since there is a new operation type (AEAD), add parameters
for this in the application.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

examples/ipsec-secgw: add AEAD parameters

Since there is a new operation type (AEAD), add parameters
for this in the application.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

app/test-crypto-perf: add AEAD parameters

Since there is a new operation type (AEAD), add parameters
for this in the application.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

examples/l2fwd-crypto: avoid too many tabs

Some extra functions have been created to avoid
too many nested conditionals.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: add AEAD parameters in crypto operation

AEAD operation parameters can be set in the new
aead structure, in the crypto operation.
This structure is within a union with the cipher
and authentication parameters, since operations can be:
- AEAD: using the aead structure

- Cipher only: using only the cipher structure

- Auth only: using only the authentication structure

- Cipher-then-auth/Auth-then-cipher: using both cipher
  and authentication structures

Therefore, all three cannot be used at the same time.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: add AEAD specific data

AEAD algorithms such as AES-GCM needed to be
used as a concatenation of a cipher transform and
an authentication transform.

Instead, a new transform and functions to handle it
are created to support these kind of algorithms,
making their use easier.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: set AES-GMAC as auth-only algo

AES-GMAC is an authentication algorithm, based on AES-GCM
without encryption. To simplify its usage, now it can be used
setting the authentication parameters, without requiring
to concatenate a ciphering transform.

Therefore, it is not required to set AAD, but authentication
data length and offset, giving the user the option
to have Scatter-Gather List in the input buffer,
as long as the driver supports it.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: remove digest length from crypto op

Digest length was duplicated in the authentication transform
and the crypto operation structures.

Since digest length is not expected to change in a same
session, it is removed from the crypto operation.

Also, the length has been shrunk to 16 bits,
which should be sufficient for any digest.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: remove AAD length from crypto op

Additional authenticated data (AAD) information was duplicated
in the authentication transform and in the crypto
operation structures.

Since AAD length is not meant to be changed in a same session,
it is removed from the crypto operation structure.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: do not use AAD in wireless algorithms

For wireless algorithms (SNOW3G, KASUMI, ZUC),
the IV for the authentication algorithms (F9, UIA2 and EIA3)
was taken from the AAD parameter, as there was no IV parameter
in the authentication structure.

Now that IV is available for all algorithms, there is need
to keep doing this, so AAD is not used for these algorithms
anymore.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: add auth IV

Authentication algorithms, such as AES-GMAC or the wireless
algorithms (like SNOW3G) use IV, like cipher algorithms.
So far, AES-GMAC has used the IV from the cipher structure,
and the wireless algorithms have used the AAD field,
which is not technically correct.

Therefore, authentication IV parameters have been added,
so API is more correct. Like cipher IV, auth IV is expected
to be copied after the crypto operation.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: move IV parameters to session

Since IV parameters (offset and length) should not
change for operations in the same session, these parameters
are moved to the crypto transform structure, so they will
be stored in the sessions.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: pass IV as offset

Since IV now is copied after the crypto operation, in
its private size, IV can be passed only with offset
and length.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

examples/ipsec-secgw: move IV to crypto op private data

Usually, IV will change for each crypto operation.
Therefore, instead of pointing at the same location,
IV is copied after each crypto operation.

This will let the IV to be passed as an offset from
the beginning of the crypto operation, instead of
a pointer.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

examples/l2fwd-crypto: move IV to crypto op private data

Usually, IV will change for each crypto operation.
Therefore, instead of pointing at the same location,
IV is copied after each crypto operation.

This will let the IV to be passed as an offset from
the beginning of the crypto operation, instead of
a pointer.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

app/crypto-perf: move IV to crypto op private data

Usually, IV will change for each crypto operation.
Therefore, instead of pointing at the same location,
IV is copied after each crypto operation.

This will let the IV to be passed as an offset from
the beginning of the crypto operation, instead of
a pointer.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

test/crypto-perf: move IV to crypto op private data

Usually, IV will change for each crypto operation.
Therefore, instead of pointing at the same location,
IV is copied after each crypto operation.

This will let the IV to be passed as an offset from
the beginning of the crypto operation, instead of
a pointer.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

test/crypto: move IV to crypto op private data

Usually, IV will change for each crypto operation.
Therefore, instead of pointing at the same location,
IV is copied after each crypto operation.

This will let the IV to be passed as an offset from
the beginning of the crypto operation, instead of
a pointer.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: add crypto op helper macros

In order to facilitate the access to the private data,
after the crypto operation, two new macros have been
implemented:

- rte_crypto_op_ctod_offset(c,t,o), which returns a pointer
  to "o" bytes after the start of the crypto operation
  (rte_crypto_op)
- rte_crypto_op_ctophys_offset(c, o), which returns
  the physical address of the data "o" bytes after the
  start of the crypto operation (rte_crypto_op)

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: remove useless alignment

rte_crypto_op and rte_crypto_sym_op structures
were marked as cache aligned.
However, since these structures are always initialized
in a mempool, this alignment is useless, since the mempool
forces the alignment of its objects.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: do not store pointer to op specific params

Instead of storing a pointer to operation specific parameters,
such as symmetric crypto parameters, use a zero-length array,
to mark that these parameters will be stored after the
generic crypto operation structure, which was already assumed
in the code, reducing the memory footprint of the crypto operation.

Besides, it is always expected to have rte_crypto_op
and rte_crypto_sym_op (the only operation specific parameters
structure right now) to be together, as they are initialized
as a single object in the crypto operation pool.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: remove opaque data pointer in crypto op

Storing a pointer to the user data is unnecessary,
since user can store additional data, after the crypto operation.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: replace enums with 1-byte variables

Instead of storing some crypto operation flags,
such as operation status, as enumerations,
store them as uint8_t, for memory efficiency.

Also, reserve extra 5 bytes in the crypto operation,
for future additions.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

cryptodev: move session type to generic crypto op

Session type (operation with or without session) is not
something specific to symmetric operations.
Therefore, the variable is moved to the generic crypto operation
structure.

Since this is an ABI change, the cryptodev library version
gets bumped.

Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Declan Doherty <declan.doherty@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

app/crypto-perf: set crypto op pool cache

Signed-off-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

app/crypto-perf: return crypto ops to mempool in bulk

Use rte_mempool_put_bulk for both latency and throughput tests instead
of rte_crypto_op_free to improve application performance.

Signed-off-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

app/crypto-perf: fix digest data for chained mbufs

Use corect mbuf segment for chained mbufs.

Fixes: f8be1786b1b8 ("app/crypto-perf: introduce performance test application")
Cc: stable@dpdk.org
Signed-off-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Acked-by: Fiona Trahe <fiona.trahe@intel.com>

crypto/qat: fix typo

Fixed a comment in QAT, referring to the IV size
for AES-GCM, that should be in bytes, and not bits.

Fixes: 53d8971cbe81 ("qat: fix AES-GCM decryption")
Cc: stable@dpdk.org
Signed-off-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Deepak Kumar Jain <deepak.k.jain@intel.com>

vhost: print reason of NUMA node query failure

syscall always returns '-1' on failure and there is no point
in printing that value. 'errno' is much more informative.

Fixes: 586e39001317 ("vhost: export numa node")

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

net/virtio-user: fix comment

Since "rte_eal_dev_init()" has been removed, the comment referred to
it should be modified simultaneously.

Fixes: 9721b4d543a3 ("eal: remove unused device init function")

Signed-off-by: Yong Wang <wang.yong19@zte.com.cn>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

net/virtio-user: fix crash when detaching device

The rte_eth_dev.data pointer is set to a reference to a static table.
Attempting to rte_free() it leads to a panic.  For example, the
following commands result in a panic if run in testpmd

  testpmd> port attach virtio_user0,path=/dev/vhost-net,iface=test0
  testpmd> port stop 2
  testpmd> port close 2
  testpmd> port detach 2

Fixes: ce2eabdd43ec ("net/virtio-user: add virtual device")
Cc: stable@dpdk.org
Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: check return of mutex initialization

Check return value of pthread_mutex_init(). Also destroy
mutex in case of other erros before returning.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: check return of pthread calls

Make sure we catch and log failed calls to pthread
functions.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: add missing check in driver registration

Add a check for strdup() return value and fail gracefully if we
get a bad return code.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: fix MTU device feature check

The MTU feature support check has to be done against MTU
feature bit mask, and not bit position.

Fixes: 72e8543093df ("vhost: add API to get MTU value")
Cc: stable@dpdk.org
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: fix checking of device features

To compare enabled features in current device we must use bit
mask instead of bit position.

Fixes: c843af3aa13e ("vhost: access header only if offloading is supported")
Cc: stable@dpdk.org
Signed-off-by: Ivan Dyukov <i.dyukov@samsung.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

doc: update virtio ring size and header size

Add more explanations about vring size changes and different
virtio_header size.

Signed-off-by: Zhiyong Yang <zhiyong.yang@intel.com>
Acked-by: John McNamara <john.mcnamara@intel.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: fix IP checksum

There is no way to bypass IP checksum verification in Linux
kernel, no matter skb->ip_summed is assigned as CHECKSUM_UNNECESSARY
or CHECKSUM_PARTIAL.

So any packets with bad IP checksum will be dropped at VM IP layer.

To correct, we check this flag PKT_TX_IP_CKSUM to calculate IP csum.

Fixes: 859b480d5afd ("vhost: add guest offload setting")
Cc: stable@dpdk.org
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: fix TCP checksum

As PKT_TX_TCP_SEG flag in mbuf->ol_flags implies PKT_TX_TCP_CKSUM,
applications, e.g., testpmd, don't set PKT_TX_TCP_CKSUM when TSO
is set.

This leads to that packets get dropped in VM tcp stack layer because
of bad TCP csum.

To fix this, we make sure TCP NEEDS_CSUM info is set into virtio net
header when PKT_TX_TCP_SEG is set, so that VM tcp stack will not
check the TCP csum.

Fixes: 859b480d5afd ("vhost: add guest offload setting")
Cc: stable@dpdk.org
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

vhost: clean up per-socket mutex

vsocket->conn_mutex was allocated with pthread_mutex_init() but never
freed with pthread_mutex_destroy().  This is a potential memory leak,
depending on how pthread_mutex_t is implemented.

Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

net/mlx5: add vectorized Rx/Tx burst for x86

To make vectorized burst routines enabled, it is required to run on x86_64
architecture. If all the conditions are met, the vectorized burst functions
are enabled automatically. The decision is made individually on RX and TX.
There's no PMD option to make a selection.

Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>