+Synchronous mode
+----------------
+
+Some cryptodevs support synchronous mode alongside with a standard asynchronous
+mode. In that case operations are performed directly when calling
+``rte_cryptodev_sym_cpu_crypto_process`` method instead of enqueuing and
+dequeuing an operation before. This mode of operation allows cryptodevs which
+utilize CPU cryptographic acceleration to have significant performance boost
+comparing to standard asynchronous approach. Cryptodevs supporting synchronous
+mode have ``RTE_CRYPTODEV_FF_SYM_CPU_CRYPTO`` feature flag set.
+
+To perform a synchronous operation a call to
+``rte_cryptodev_sym_cpu_crypto_process`` has to be made with vectorized
+operation descriptor (``struct rte_crypto_sym_vec``) containing:
+
+- ``num`` - number of operations to perform,
+- pointer to an array of size ``num`` containing a scatter-gather list
+ descriptors of performed operations (``struct rte_crypto_sgl``). Each instance
+ of ``struct rte_crypto_sgl`` consists of a number of segments and a pointer to
+ an array of segment descriptors ``struct rte_crypto_vec``;
+- pointers to arrays of size ``num`` containing IV, AAD and digest information
+ in the ``cpu_crypto`` sub-structure,
+- pointer to an array of size ``num`` where status information will be stored
+ for each operation.
+
+Function returns a number of successfully completed operations and sets
+appropriate status number for each operation in the status array provided as
+a call argument. Status different than zero must be treated as error.
+
+For more details, e.g. how to convert an mbuf to an SGL, please refer to an
+example usage in the IPsec library implementation.
+
+Cryptodev Raw Data-path APIs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Crypto Raw data-path APIs are a set of APIs designed to enable external
+libraries/applications to leverage the cryptographic processing provided by
+DPDK crypto PMDs through the cryptodev API but in a manner that is not
+dependent on native DPDK data structures (eg. rte_mbuf, rte_crypto_op, ... etc)
+in their data-path implementation.
+
+The raw data-path APIs have the following advantages:
+
+- External data structure friendly design. The new APIs uses the operation
+ descriptor ``struct rte_crypto_sym_vec`` that supports raw data pointer and
+ IOVA addresses as input. Moreover, the APIs does not require the user to
+ allocate the descriptor from mempool, nor requiring mbufs to describe input
+ data's virtual and IOVA addresses. All these features made the translation
+ from user's own data structure into the descriptor easier and more efficient.
+
+- Flexible enqueue and dequeue operation. The raw data-path APIs gives the
+ user more control to the enqueue and dequeue operations, including the
+ capability of precious enqueue/dequeue count, abandoning enqueue or dequeue
+ at any time, and operation status translation and set on the fly.
+
+Cryptodev PMDs which support the raw data-path APIs will have
+``RTE_CRYPTODEV_FF_SYM_RAW_DP`` feature flag presented. To use this feature,
+the user shall create a local ``struct rte_crypto_raw_dp_ctx`` buffer and
+extend to at least the length returned by ``rte_cryptodev_get_raw_dp_ctx_size``
+function call. The created buffer is then initialized using
+``rte_cryptodev_configure_raw_dp_ctx`` function with the ``is_update``
+parameter as 0. The library and the crypto device driver will then set the
+buffer and attach either the cryptodev sym session, the rte_security session,
+or the cryptodev xform for session-less operation into the ctx buffer, and
+set the corresponding enqueue and dequeue function handlers based on the
+algorithm information stored in the session or xform. When the ``is_update``
+parameter passed into ``rte_cryptodev_configure_raw_dp_ctx`` is 1, the driver
+will not initialize the buffer but only update the session or xform and
+the function handlers accordingly.
+
+After the ``struct rte_crypto_raw_dp_ctx`` buffer is initialized, it is now
+ready for enqueue and dequeue operation. There are two different enqueue
+functions: ``rte_cryptodev_raw_enqueue`` to enqueue single raw data
+operation, and ``rte_cryptodev_raw_enqueue_burst`` to enqueue a descriptor
+with multiple operations. In case of the application uses similar approach to
+``struct rte_crypto_sym_vec`` to manage its data burst but with different
+data structure, using the ``rte_cryptodev_raw_enqueue_burst`` function may be
+less efficient as this is a situation where the application has to loop over
+all crypto operations to assemble the ``struct rte_crypto_sym_vec`` descriptor
+from its own data structure, and then the driver will loop over them again to
+translate every operation in the descriptor to the driver's specific queue data.
+The ``rte_cryptodev_raw_enqueue`` should be used to save one loop for each data
+burst instead.
+
+The ``rte_cryptodev_raw_enqueue`` and ``rte_cryptodev_raw_enqueue_burst``
+functions will return or set the enqueue status. ``rte_cryptodev_raw_enqueue``
+will return the status directly, ``rte_cryptodev_raw_enqueue_burst`` will
+return the number of operations enqueued or stored (explained as follows) and
+set the ``enqueue_status`` buffer provided by the user. The possible
+enqueue status values are:
+
+- ``1``: the operation(s) is/are enqueued successfully.
+- ``0``: the operation(s) is/are cached successfully in the crypto device queue
+ but is not actually enqueued. The user shall call
+ ``rte_cryptodev_raw_enqueue_done`` function after the expected operations
+ are stored. The crypto device will then start enqueuing all of them at
+ once.
+- The negative integer: error occurred during enqueue.
+
+Calling ``rte_cryptodev_configure_raw_dp_ctx`` with the parameter ``is_update``
+set as 0 twice without the enqueue function returning or setting enqueue status
+to 1 or ``rte_cryptodev_raw_enqueue_done`` function being called in between will
+invalidate any operation stored in the device queue but not enqueued. This
+feature is useful when the user wants to abandon partially enqueued operations
+for a failed enqueue burst operation and try enqueuing in a whole later.
+
+Similar as enqueue, there are two dequeue functions:
+``rte_cryptodev_raw_dequeue`` for dequeing single operation, and
+``rte_cryptodev_raw_dequeue_burst`` for dequeuing a burst of operations (e.g.
+all operations in a ``struct rte_crypto_sym_vec`` descriptor). The
+``rte_cryptodev_raw_dequeue_burst`` function allows the user to provide callback
+functions to retrieve dequeue count from the enqueued user data and write the
+expected status value to the user data on the fly. The dequeue functions also
+set the dequeue status:
+
+- ``1``: the operation(s) is/are dequeued successfully.
+- ``0``: the operation(s) is/are completed but is not actually dequeued (hence
+ still kept in the device queue). The user shall call the
+ ``rte_cryptodev_raw_dequeue_done`` function after the expected number of
+ operations (e.g. all operations in a descriptor) are dequeued. The crypto
+ device driver will then free them from the queue at once.
+- The negative integer: error occurred during dequeue.
+
+Calling ``rte_cryptodev_configure_raw_dp_ctx`` with the parameter ``is_update``
+set as 0 twice without the dequeue functions execution changed dequeue_status
+to 1 or ``rte_cryptodev_raw_dequeue_done`` function being called in between will
+revert the crypto device queue's dequeue effort to the moment when the
+``struct rte_crypto_raw_dp_ctx`` buffer is initialized. This feature is useful
+when the user wants to abandon partially dequeued data and try dequeuing again
+later in a whole.
+
+There are a few limitations to the raw data path APIs:
+
+* Only support in-place operations.
+* APIs are NOT thread-safe.
+* CANNOT mix the raw data-path API's enqueue with rte_cryptodev_enqueue_burst,
+ or vice versa.
+
+See *DPDK API Reference* for details on each API definitions.
+
+Sample code
+-----------
+
+There are various sample applications that show how to use the cryptodev library,
+such as the L2fwd with Crypto sample application (L2fwd-crypto) and
+the IPsec Security Gateway application (ipsec-secgw).
+
+While these applications demonstrate how an application can be created to perform
+generic crypto operation, the required complexity hides the basic steps of
+how to use the cryptodev APIs.
+
+The following sample code shows the basic steps to encrypt several buffers
+with AES-CBC (although performing other crypto operations is similar),
+using one of the crypto PMDs available in DPDK.
+
+.. code-block:: c
+
+ /*
+ * Simple example to encrypt several buffers with AES-CBC using
+ * the Cryptodev APIs.
+ */
+
+ #define MAX_SESSIONS 1024
+ #define NUM_MBUFS 1024
+ #define POOL_CACHE_SIZE 128
+ #define BURST_SIZE 32
+ #define BUFFER_SIZE 1024
+ #define AES_CBC_IV_LENGTH 16
+ #define AES_CBC_KEY_LENGTH 16
+ #define IV_OFFSET (sizeof(struct rte_crypto_op) + \
+ sizeof(struct rte_crypto_sym_op))
+
+ struct rte_mempool *mbuf_pool, *crypto_op_pool;
+ struct rte_mempool *session_pool, *session_priv_pool;
+ unsigned int session_size;
+ int ret;
+
+ /* Initialize EAL. */
+ ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
+
+ uint8_t socket_id = rte_socket_id();
+
+ /* Create the mbuf pool. */
+ mbuf_pool = rte_pktmbuf_pool_create("mbuf_pool",
+ NUM_MBUFS,
+ POOL_CACHE_SIZE,
+ 0,
+ RTE_MBUF_DEFAULT_BUF_SIZE,
+ socket_id);
+ if (mbuf_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+
+ /*
+ * The IV is always placed after the crypto operation,
+ * so some private data is required to be reserved.
+ */
+ unsigned int crypto_op_private_data = AES_CBC_IV_LENGTH;
+
+ /* Create crypto operation pool. */
+ crypto_op_pool = rte_crypto_op_pool_create("crypto_op_pool",
+ RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+ NUM_MBUFS,
+ POOL_CACHE_SIZE,
+ crypto_op_private_data,
+ socket_id);
+ if (crypto_op_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create crypto op pool\n");
+
+ /* Create the virtual crypto device. */
+ char args[128];
+ const char *crypto_name = "crypto_aesni_mb0";
+ snprintf(args, sizeof(args), "socket_id=%d", socket_id);
+ ret = rte_vdev_init(crypto_name, args);
+ if (ret != 0)
+ rte_exit(EXIT_FAILURE, "Cannot create virtual device");
+
+ uint8_t cdev_id = rte_cryptodev_get_dev_id(crypto_name);
+
+ /* Get private session data size. */
+ session_size = rte_cryptodev_sym_get_private_session_size(cdev_id);
+
+ #ifdef USE_TWO_MEMPOOLS
+ /* Create session mempool for the session header. */
+ session_pool = rte_cryptodev_sym_session_pool_create("session_pool",
+ MAX_SESSIONS,
+ 0,
+ POOL_CACHE_SIZE,
+ 0,
+ socket_id);
+
+ /*
+ * Create session private data mempool for the
+ * private session data for the crypto device.
+ */
+ session_priv_pool = rte_mempool_create("session_pool",
+ MAX_SESSIONS,
+ session_size,
+ POOL_CACHE_SIZE,
+ 0, NULL, NULL, NULL,
+ NULL, socket_id,
+ 0);
+
+ #else
+ /* Use of the same mempool for session header and private data */
+ session_pool = rte_cryptodev_sym_session_pool_create("session_pool",
+ MAX_SESSIONS * 2,
+ session_size,
+ POOL_CACHE_SIZE,
+ 0,
+ socket_id);
+
+ session_priv_pool = session_pool;
+
+ #endif
+
+ /* Configure the crypto device. */
+ struct rte_cryptodev_config conf = {
+ .nb_queue_pairs = 1,
+ .socket_id = socket_id
+ };
+
+ struct rte_cryptodev_qp_conf qp_conf = {
+ .nb_descriptors = 2048,
+ .mp_session = session_pool,
+ .mp_session_private = session_priv_pool
+ };
+
+ if (rte_cryptodev_configure(cdev_id, &conf) < 0)
+ rte_exit(EXIT_FAILURE, "Failed to configure cryptodev %u", cdev_id);
+
+ if (rte_cryptodev_queue_pair_setup(cdev_id, 0, &qp_conf, socket_id) < 0)
+ rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n");
+
+ if (rte_cryptodev_start(cdev_id) < 0)
+ rte_exit(EXIT_FAILURE, "Failed to start device\n");
+
+ /* Create the crypto transform. */
+ uint8_t cipher_key[16] = {0};
+ struct rte_crypto_sym_xform cipher_xform = {
+ .next = NULL,
+ .type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+ .cipher = {
+ .op = RTE_CRYPTO_CIPHER_OP_ENCRYPT,
+ .algo = RTE_CRYPTO_CIPHER_AES_CBC,
+ .key = {
+ .data = cipher_key,
+ .length = AES_CBC_KEY_LENGTH
+ },
+ .iv = {
+ .offset = IV_OFFSET,
+ .length = AES_CBC_IV_LENGTH
+ }
+ }
+ };
+
+ /* Create crypto session and initialize it for the crypto device. */
+ struct rte_cryptodev_sym_session *session;
+ session = rte_cryptodev_sym_session_create(session_pool);
+ if (session == NULL)
+ rte_exit(EXIT_FAILURE, "Session could not be created\n");
+
+ if (rte_cryptodev_sym_session_init(cdev_id, session,
+ &cipher_xform, session_priv_pool) < 0)
+ rte_exit(EXIT_FAILURE, "Session could not be initialized "
+ "for the crypto device\n");
+
+ /* Get a burst of crypto operations. */
+ struct rte_crypto_op *crypto_ops[BURST_SIZE];
+ if (rte_crypto_op_bulk_alloc(crypto_op_pool,
+ RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+ crypto_ops, BURST_SIZE) == 0)
+ rte_exit(EXIT_FAILURE, "Not enough crypto operations available\n");
+
+ /* Get a burst of mbufs. */
+ struct rte_mbuf *mbufs[BURST_SIZE];
+ if (rte_pktmbuf_alloc_bulk(mbuf_pool, mbufs, BURST_SIZE) < 0)
+ rte_exit(EXIT_FAILURE, "Not enough mbufs available");
+
+ /* Initialize the mbufs and append them to the crypto operations. */
+ unsigned int i;
+ for (i = 0; i < BURST_SIZE; i++) {
+ if (rte_pktmbuf_append(mbufs[i], BUFFER_SIZE) == NULL)
+ rte_exit(EXIT_FAILURE, "Not enough room in the mbuf\n");
+ crypto_ops[i]->sym->m_src = mbufs[i];
+ }
+
+ /* Set up the crypto operations. */
+ for (i = 0; i < BURST_SIZE; i++) {
+ struct rte_crypto_op *op = crypto_ops[i];
+ /* Modify bytes of the IV at the end of the crypto operation */
+ uint8_t *iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *,
+ IV_OFFSET);
+
+ generate_random_bytes(iv_ptr, AES_CBC_IV_LENGTH);
+
+ op->sym->cipher.data.offset = 0;
+ op->sym->cipher.data.length = BUFFER_SIZE;
+
+ /* Attach the crypto session to the operation */
+ rte_crypto_op_attach_sym_session(op, session);
+ }
+
+ /* Enqueue the crypto operations in the crypto device. */
+ uint16_t num_enqueued_ops = rte_cryptodev_enqueue_burst(cdev_id, 0,
+ crypto_ops, BURST_SIZE);
+
+ /*
+ * Dequeue the crypto operations until all the operations
+ * are processed in the crypto device.
+ */
+ uint16_t num_dequeued_ops, total_num_dequeued_ops = 0;
+ do {
+ struct rte_crypto_op *dequeued_ops[BURST_SIZE];
+ num_dequeued_ops = rte_cryptodev_dequeue_burst(cdev_id, 0,
+ dequeued_ops, BURST_SIZE);
+ total_num_dequeued_ops += num_dequeued_ops;
+
+ /* Check if operation was processed successfully */
+ for (i = 0; i < num_dequeued_ops; i++) {
+ if (dequeued_ops[i]->status != RTE_CRYPTO_OP_STATUS_SUCCESS)
+ rte_exit(EXIT_FAILURE,
+ "Some operations were not processed correctly");
+ }
+
+ rte_mempool_put_bulk(crypto_op_pool, (void **)dequeued_ops,
+ num_dequeued_ops);
+ } while (total_num_dequeued_ops < num_enqueued_ops);