From f546c1ed0bc8b2e7766e1f7d961b80cb05ae1713 Mon Sep 17 00:00:00 2001 From: Fiona Trahe Date: Mon, 3 Apr 2017 15:13:22 +0100 Subject: [PATCH] doc: restructure QAT guide Restructure QAT PMD instructions and add a device table to minimise duplication for each device and make it easier to add devices. Fix some device name typos and poor formatting. Signed-off-by: Fiona Trahe Acked-by: John McNamara --- doc/guides/cryptodevs/qat.rst | 416 +++++++++++++++------------------- 1 file changed, 179 insertions(+), 237 deletions(-) diff --git a/doc/guides/cryptodevs/qat.rst b/doc/guides/cryptodevs/qat.rst index cecaee029a..0c4730d90d 100644 --- a/doc/guides/cryptodevs/qat.rst +++ b/doc/guides/cryptodevs/qat.rst @@ -30,9 +30,12 @@ Intel(R) QuickAssist (QAT) Crypto Poll Mode Driver ================================================== -The QAT PMD provides poll mode crypto driver support for **Intel QuickAssist -Technology DH895xxC**, **Intel QuickAssist Technology C62x** and -**Intel QuickAssist Technology C3xxx** hardware accelerator. +The QAT PMD provides poll mode crypto driver support for the following +hardware accelerator devices: + +* ``Intel QuickAssist Technology DH895xCC`` +* ``Intel QuickAssist Technology C62x`` +* ``Intel QuickAssist Technology C3xxx`` Features @@ -89,332 +92,271 @@ Limitations Installation ------------ -To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The -VF devices exposed by this driver will be used by QAT PMD. - -To enable QAT in DPDK, follow the instructions mentioned in -http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html +To enable QAT in DPDK, follow the instructions for modifying the compile-time +configuration file as described `here `_. -Quick instructions as follows: +Quick instructions are as follows: .. code-block:: console + cd to the top-level DPDK directory make config T=x86_64-native-linuxapp-gcc sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_QAT\)=n,\1=y,' build/.config make -If you are running on kernel 4.4 or greater, see instructions for -`Installation using kernel.org driver`_ below. If you are on a kernel earlier -than 4.4, see `Installation using 01.org QAT driver`_. - -For **Intel QuickAssist Technology C62x** and **Intel QuickAssist Technology C3xxx** -device, kernel 4.5 or greater is needed. -See instructions for `Installation using kernel.org driver`_ below. - - -Installation using 01.org QAT driver ------------------------------------- - -NOTE: There is no driver available for **Intel QuickAssist Technology C62x** and -**Intel QuickAssist Technology C3xxx** devices on 01.org. - -Download the latest QuickAssist Technology Driver from `01.org -`_ -Consult the *Getting Started Guide* at the same URL for further information. +To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The VF +devices exposed by this driver will be used by the QAT PMD. The devices and +available kernel drivers and device ids are : -The steps below assume you are: +.. _table_qat_pmds_drivers: -* Building on a platform with one ``DH895xCC`` device. -* Using package ``qatmux.l.2.3.0-34.tgz``. -* On Fedora21 kernel ``3.17.4-301.fc21.x86_64``. +.. table:: QAT devices and drivers -In the BIOS ensure that SRIOV is enabled and VT-d is disabled. + +----------+--------+---------------+------------+--------+---------+--------+------------+ + | Device | Driver | Kernel Module | Pci Driver | PF Did | Num PFs | Vf Did | VFs per PF | + +==========+========+===============+============+========+=========+========+============+ + | DH895xCC | 01.org | icp_qa_al | n/a | 435 | 1 | 443 | 32 | + +----------+--------+---------------+------------+--------+---------+--------+------------+ + | DH895xCC | 4.4+ | qat_dh895xcc | dh895xcc | 435 | 1 | 443 | 32 | + +----------+--------+---------------+------------+--------+---------+--------+------------+ + | C62x | 4.5+ | qat_c62x | c6xx | 37c8 | 3 | 37c9 | 16 | + +----------+--------+---------------+------------+--------+---------+--------+------------+ + | C3xxx | 4.5+ | qat_c3xxx | c3xxx | 19e2 | 1 | 19e3 | 16 | + +----------+--------+---------------+------------+--------+---------+--------+------------+ -Uninstall any existing QAT driver, for example by running: -* ``./installer.sh uninstall`` in the directory where originally installed. +The ``Driver`` column indicates either the Linux kernel version in which +support for this device was introduced or a driver available on Intel's 01.org +website. There are both linux and 01.org kernel drivers available for some +devices. -* or ``rmmod qat_dh895xcc; rmmod intel_qat``. - -Build and install the SRIOV-enabled QAT driver:: - - mkdir /QAT - cd /QAT - # copy qatmux.l.2.3.0-34.tgz to this location - tar zxof qatmux.l.2.3.0-34.tgz - - export ICP_WITHOUT_IOMMU=1 - ./installer.sh install QAT1.6 host - -You can use ``cat /proc/icp_dh895xcc_dev0/version`` to confirm the driver is correctly installed. -You can use ``lspci -d:443`` to confirm the bdf of the 32 VF devices are available per ``DH895xCC`` device. - -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. - -**Note**: If using a later kernel and the build fails with an error relating to ``strict_stroul`` not being available apply the following patch: - -.. code-block:: diff - - /QAT/QAT1.6/quickassist/utilities/downloader/Target_CoreLibs/uclo/include/linux/uclo_platform.h - + #if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,5) - + #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (kstrtoul((str), (base), (num))) printk("Error strtoull convert %s\n", str); } - + #else - #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,38) - #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (strict_strtoull((str), (base), (num))) printk("Error strtoull convert %s\n", str); } - #else - #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25) - #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; strict_strtoll((str), (base), (num));} - #else - #define STR_TO_64(str, base, num, endPtr) \ - do { \ - if (str[0] == '-') \ - { \ - *(num) = -(simple_strtoull((str+1), &(endPtr), (base))); \ - }else { \ - *(num) = simple_strtoull((str), &(endPtr), (base)); \ - } \ - } while(0) - + #endif - #endif - #endif - - -If the build fails due to missing header files you may need to do following: - -* ``sudo yum install zlib-devel`` -* ``sudo yum install openssl-devel`` - -If the build or install fails due to mismatching kernel sources you may need to do the following: - -* ``sudo yum install kernel-headers-`uname -r``` -* ``sudo yum install kernel-src-`uname -r``` -* ``sudo yum install kernel-devel-`uname -r``` +If you are running on a kernel which includes a driver for your device, see +`Installation using kernel.org driver`_ below. Otherwise see +`Installation using 01.org QAT driver`_. Installation using kernel.org driver ------------------------------------ -For **Intel QuickAssist Technology DH895xxC**: - -Assuming you are running on at least a 4.4 kernel, you can use the stock kernel.org QAT -driver to start the QAT hardware. - -The steps below assume you are: +The examples below are based on the C62x device, if you have a different device +use the corresponding values in the above table. -* Running DPDK on a platform with one ``DH895xCC`` device. -* On a kernel at least version 4.4. +In BIOS ensure that SRIOV is enabled and either: -In BIOS ensure that SRIOV is enabled and either -a) disable VT-d or -b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. +* Disable VT-d or +* Enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. -Ensure the QAT driver is loaded on your system, by executing:: +Check that the QAT driver is loaded on your system, by executing:: - lsmod | grep qat + lsmod | grep qa -You should see the following output:: +You should see the kernel module for your device listed, e.g.:: - qat_dh895xcc 5626 0 - intel_qat 82336 1 qat_dh895xcc + qat_c62x 5626 0 + intel_qat 82336 1 qat_c62x Next, you need to expose the Virtual Functions (VFs) using the sysfs file system. -First find the bdf of the physical function (PF) of the DH895xCC device:: +First find the BDFs (Bus-Device-Function) of the physical functions (PFs) of +your device, e.g.:: - lspci -d : 435 + lspci -d : 37c8 You should see output similar to:: - 03:00.0 Co-processor: Intel Corporation Coleto Creek PCIe Endpoint - -Using the sysfs, enable the VFs:: - - echo 32 > /sys/bus/pci/drivers/dh895xcc/0000\:03\:00.0/sriov_numvfs - -If you get an error, it's likely you're using a QAT kernel driver earlier than kernel 4.4. - -To verify that the VFs are available for use - use ``lspci -d:443`` to confirm -the bdf of the 32 VF devices are available per ``DH895xCC`` device. + 1a:00.0 Co-processor: Intel Corporation Device 37c8 + 3d:00.0 Co-processor: Intel Corporation Device 37c8 + 3f:00.0 Co-processor: Intel Corporation Device 37c8 -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. +Enable the VFs for each PF by echoing the number of VFs per PF to the pci driver:: -**Note**: If the QAT kernel modules are not loaded and you see an error like - ``Failed to load MMP firmware qat_895xcc_mmp.bin`` this may be as a - result of not using a distribution, but just updating the kernel directly. + echo 16 > /sys/bus/pci/drivers/c6xx/0000:1a:00.0/sriov_numvfs + echo 16 > /sys/bus/pci/drivers/c6xx/0000:3d:00.0/sriov_numvfs + echo 16 > /sys/bus/pci/drivers/c6xx/0000:3f:00.0/sriov_numvfs -Download firmware from the kernel firmware repo at: -http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/ +Check that the VFs are available for use. For example ``lspci -d:37c9`` should +list 48 VF devices available for a ``C62x`` device. -Copy qat binaries to /lib/firmware: -* ``cp qat_895xcc.bin /lib/firmware`` -* ``cp qat_895xcc_mmp.bin /lib/firmware`` +To complete the installation follow the instructions in +`Binding the available VFs to the DPDK UIO driver`_. -cd to your linux source root directory and start the qat kernel modules: -* ``insmod ./drivers/crypto/qat/qat_common/intel_qat.ko`` -* ``insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko`` +.. Note:: -**Note**:The following warning in /var/log/messages can be ignored: - ``IOMMU should be enabled for SR-IOV to work correctly`` + If the QAT kernel modules are not loaded and you see an error like ``Failed + to load MMP firmware qat_895xcc_mmp.bin`` in kernel logs, this may be as a + result of not using a distribution, but just updating the kernel directly. -For **Intel QuickAssist Technology C62x**: -Assuming you are running on at least a 4.5 kernel, you can use the stock kernel.org QAT -driver to start the QAT hardware. + Download firmware from the `kernel firmware repo + `_. -The steps below assume you are: + Copy qat binaries to ``/lib/firmware``:: -* Running DPDK on a platform with one ``C62x`` device. -* On a kernel at least version 4.5. + cp qat_895xcc.bin /lib/firmware + cp qat_895xcc_mmp.bin /lib/firmware -In BIOS ensure that SRIOV is enabled and either -a) disable VT-d or -b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. + Change to your linux source root directory and start the qat kernel modules:: -Ensure the QAT driver is loaded on your system, by executing:: + insmod ./drivers/crypto/qat/qat_common/intel_qat.ko + insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko - lsmod | grep qat -You should see the following output:: +.. Note:: - qat_c62x 16384 0 - intel_qat 122880 1 qat_c62x + If you see the following warning in ``/var/log/messages`` it can be ignored: + ``IOMMU should be enabled for SR-IOV to work correctly``. -Next, you need to expose the VFs using the sysfs file system. -First find the bdf of the C62x device:: +Installation using 01.org QAT driver +------------------------------------ - lspci -d:37c8 +Download the latest QuickAssist Technology Driver from `01.org +`_. +Consult the *Getting Started Guide* at the same URL for further information. -You should see output similar to:: +The steps below assume you are: - 1a:00.0 Co-processor: Intel Corporation Device 37c8 - 3d:00.0 Co-processor: Intel Corporation Device 37c8 - 3f:00.0 Co-processor: Intel Corporation Device 37c8 +* Building on a platform with one ``DH895xCC`` device. +* Using package ``qatmux.l.2.3.0-34.tgz``. +* On Fedora21 kernel ``3.17.4-301.fc21.x86_64``. -For each c62x device there are 3 PFs. -Using the sysfs, for each PF, enable the 16 VFs:: +In the BIOS ensure that SRIOV is enabled and VT-d is disabled. - echo 16 > /sys/bus/pci/drivers/c6xx/0000\:1a\:00.0/sriov_numvfs +Uninstall any existing QAT driver, for example by running: -If you get an error, it's likely you're using a QAT kernel driver earlier than kernel 4.5. +* ``./installer.sh uninstall`` in the directory where originally installed. -To verify that the VFs are available for use - use ``lspci -d:37c9`` to confirm -the bdf of the 48 VF devices are available per ``C62x`` device. +* or ``rmmod qat_dh895xcc; rmmod intel_qat``. -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. +Build and install the SRIOV-enabled QAT driver:: -For **Intel QuickAssist Technology C3xxx**: -Assuming you are running on at least a 4.5 kernel, you can use the stock kernel.org QAT -driver to start the QAT hardware. + mkdir /QAT + cd /QAT -The steps below assume you are: + # Copy qatmux.l.2.3.0-34.tgz to this location + tar zxof qatmux.l.2.3.0-34.tgz -* Running DPDK on a platform with one ``C3xxx`` device. -* On a kernel at least version 4.5. + export ICP_WITHOUT_IOMMU=1 + ./installer.sh install QAT1.6 host -In BIOS ensure that SRIOV is enabled and either -a) disable VT-d or -b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. +You can use ``cat /proc/icp_dh895xcc_dev0/version`` to confirm the driver is correctly installed. +You can use ``lspci -d:443`` to confirm the of the 32 VF devices available per ``DH895xCC`` device. -Ensure the QAT driver is loaded on your system, by executing:: +To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. - lsmod | grep qat +.. Note:: -You should see the following output:: + If using a later kernel and the build fails with an error relating to + ``strict_stroul`` not being available apply the following patch: - qat_c3xxx 16384 0 - intel_qat 122880 1 qat_c3xxx + .. code-block:: diff -Next, you need to expose the Virtual Functions (VFs) using the sysfs file system. + /QAT/QAT1.6/quickassist/utilities/downloader/Target_CoreLibs/uclo/include/linux/uclo_platform.h + + #if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,5) + + #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (kstrtoul((str), (base), (num))) printk("Error strtoull convert %s\n", str); } + + #else + #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,38) + #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (strict_strtoull((str), (base), (num))) printk("Error strtoull convert %s\n", str); } + #else + #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25) + #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; strict_strtoll((str), (base), (num));} + #else + #define STR_TO_64(str, base, num, endPtr) \ + do { \ + if (str[0] == '-') \ + { \ + *(num) = -(simple_strtoull((str+1), &(endPtr), (base))); \ + }else { \ + *(num) = simple_strtoull((str), &(endPtr), (base)); \ + } \ + } while(0) + + #endif + #endif + #endif -First find the bdf of the physical function (PF) of the C3xxx device - lspci -d:19e2 +.. Note:: -You should see output similar to:: + If the build fails due to missing header files you may need to do following:: - 01:00.0 Co-processor: Intel Corporation Device 19e2 + sudo yum install zlib-devel + sudo yum install openssl-devel -For c3xxx device there is 1 PFs. -Using the sysfs, enable the 16 VFs:: +.. Note:: - echo 16 > /sys/bus/pci/drivers/c3xxx/0000\:01\:00.0/sriov_numvfs + If the build or install fails due to mismatching kernel sources you may need to do the following:: -If you get an error, it's likely you're using a QAT kernel driver earlier than kernel 4.5. + sudo yum install kernel-headers-`uname -r` + sudo yum install kernel-src-`uname -r` + sudo yum install kernel-devel-`uname -r` -To verify that the VFs are available for use - use ``lspci -d:19e3`` to confirm -the bdf of the 16 VF devices are available per ``C3xxx`` device. -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. Binding the available VFs to the DPDK UIO driver ------------------------------------------------ -For **Intel(R) QuickAssist Technology DH895xcc** device: -The unbind command below assumes ``bdfs`` of ``03:01.00-03:04.07``, if yours are different adjust the unbind command below:: +Unbind the VFs from the stock driver so they can be bound to the uio driver. - cd $RTE_SDK - modprobe uio - insmod ./build/kmod/igb_uio.ko +For an Intel(R) QuickAssist Technology DH895xCC device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - for device in $(seq 1 4); do \ - for fn in $(seq 0 7); do \ - echo -n 0000:03:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:03\:0${device}.${fn}/driver/unbind; \ - done; \ - done +The unbind command below assumes ``BDFs`` of ``03:01.00-03:04.07``, if your +VFs are different adjust the unbind command below:: - echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id + for device in $(seq 1 4); do \ + for fn in $(seq 0 7); do \ + echo -n 0000:03:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:03\:0${device}.${fn}/driver/unbind; \ + done; \ + done -You can use ``lspci -vvd:443`` to confirm that all devices are now in use by igb_uio kernel driver. +For an Intel(R) QuickAssist Technology C62x device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -For **Intel(R) QuickAssist Technology C62x** device: -The unbind command below assumes ``bdfs`` of ``1a:01.00-1a:02.07``, ``3d:01.00-3d:02.07`` and ``3f:01.00-3f:02.07``, -if yours are different adjust the unbind command below:: +The unbind command below assumes ``BDFs`` of ``1a:01.00-1a:02.07``, +``3d:01.00-3d:02.07`` and ``3f:01.00-3f:02.07``, if your VFs are different +adjust the unbind command below:: - cd $RTE_SDK - modprobe uio - insmod ./build/kmod/igb_uio.ko + for device in $(seq 1 2); do \ + for fn in $(seq 0 7); do \ + echo -n 0000:1a:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:1a\:0${device}.${fn}/driver/unbind; \ - for device in $(seq 1 2); do \ - for fn in $(seq 0 7); do \ - echo -n 0000:1a:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:1a\:0${device}.${fn}/driver/unbind; \ + echo -n 0000:3d:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:3d\:0${device}.${fn}/driver/unbind; \ - echo -n 0000:3d:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:3d\:0${device}.${fn}/driver/unbind; \ + echo -n 0000:3f:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:3f\:0${device}.${fn}/driver/unbind; \ + done; \ + done - echo -n 0000:3f:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:3f\:0${device}.${fn}/driver/unbind; \ - done; \ - done +For Intel(R) QuickAssist Technology C3xxx device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - echo "8086 37c9" > /sys/bus/pci/drivers/igb_uio/new_id +The unbind command below assumes ``BDFs`` of ``01:01.00-01:02.07``, if your +VFs are different adjust the unbind command below:: -You can use ``lspci -vvd:37c9`` to confirm that all devices are now in use by igb_uio kernel driver. + for device in $(seq 1 2); do \ + for fn in $(seq 0 7); do \ + echo -n 0000:01:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:01\:0${device}.${fn}/driver/unbind; \ + done; \ + done -For **Intel(R) QuickAssist Technology C3xxx** device: -The unbind command below assumes ``bdfs`` of ``01:01.00-01:02.07``, -if yours are different adjust the unbind command below:: +Bind to the DPDK uio driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~ - cd $RTE_SDK - modprobe uio - insmod ./build/kmod/igb_uio.ko +Install the DPDK igb_uio driver, bind the VF PCI Device id to it and use lspci +to confirm the VF devices are now in use by igb_uio kernel driver, +e.g. for the C62x device:: - for device in $(seq 1 2); do \ - for fn in $(seq 0 7); do \ - echo -n 0000:01:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:01\:0${device}.${fn}/driver/unbind; \ + cd to the top-level DPDK directory + modprobe uio + insmod ./build/kmod/igb_uio.ko + echo "8086 37c9" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vvd:37c9 - done; \ - done - echo "8086 19e3" > /sys/bus/pci/drivers/igb_uio/new_id - -You can use ``lspci -vvd:19e3`` to confirm that all devices are now in use by igb_uio kernel driver. - - -The other way to bind the VFs to the DPDK UIO driver is by using the ``dpdk-devbind.py`` script: - -.. code-block:: console +Another way to bind the VFs to the DPDK UIO driver is by using the +``dpdk-devbind.py`` script:: - cd $RTE_SDK + cd to the top-level DPDK directory ./usertools/dpdk-devbind.py -b igb_uio 0000:03:01.1 -- 2.20.1