From: Jasvinder Singh Date: Tue, 26 Nov 2019 14:28:29 +0000 (+0000) Subject: doc: update QoS scheduler guides X-Git-Url: http://git.droids-corp.org/?a=commitdiff_plain;h=694fd2cb8d10be0deb8bd7d41dee73601f31d415;p=dpdk.git doc: update QoS scheduler guides Updates documentation to reflect the changes in the QoS scheduler library and example. Signed-off-by: Jasvinder Singh Acked-by: Cristian Dumitrescu --- diff --git a/doc/guides/prog_guide/img/sched_hier_per_port.png b/doc/guides/prog_guide/img/sched_hier_per_port.png deleted file mode 100644 index 462e88aaab..0000000000 Binary files a/doc/guides/prog_guide/img/sched_hier_per_port.png and /dev/null differ diff --git a/doc/guides/prog_guide/img/sched_hier_per_port.svg b/doc/guides/prog_guide/img/sched_hier_per_port.svg new file mode 100644 index 0000000000..759dead832 --- /dev/null +++ b/doc/guides/prog_guide/img/sched_hier_per_port.svg @@ -0,0 +1,492 @@ + + + + + + + + + + + + + + + + + + Page-1 + + + + Sheet.234 + + Sheet.1 + + + + Sheet.2 + + + + Sheet.3 + + + + Sheet.4 + + + + Sheet.6 + + + + Sheet.7 + + + + Sheet.8 + + + + Sheet.9 + + + + Sheet.10 + + + + Sheet.11 + + + + Sheet.12 + + + + Sheet.13 + + + + Sheet.14 + + + + Sheet.15 + + + + Sheet.16 + + + + Sheet.17 + + + + Sheet.26 + + + + Sheet.27 + + + + Sheet.28 + + + + Sheet.29 + + + + Sheet.30 + + + + Sheet.31 + + + + Sheet.32 + + + + Sheet.33 + + + + Sheet.40 + + + + Sheet.42 + + + + Sheet.44 + + + + Sheet.46 + + + + Sheet.47 + + + + Sheet.48 + + + + Sheet.49 + + + + Sheet.50 + + + + Dynamic connector + + + + Dynamic connector.52 + + + + Dynamic connector.157 + + + + Dynamic connector.158 + + + + Dynamic connector.159 + + + + Dynamic connector.160 + + + + Dynamic connector.161 + + + + Dynamic connector.162 + + + + Dynamic connector.163 + + + + Dynamic connector.164 + + + + Dynamic connector.165 + + + + Dynamic connector.166 + + + + Dynamic connector.167 + + + + Dynamic connector.168 + + + + Dynamic connector.170 + + + + Dynamic connector.171 + + + + Dynamic connector.172 + + + + Dynamic connector.173 + + + + Dynamic connector.174 + + + + Dynamic connector.175 + + + + Sheet.176 + Queue + + + + Queue + + Sheet.177 + + + + Sheet.178 + + + + Sheet.179 + + + + Sheet.180 + + + + Sheet.181 + + + + Sheet.182 + + + + Sheet.183 + + + + Sheet.184 + + + + Sheet.185 + + + + Sheet.186 + + + + Sheet.187 + + + + Sheet.188 + + + + Sheet.189 + + + + Sheet.190 + + + + Sheet.191 + + + + Sheet.192 + + + + Sheet.193 + + + + Sheet.194 + + + + Sheet.195 + + + + Sheet.196 + + + + Sheet.197 + + + + Sheet.198 + + + + Sheet.199 + + + + Sheet.200 + + + + Sheet.201 + + + + Sheet.202 + + + + Sheet.203 + + + + Sheet.204 + + + + Sheet.205 + + + + Sheet.206 + + + + Sheet.207 + + + + Sheet.208 + + + + Dynamic connector.209 + + + + Dynamic connector.210 + + + + Dynamic connector.211 + + + + Dynamic connector.212 + + + + Dynamic connector.213 + + + + Dynamic connector.214 + + + + Dynamic connector.215 + + + + Dynamic connector.216 + + + + Dynamic connector.217 + + + + Dynamic connector.218 + + + + Dynamic connector.219 + + + + Dynamic connector.220 + + + + Dynamic connector.221 + + + + Dynamic connector.222 + + + + Dynamic connector.223 + + + + Dynamic connector.224 + + + + Dynamic connector.225 + + + + Dynamic connector.226 + + + + Dynamic connector.227 + + + + Dynamic connector.228 + + + + Sheet.230 + Traffic Class + + + + TrafficClass + + Sheet.231 + Pipe + + + + Pipe + + Sheet.232 + Subport + + + + Subport + + Sheet.233 + Port + + + + Port + + + diff --git a/doc/guides/prog_guide/qos_framework.rst b/doc/guides/prog_guide/qos_framework.rst index a7527b21ca..a159709450 100644 --- a/doc/guides/prog_guide/qos_framework.rst +++ b/doc/guides/prog_guide/qos_framework.rst @@ -171,7 +171,7 @@ The functionality of each hierarchical level is detailed in the following table. | | | | token bucket per pipe. | | | | | | +---+--------------------+----------------------------+---------------------------------------------------------------+ - | 4 | Traffic Class (TC) | 4 | #. TCs of the same pipe handled in strict priority order. | + | 4 | Traffic Class (TC) | 13 | #. TCs of the same pipe handled in strict priority order. | | | | | | | | | | #. Upper limit enforced per TC at the pipe level. | | | | | | @@ -183,8 +183,13 @@ The functionality of each hierarchical level is detailed in the following table. | | | | adjusted value that is shared by all the subport pipes. | | | | | | +---+--------------------+----------------------------+---------------------------------------------------------------+ - | 5 | Queue | 4 | #. Queues of the same TC are serviced using Weighted Round | - | | | | Robin (WRR) according to predefined weights. | + | 5 | Queue | High priority TCs: 1, | #. All the high priority TCs (TC0, TC1, ...,TC11) have | + | | | Lowest priority TC: 4 | exactly 1 queue, while the lowest priority TC (TC12), | + | | | | called Best Effort (BE), has 4 queues. | + | | | | | + | | | | #. Queues of the lowest priority TC (BE) are serviced using | + | | | | Weighted Round Robin (WRR) according to predefined weights| + | | | | weights. | | | | | | +---+--------------------+----------------------------+---------------------------------------------------------------+ @@ -730,10 +735,10 @@ Implementation of Strict Priority Scheduling Strict priority scheduling of traffic classes within the same pipe is implemented by the pipe dequeue state machine, which selects the queues in ascending order. -Therefore, queues 0..3 (associated with TC 0, highest priority TC) are handled before -queues 4..7 (TC 1, lower priority than TC 0), -which are handled before queues 8..11 (TC 2), -which are handled before queues 12..15 (TC 3, lowest priority TC). +Therefore, queue 0 (associated with TC 0, highest priority TC) is handled before +queue 1 (TC 1, lower priority than TC 0), +which is handled before queue 2 (TC 2, lower priority than TC 1) and it conitnues until queues of all TCs except the +lowest priority TC are handled. At last, queues 12..15 (best effort TC, lowest priority TC) are handled. Upper Limit Enforcement ''''''''''''''''''''''' @@ -753,14 +758,14 @@ as described in :numref:`table_qos_10` and :numref:`table_qos_11`. | # | Subport or pipe field | Unit | Description | | | | | | +===+=======================+=======+=======================================================================+ - | 1 | tc_time | Bytes | Time of the next update (upper limit refill) for the 4 TCs of the | + | 1 | tc_time | Bytes | Time of the next update (upper limit refill) for the TCs of the | | | | | current subport / pipe. | | | | | | | | | | See Section `Internal Time Reference`_ for the | | | | | explanation of why the time is maintained in byte units. | | | | | | +---+-----------------------+-------+-----------------------------------------------------------------------+ - | 2 | tc_period | Bytes | Time between two consecutive updates for the 4 TCs of the current | + | 2 | tc_period | Bytes | Time between two consecutive updates for the all TCs of the current | | | | | subport / pipe. This is expected to be many times bigger than the | | | | | typical value of the token bucket tb_period. | | | | | | @@ -815,7 +820,7 @@ as described in :numref:`table_qos_10` and :numref:`table_qos_11`. Weighted Round Robin (WRR) """""""""""""""""""""""""" -The evolution of the WRR design solution from simple to complex is shown in :numref:`table_qos_12`. +The evolution of the WRR design solution for the lowest priority traffic class (best effort TC) from simple to complex is shown in :numref:`table_qos_12`. .. _table_qos_12: @@ -977,33 +982,33 @@ with the third approach selected for implementation. | | | | +-----+---------------------------+-------------------------------------------------------------------------+ -Typically, the subport TC oversubscription feature is enabled only for the lowest priority traffic class (TC 3), +Typically, the subport TC oversubscription feature is enabled only for the lowest priority traffic class, which is typically used for best effort traffic, with the management plane preventing this condition from occurring for the other (higher priority) traffic classes. -To ease implementation, it is also assumed that the upper limit for subport TC 3 is set to 100% of the subport rate, -and that the upper limit for pipe TC 3 is set to 100% of pipe rate for all subport member pipes. +To ease implementation, it is also assumed that the upper limit for subport best effort TC is set to 100% of the subport rate, +and that the upper limit for pipe best effort TC is set to 100% of pipe rate for all subport member pipes. Implementation Overview ''''''''''''''''''''''' The algorithm computes a watermark, which is periodically updated based on the current demand experienced by the subport member pipes, -whose purpose is to limit the amount of traffic that each pipe is allowed to send for TC 3. +whose purpose is to limit the amount of traffic that each pipe is allowed to send for best effort TC. The watermark is computed at the subport level at the beginning of each traffic class upper limit enforcement period and the same value is used by all the subport member pipes throughout the current enforcement period. illustrates how the watermark computed as subport level at the beginning of each period is propagated to all subport member pipes. At the beginning of the current enforcement period (which coincides with the end of the previous enforcement period), -the value of the watermark is adjusted based on the amount of bandwidth allocated to TC 3 at the beginning of the previous period that +the value of the watermark is adjusted based on the amount of bandwidth allocated to best effort TC at the beginning of the previous period that was not left unused by the subport member pipes at the end of the previous period. -If there was subport TC 3 bandwidth left unused, +If there was subport best effort TC bandwidth left unused, the value of the watermark for the current period is increased to encourage the subport member pipes to consume more bandwidth. -Otherwise, the value of the watermark is decreased to enforce equality of bandwidth consumption among subport member pipes for TC 3. +Otherwise, the value of the watermark is decreased to enforce equality of bandwidth consumption among subport member pipes for best effort TC. The increase or decrease in the watermark value is done in small increments, so several enforcement periods might be required to reach the equilibrium state. -This state can change at any moment due to variations in the demand experienced by the subport member pipes for TC 3, for example, +This state can change at any moment due to variations in the demand experienced by the subport member pipes for best effort TC, for example, as a result of demand increase (when the watermark needs to be lowered) or demand decrease (when the watermark needs to be increased). When demand is low, the watermark is set high to prevent it from impeding the subport member pipes from consuming more bandwidth. @@ -1084,10 +1089,27 @@ The highest value for the watermark is picked as the highest rate configured for | | | | | | | tc3_cons = subport_tc3_credits_per_period - subport_tc3_credits; | | | | | - | | | tc3_cons_max = subport_tc3_credits_per_period - (tc0_cons + tc1_cons + | - | | | tc2_cons); | + | | | tc4_cons = subport_tc4_credits_per_period - subport_tc4_credits; | + | | | | + | | | tc5_cons = subport_tc5_credits_per_period - subport_tc5_credits; | + | | | | + | | | tc6_cons = subport_tc6_credits_per_period - subport_tc6_credits; | + | | | | + | | | tc7_cons = subport_tc7_credits_per_period - subport_tc7_credits; | + | | | | + | | | tc8_cons = subport_tc8_credits_per_period - subport_tc8_credits; | | | | | - | | | if(tc3_consumption > (tc3_consumption_max - MTU)){ | + | | | tc9_cons = subport_tc9_credits_per_period - subport_tc9_credits; | + | | | | + | | | tc10_cons = subport_tc10_credits_per_period - subport_tc10_credits; | + | | | | + | | | tc11_cons = subport_tc11_credits_per_period - subport_tc11_credits; | + | | | | + | | | tc_be_cons_max = subport_tc_be_credits_per_period - (tc0_cons + tc1_cons + | + | | | tc2_cons + tc3_cons + tc4_cons + tc5_cons + tc6_cons + tc7_cons + tc8_cons + | + | | | tc9_cons + tc10_cons + tc11_cons); | + | | | | + | | | if(tc_be_consumption > (tc_be_consumption_max - MTU)){ | | | | | | | | wm -= wm >> 7; | | | | | @@ -1555,6 +1577,52 @@ A sample RED configuration is shown below. In this example, the queue size is 64 tc 3 wred inv prob = 10 10 10 tc 3 wred weight = 9 9 9 + tc 4 wred min = 28 22 16 + tc 4 wred max = 32 32 32 + tc 4 wred inv prob = 10 10 10 + tc 4 wred weight = 9 9 9 + + tc 5 wred min = 28 22 16 + tc 5 wred max = 32 32 32 + tc 5 wred inv prob = 10 10 10 + tc 5 wred weight = 9 9 9 + + tc 6 wred min = 28 22 16 + tc 6 wred max = 32 32 32 + tc 6 wred inv prob = 10 10 10 + tc 6 wred weight = 9 9 9 + + tc 7 wred min = 28 22 16 + tc 7 wred max = 32 32 32 + tc 7 wred inv prob = 10 10 10 + tc 7 wred weight = 9 9 9 + + tc 8 wred min = 28 22 16 + tc 8 wred max = 32 32 32 + tc 8 wred inv prob = 10 10 10 + tc 8 wred weight = 9 9 9 + + tc 9 wred min = 28 22 16 + tc 9 wred max = 32 32 32 + tc 9 wred inv prob = 10 10 10 + tc 9 wred weight = 9 9 9 + + + tc 10 wred min = 28 22 16 + tc 10 wred max = 32 32 32 + tc 10 wred inv prob = 10 10 10 + tc 10 wred weight = 9 9 9 + + tc 11 wred min = 28 22 16 + tc 11 wred max = 32 32 32 + tc 11 wred inv prob = 10 10 10 + tc 11 wred weight = 9 9 9 + + tc 12 wred min = 28 22 16 + tc 12 wred max = 32 32 32 + tc 12 wred inv prob = 10 10 10 + tc 12 wred weight = 9 9 9 + With this configuration file, the RED configuration that applies to green, yellow and red packets in traffic class 0 is shown in :numref:`table_qos_18`. diff --git a/doc/guides/sample_app_ug/ip_pipeline.rst b/doc/guides/sample_app_ug/ip_pipeline.rst index 4da0fcf896..56014be174 100644 --- a/doc/guides/sample_app_ug/ip_pipeline.rst +++ b/doc/guides/sample_app_ug/ip_pipeline.rst @@ -249,27 +249,35 @@ Traffic manager tmgr subport profile - + + + - + pps + qsize + + + Add traffic manager pipe profile :: tmgr pipe profile - + + + - + + Create traffic manager port :: tmgr rate spp - pps - qsize - - fo mtu cpu + fo + mtu + cpu Configure traffic manager subport :: diff --git a/doc/guides/sample_app_ug/qos_scheduler.rst b/doc/guides/sample_app_ug/qos_scheduler.rst index cdd29d90c3..b5010657a7 100644 --- a/doc/guides/sample_app_ug/qos_scheduler.rst +++ b/doc/guides/sample_app_ug/qos_scheduler.rst @@ -22,7 +22,7 @@ There are two flavors of the runtime execution for this application, with two or three threads per each packet flow configuration being used. The RX thread reads packets from the RX port, classifies the packets based on the double VLAN (outer and inner) and -the lower two bytes of the IP destination address and puts them into the ring queue. +the lower byte of the IP destination address and puts them into the ring queue. The worker thread dequeues the packets from the ring and calls the QoS scheduler enqueue/dequeue functions. If a separate TX core is used, these are sent to the TX ring. Otherwise, they are sent directly to the TX port. @@ -129,18 +129,28 @@ The profile file has the following format: frame overhead = 24 number of subports per port = 1 - number of pipes per subport = 4096 - queue sizes = 64 64 64 64 ; Subport configuration [subport 0] + number of pipes per subport = 4096 + queue sizes = 64 64 64 64 64 64 64 64 64 64 64 64 64 tb rate = 1250000000; Bytes per second tb size = 1000000; Bytes tc 0 rate = 1250000000; Bytes per second tc 1 rate = 1250000000; Bytes per second tc 2 rate = 1250000000; Bytes per second tc 3 rate = 1250000000; Bytes per second + tc 4 rate = 1250000000; Bytes per second + tc 5 rate = 1250000000; Bytes per second + tc 6 rate = 1250000000; Bytes per second + tc 7 rate = 1250000000; Bytes per second + tc 8 rate = 1250000000; Bytes per second + tc 9 rate = 1250000000; Bytes per second + tc 10 rate = 1250000000; Bytes per second + tc 11 rate = 1250000000; Bytes per second + tc 12 rate = 1250000000; Bytes per second + tc period = 10; Milliseconds tc oversubscription period = 10; Milliseconds @@ -156,17 +166,32 @@ The profile file has the following format: tc 1 rate = 305175; Bytes per second tc 2 rate = 305175; Bytes per second tc 3 rate = 305175; Bytes per second + tc 4 rate = 305175; Bytes per second + tc 5 rate = 305175; Bytes per second + tc 6 rate = 305175; Bytes per second + tc 7 rate = 305175; Bytes per second + tc 8 rate = 305175; Bytes per second + tc 9 rate = 305175; Bytes per second + tc 10 rate = 305175; Bytes per second + tc 11 rate = 305175; Bytes per second + tc 12 rate = 305175; Bytes per second tc period = 40; Milliseconds tc 0 oversubscription weight = 1 tc 1 oversubscription weight = 1 tc 2 oversubscription weight = 1 tc 3 oversubscription weight = 1 - - tc 0 wrr weights = 1 1 1 1 - tc 1 wrr weights = 1 1 1 1 - tc 2 wrr weights = 1 1 1 1 - tc 3 wrr weights = 1 1 1 1 + tc 4 oversubscription weight = 1 + tc 5 oversubscription weight = 1 + tc 6 oversubscription weight = 1 + tc 7 oversubscription weight = 1 + tc 8 oversubscription weight = 1 + tc 9 oversubscription weight = 1 + tc 10 oversubscription weight = 1 + tc 11 oversubscription weight = 1 + tc 12 oversubscription weight = 1 + + tc 12 wrr weights = 1 1 1 1 ; RED params per traffic class and color (Green / Yellow / Red) @@ -191,6 +216,51 @@ The profile file has the following format: tc 3 wred inv prob = 10 10 10 tc 3 wred weight = 9 9 9 + tc 4 wred min = 48 40 32 + tc 4 wred max = 64 64 64 + tc 4 wred inv prob = 10 10 10 + tc 4 wred weight = 9 9 9 + + tc 5 wred min = 48 40 32 + tc 5 wred max = 64 64 64 + tc 5 wred inv prob = 10 10 10 + tc 5 wred weight = 9 9 9 + + tc 6 wred min = 48 40 32 + tc 6 wred max = 64 64 64 + tc 6 wred inv prob = 10 10 10 + tc 6 wred weight = 9 9 9 + + tc 7 wred min = 48 40 32 + tc 7 wred max = 64 64 64 + tc 7 wred inv prob = 10 10 10 + tc 7 wred weight = 9 9 9 + + tc 8 wred min = 48 40 32 + tc 8 wred max = 64 64 64 + tc 8 wred inv prob = 10 10 10 + tc 8 wred weight = 9 9 9 + + tc 9 wred min = 48 40 32 + tc 9 wred max = 64 64 64 + tc 9 wred inv prob = 10 10 10 + tc 9 wred weight = 9 9 9 + + tc 10 wred min = 48 40 32 + tc 10 wred max = 64 64 64 + tc 10 wred inv prob = 10 10 10 + tc 10 wred weight = 9 9 9 + + tc 11 wred min = 48 40 32 + tc 11 wred max = 64 64 64 + tc 11 wred inv prob = 10 10 10 + tc 11 wred weight = 9 9 9 + + tc 12 wred min = 48 40 32 + tc 12 wred max = 64 64 64 + tc 12 wred inv prob = 10 10 10 + tc 12 wred weight = 9 9 9 + Interactive mode ~~~~~~~~~~~~~~~~ @@ -295,11 +365,11 @@ This application classifies based on the QinQ double VLAN tags and the IP destin | Pipe | Config (4k) | Traffic shaped (token bucket) | Inner VLAN tag | | | | | | +----------------+-------------------------+--------------------------------------------------+----------------------------------+ - | Traffic Class | 4 | TCs of the same pipe services in strict priority | Destination IP address (0.0.X.0) | + | Traffic Class | 13 | TCs of the same pipe services in strict priority | Destination IP address (0.0.0.X) | | | | | | +----------------+-------------------------+--------------------------------------------------+----------------------------------+ - | Queue | 4 | Queue of the same TC serviced in WRR | Destination IP address (0.0.0.X) | - | | | | | + | Queue | High Priority TC: 1, | Queue of lowest priority traffic | Destination IP address (0.0.0.X) | + | | Lowest Priority TC: 4 | class (Best effort) serviced in WRR | | +----------------+-------------------------+--------------------------------------------------+----------------------------------+ Please refer to the "QoS Scheduler" chapter in the *DPDK Programmer's Guide* for more information about these parameters.