net/cxgbe: improve latency for slow traffic

net/cxgbe: improve latency for slow traffic

TX coalescing waits for ETH_COALESCE_PKT_NUM packets to be coalesced
across bursts before transmitting them.  For slow traffic, such as
100 PPS, this approach increases latency since packets are received
one at a time and tx coalescing has to wait for ETH_COALESCE_PKT
number of packets to arrive before transmitting.

To fix this:

- Update rx path to use status page instead and only receive packets
  when either the ingress interrupt timer threshold (5 us) or
  the ingress interrupt packet count threshold (32 packets) fires.
  (i.e. whichever happens first).

- If number of packets coalesced is <= number of packets sent
  by tx burst function, stop coalescing and transmit these packets
  immediately.

Also added compile time option to favor throughput over latency by
default.

Signed-off-by: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
Signed-off-by: Kumar Sanghvi <kumaras@chelsio.com>