common/mlx5: fix mempool registration

common/mlx5: fix mempool registration

Mempool registration was not correctly processing
mempools with RTE_PKTMBUF_F_PINEND_EXT_BUF flag set
("pinned mempools" for short), because it is not known
at registration time whether the mempool is a pktmbuf one,
and its elements may not yet be initialized to analyze them.
Attempts had been made to recognize such pools,
but there was no robust solution, only the owner of a mempool
(the application or a device) knows its type.
This patch extends common/mlx5 registration code
to accept a hint that the mempool is a pinned one
and uses this capability from net/mlx5 driver.

1. Remove all code assuming pktmbuf pool type
   or trying to recognize the type of a pool.
2. Register pinned mempools used for Rx
   and their external memory on port start.
   Populate the MR cache with all their MRs.
3. Change Tx slow path logic as follows:
   3.1. Search the mempool database for a memory region (MR)
        by the mbuf pool and its buffer address.
   3.2. If not MR for the address is found for the mempool,
	and the mempool contains only pinned external buffers,
	perform the mempool registration of the mempool
	and its external pinned memory.
   3.3. Fall back to using page-based MRs in other cases
	(for example, a buffer with externally attached memory,
	but not from a pinned mempool).

Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities")
Fixes: fec28ca0e3a9 ("net/mlx5: support mempool registration")

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Reviewed-by: Matan Azrad <matan@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>