net/mlx5: close all ports on remove

net/mlx5: close all ports on remove

With the introduction of representors several eth devices are using
the same rte device (e.g. a PCI bus). When calling port detach on one
eth device it is required that all eth devices belonging to the
same rte device have been closed in advance, then the rte device
itself can be removed/detached.
This commit implements this requirement implicitly by adding a
remove callback to struct rte_pci_driver.
The new behavior can be demonstrated in testpmd.
First we attach a representor 0 using PCI address 0000:08:00.0
testpmd> port attach  0000:08:00.0,representor=[0]
Attaching a new port...
EAL: PCI device 0000:08:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1013 net_mlx5
Port 0 is attached.
Done
Port 1 is attached.
Done

Port 0 is the master device (PF) - an ethdev of the PCI address.
Port 1 is representor 0 - another ethdev (representing a VF) using the
same PCI address. Next we detach port 1
testpmd> port detach 1
Removing a device...
Port 0 is closed
Port 1 is closed
Now total ports is 0
Done

Since port 0 has been implicitly closed we cannot act on it anymore.
testpmd> port stop 0
Invalid port 0

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>