vdpa/mlx5: fix guest notification timing

vdpa/mlx5: fix guest notification timing

When the HW finishes to consume the guest Rx descriptors, it creates a
CQE in the CQ.

The mlx5 driver arms the CQ to get notifications when a specific CQE
index is created - the index to be armed is the next CQE index which
should be polled by the driver.

The mlx5 driver configured the kernel driver to send notification to the
guest callfd in the same time it arrives to the mlx5 driver.

It means that the guest was notified only for each first CQE in a poll
cycle, so if the driver polled CQEs of all the virtio queue available
descriptors, the guest was not notified again for the rest because
there was no any new cycle for polling.

Hence, the Rx queues might be stuck when the guest didn't work with
poll mode.

Move the guest notification to be after the driver consumes all the
SW own CQEs.
By this way, guest will be notified only after all the SW CQEs are
polled.

Also init the CQ to be with HW owner in the start.

Fixes: 8395927cdfaf ("vdpa/mlx5: prepare HW queues")

Signed-off-by: Matan Azrad <matan@mellanox.com>