net/failsafe: mitigate data plane atomic operations

net/failsafe: mitigate data plane atomic operations

Fail-safe uses atomic operations to protect sub-device close operation
calling by host thread in removal time while the removed sub-device
burst functions are still in process by application threads.

Using "set" atomic operations is a little bit more efficient than "add"
or "sub" atomic operations because "set" shouldn't read the value and
in fact, it does not need a special atomic mechanism in x86 platforms.

Replace "add 1" and "sub 1" atomic operations by "set 1" and "set 0"
atomic operations.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Gaetan Rivet <gaetan.rivet@6wind.com>