net/vmxnet3: improve error checks and return values