malloc: fix DMA mask check