mem: retry malloc with smaller block size when failure