test/crypto: add RSA and Mod tests