Using a user-specific conf that wasn't owned by the SSH user wasn't working out.