If you want to remove a passphrase, just use ssh -p.