If your OS supports S/Key, you might find that a useful option. S/Key
authentication uses one-time-use passwords; much safer than general use
password protection. You (and any others who need it) can keep a collection
of S/Key passphrases in a wallet, and use them even with public-access
PCs in internet cafes or libraries -- notorious for having keyboard capture
virii and other ills.

To support S/Key -- if the OS allows it -- one merely needs

ChallengeResponseAuthentication yes

in sshd_config.

I use OpenBSD. In that environement, to use S/Key, I merely connect with
":skey" as the login userid to enable the challenge and response. I
get a prompt that shows me which passphrase to use from my list of passphrases.

If I don't use the "...:skey" login 'style' then the only other allowable
authentication is RSA public key.

When using S/Key, I just need to ensure no one else can read the passphrase
list when I remove it from my wallet.