I took a quick look at using a DSA key and it appears you're correct - the 1028 bit DSA key I generated on OS X Server 10.6.8 works fine for SFTP connections in jEdit 5.0 with FTP 1.0.3. I'll need to play around with the various keys which can be generated with ssh-keygen and see how various types of keys behave.
Since my last look into this issue I've updated the local machine to OS X 10.9 so...
Sorry if my "steps to reproduce" weren't clear - I'm not entering a password into the "Connect to Secure FTP Server Dialog", I'm entering the password / passphrase into the "Enter passphrase for private key file" dialog which pops up when attempting to connect to the remote server using SFTP. The passphrase is never accepted even if it is correct and the private key...
jEdit Version Numbers: 5.0.0 / 5.1pre1
Plugins: FTP 1.0.3 / SshConsole 1.0.6a
Platform: OS X version 10.8.4
Java Version: 1.7.0_25-b15
Steps to Reproduce:
- Create an SSH2 key pair using the default settings with 'ssh-keygen' on OS X 10.8 - this results in a AES-128-CBC encrypted key.
- Add the public key to server's authorized keys and log in via SSH in Terminal to verify the new key...