When right-clicking on a file in the File Browser, the encoding menu should display the encoding that has been saved in the recent.xml file as selected if there is an entry for that file.
Consequently, when opening a file in the file browser, or using file->open, that has an entry in the recent.xml it should be opened with the encoding that is specified in recent.xml. I believe it's intuitive this way since the data is already saved in recent.xml and users will expect that jEdit will open the file in the encoding it was last saved in.
Currently, unless the file is opened from the recent files menu the default encoding is always used (However, unintuitively, caret data that is saved in the recent.xml file is applied to the buffer no matter how the file is opened). With this functionality it is easy to open files in an incorrect encoding. If the user saves the file in an incorrect encoding it can cause corruption of the file.