This is jEdit5.2pre1 on Mac OS X and Java 7u55.
Using the app launcher, the default encoding comes out as US-ASCII. On the command-line with java -jar jedit.jar it is UTF-8 as expected.
java -jar jedit.jar
What does Utilities -> Global Options... -> Encodings show in
Default character encoding?
Trying this again after several weeks of vacation, I get UTF-8 as default (also in the options).
I will check again later, with another Mac OS X installation, when I am back to my office.
I have tried again with a fresh installation of:
Mac OS X 10.9.4 Mavericks
jEdit 5.2pre1 "Mac OS X package"
i.e. it is the DMG that contains an APP.
In this configuration, the problem is absent! Some weeks ago, I could reliably reproduce it 3 times, but not today.
The default of the buffer and the Global Options are both UTF-8.