The cocoa implementation currently doesn't support SCI_{GET|SET}FONTQUALITY; the attached patch adds it. I don't know how to add it to the unit tests (actually, I don't know how to build Scintilla tests that are relevant to cocoa), so I randomly added it to the Font menu in the demo app. Note that I'm not very sure I hacked the .nib correctly - but it does seem to show up and work correctly.
Committed as [3b2f50].
Wrapped in some version checking since CGContextSetAllowsFontSubpixelPositioning is not available in OS X 10.5.
Related
Commit: [3b2f50]
There is also CGContextSetAllowsFontSmoothing. Adding this code makes the text look better to me:
CGContextSetAllowsFontSmoothing((CGContextRef)gc,
vs.extraFontFlag != SC_EFF_QUALITY_NON_ANTIALIASED);