#326 SDL2 UI

[Gergely Szasz]

Hi All!

Here is a patch to add SDL2 UI to FUSE.

We could observe a significant rendering speed improvement compared to SDL1 UI. I made some test on a debian linux (H61 mainboard, Celeron G550@2.60GHz, Sandy Bridge HD 2000 GPU).

GTK UI may do better performance, because my desktop run on 32bit RGB visual like GTK UI. SDL and X UIs work on RGB565 so underlaying driver or fuse has to do extra RGB565->RGB888 conversions...

---

Results:

OPIUM4K demo: ./fuse SHOCK.TAP -m 128 -g hq3x --speed 1000
SHOCK demo part 2: ./fuse SHOCK.TAP -m se -g hq2x --speed 1000
SHOCKPEN demo part 4: ./fuse SHOCKPEN.SCL -m pentagon -g hq3x --speed 1000
SHOCKPEN demo part 4(T): ./fuse SHOCKPEN.SCL -m pentagon -g 3x --speed 1000
SHOCK demo part 2(X): ./fuse SHOCK.TAP -m se -g timex15x --speed 1000

Table of measured emulation speeds:

TEST	SDL	SDL2/max	GTK	X11	XV
OPIUM4K	138%	216%/215%	240%	129%	112%
SHOCK	35%	49%/49%	-	-	-
SHOCKP	62%	98%/94%	100%	57%	50%
SHOCKP T	166%	368%/317%	345%	157%	133%
SHOCK X	166%	355%/310%	320%	152%	132%

Note: OPIUM4k is Opium (4k intro), SHOCK is Shock Megademo from WOS, SHOCKP is Shock Megademo (Pentagon Fix). From SHOCK/SHOCKPEN i used the 'border stripes' part of the demo. T is 'Triple size', X is 'Timex 1.5x' filter.

Note: The second value in SDL2 column measured with maximized window (1400x1050)

Note: There is no practical difference between fullscreen and ~960x720 or ~640x480 resolution with XV UI, so all measures made on maximized window (1400x1050).

We can see: SDL2 UI overperform the SDL UI with 50%-60%, and with an easy scaler even surpasses the GTK UI.

With SDL2s 'virtual resolution' we can easily implement a resizeable main window. So, now users can freely resize the main window. If SDL2 run over X11, fuse can use X11/Xutils WMSizeHints to force main window aspect ratio during resizing. (I don't know about similar feature for Windows or other underlying subsytem...??)
If SDL run on other WM or we disable this feature at configuration time (--disable-wm-aspect), fuse try to keep aspect ratio with adjusting the window size after user resize event.

---

Changes:
* Makefile.am:
- add SDL2 LIBS
* configure.ac:
- add SDL2 detection
--with-sdl: first try SDL2 next SDL1
--with-sdl=1: force use of SDL1
--disable-wm-aspect: disable using WM aspect (size) hints with X11 subsytem
* keysyms.{dat,pl}, input.h:
- add some new key symbol to SDL1/2 UI can use more ASCII char in widgets (e.g. underscore, braces, tilde, at...)
* sound/Makefile.am:
- add SDL2 LIBS
* timer/Makefile.am:
- add SDL2 LIBS
* ui/options.dat:
- add fulscreen option for SDL2 UI
* widget/widget.c:
- make 'ui_statusbar_...' functions work from SDL2 UI
* sdl/Makefile.am:
- add SDL2 LIBS
* sdl/sdl2display.h
- add some #define to eliminate the difference between SDL2/SDL1 keycodes (SDLK_...)
* sdl/display.h
- add externs for SDL2
* sdl/sdl2display.c
- new file to implement SDL2 rendering logic. The main difference is SDL2 uses windows, renderers and textures instead of surfaces. The SDL2 UI uses SDL2 renderer virtual size, so very easy to implement rescalable main window. Note: all other files used both (SDL1/SDL2) UI with #ifdefs.
* sdl/sdlui.c:
- SDL2 uses WINDOW events instead of VIDEOEXPOSE, TEXTINPUT for unicode keyboard input, SetWindowTitle instead of WM_SetCaption, SetRelativeMouseMode for grab/ungrab mouse, etc..
- implement poor man's aspect ratio handling, if there is no X11 subsytem (or disabled)
* sdl/sdlkeyboard.{h,c}
- new function to handle SDL2 TEXTINPUT
- we use TEXTINPUT only for widget system
* sdl/sdljoystick.c
- add UI_SDL2 #ifdefs

[Fredrick Meunier]

Looks very interesting:

If SDL2 run over X11, fuse can use X11/Xutils WMSizeHints to force main window aspect ratio during resizing. (I don't know about similar feature for Windows or other underlying subsytem...??)

Similar features exist in e.g. Mac OS X. I'd suggest using a hint like that where possible but leaving it to a later patch to enable on each platform. I don't think that switching on presence of X11 headers is correct as e.g. my Mac has a native SDL but I also have X11 headers so the combination wouldn't be good.

ui/options.dat: I presume there was a problem with the preprocessor that requires the duplication of "Checkbox, Full (s)creen, full_screen, INPUT_KEY_s"?

For 16 bit vs 32 bit, the choice to use 16 bit was just from a time when 32 bit was rare and likely to be an unwanted overhead. SDL UI could be changed to use 16 bit or 32 bit based on the current mode similar to GTK+ and to eliminate the extra conversion where possible.

[Sergio Baldoví]

Hi Gergely,

I have done some testing:

• Resizing window while menu is open don't refresh the screen.
• I have a 16:9 monitor and going fullscreen or maximized window show garbage at the left/right border. I would expect black bars.
• Switching from normal to 2x filter show garbage.
• sdl2display.c: pointless condition: if( 1 || !SDL_SetWindowFullscreen( sdlwin, SDL_WINDOW_FULLSCREEN ) )
• sdl2display.c header: wrong filename and Id tag.
• sdl2keyboard.h header: wrong Id tag.
• I would conditionally compile sdl2display.c or sdldisplay.c rather than empty files, e.g.,

--

libuisdl_a_SOURCES = \
             sdljoystick.c \
             sdlkeyboard.c \
             sdlui.c \
             keysyms.c

if HAVE_SDL2
libuisdl_a_SOURCES += sdl2display.c
else
libuisdl_a_SOURCES += sdldisplay.c
endif

[Gergely Szasz]

Hi Fred!

Now i add a more smarter WM HINT selection part to configure.ac (with x11 subsystem only :-),

Now, the hint switching is not based on just X11 headers. We check SDL2 WM subsystem too. (if ... || wminfo.subsystem != SDL_SYSWM_X11) in sdl2display.c:wm_setsizehints()

In addition, with native SDL2 and 'X11 WM hints', fuse cannot complie, because SDL_SysWMinfo struct does not contain the .info.x11 union...

The duplication of "fullscreen...": ui/options.dat "preprocessed" with ui/widget/options.pl and ui/widget/options-header.pl, so these scripts cannot understand cpp syntax perfectly (e.g.: if defined X1 || defined X2 does not work)

[Gergely Szasz]

Hi Sergio!

Thanks for testing...

Resizing window while menu is open don't refresh the screen. - fixed
Switching from normal to 2x filter show garbage. - fixed
sdl2display.c: pointless condition: if( 1 || !SDL_SetWindowFullscreen( sdlwin, SDL_WINDOW_FULLSCREEN ) ) - fixed SDL2 has two fullscreen mode. Now we use only the 'desktop fullscreen' mode.
sdl2display.c header: wrong filename and Id tag. - fixed
sdl2keyboard.h header: wrong Id tag. - fixed
I would conditionally compile sdl2display.c or sdldisplay.c rather than empty files, - fixed

I have a 16:9 monitor and going fullscreen or maximized window show garbage at the left/right border. I would expect black bars. - I have a wide monitor too, but i always get black bars with fullscreen mode, and maximized window...??

See diff.sdl2_05.patch

[Sergio Baldoví]

Hi Gergely,

I have a wide monitor too, but i always get black bars with fullscreen mode, and maximized window...??

I'm using xfwm4. I suspect renderer should be cleared to black somewhere. Screenshots attached.

Another thing I've seen, menu_data.dat should be patched to show fullscreen entry.

[Gergely Szasz]

I hope, this patch eliminate the garbage on left/right...

menu_data.dat should be patched to show fullscreen entry.

"Fullscreen option" is in ui/options.dat, not in menu_data...
F1->Options->General->Full Screen ... or you thinking something else?..

BTW: I add a hotkey (F11) to Fullscreen...

But I have an another problem:
1. maximize fuse window
2. set scaler to e.g. double
Now the window is remain "maximized" (as we expected), but the screen is resized to 640x480 in the left-bottom corner.

Basically this is not a 'fuse problem', because fuse do nothing to rescale 320x240 or 640x480 to "maximized" size. This done by the SDL2 renderer...

I cannot find (yet) any workaround, because SDL always report the desired window size, not the real. When user maximize the window, fuse cannot resize it. (this is ok., we expected this) But, after fuse try to resize it e.g. to 640x480 SDL_GetWindowSize() return 640x480 not the real (maximized) size of the window (e.g..1402x1052)... ??

In addition: SDL2 never report window is maximized. I never get a SDL_WINDOWEVENT_MAXIMIZED event and SDL_GetWindowFlags() never report: SDL_WINDOW_MAXIMIZED...??

I use FVWM2, but I tested on XFWM4 too.

[Sergio Baldoví]

I hope, this patch eliminate the garbage on left/right...

Yes, now works properly.

"Fullscreen option" is in ui/options.dat, not in menu_data...
F1->Options->General->Full Screen ... or you thinking something else?..
BTW: I add a hotkey (F11) to Fullscreen...

SDL 1 has a menu entry (defined by menu_data.dat):
Options->Full screen (F11)

and a general option (defined by options.dat):
Options->General->Full screen

I'm thinking in using the same menu entries and general options for both versions. Not sure about which implementation is better.

[Gergely Szasz]

Hmm... the SDL2 maximize event bug reported at 2012-03-08 and the wrong size reporting bug reported at 2014-03-16.

... no hope ...

with this new patch, fuse destroy and recreate the window if user switch between scalers with different scaling factor...

[Sergio Baldoví]

with this new patch, fuse destroy and recreate the window if user switch between scalers with different scaling factor...

Window hints don't work after switching scalers.

The window can be resized to a small/unusable size. Would be setting minimum size to 320x240 (plus decorations) reasonable?

In the other hand, is resizing down a 3x scaler a wasteful task? That makes me wonder if using a fixed scaler and a variable scaler at the same time is advisable.

OPTION 1 (current):

• First select a fixed scaler.
• Allow SDL resize window based on current scaler.

OPTION 2 (best of both):

• Window with fixed scalers cannot be resized.
• A variable scaler (SDL) allow resizing from 1x.

OPTION 3 (simple/transgressor):

• Get rid of fixed scalers.
• Use only a variable scaler by SDL.
• 2x and 3x options set the window size to a smooth factor of 1x.

Opinions?

[Gergely Szasz]

Here is a patch with fixed WM hints, and now fuse set minimal window size to scaler natural size (e.g. 640x480 for 2x, etc...), so users can only scale-up...

BTW: there are two scaler groups which not just scale the image, but recall some original visual "effect": TV and PAL TV filters...

[Sergio Baldoví]

When WM hints are disabled, the spectrum screen snap to the low/left border. Is there any chance to center the display in the window? In fullscreen this happens automatically.

[Gergely Szasz]

Hmm... Not, not really... because we do not know the geometry of the window that time...

1. User maximize the window
2. fuse try to resize to 4:3 aspect (e.g. if monitor is widescreen than reduce the width of the window)
3. WM deny to redsize the window (this is o.k. because window is maximized)
4. because SDL bugs, fuse cannot get any information about the 'failed' resize request* (we cannot informed about window is maximized or size didn't changed)
5. SDL render picture...

*actually there is a not so easy way to detect maximized windows. SDL sent a SIZE_CHANGE event then a RESIZE event if window resize succes. If not, only the SIZE_CHANGE event issued. So, if we want to detect the 'failed' request, we need some timeout while waiting for the RESIZE event, then resize back to the 'original' size... then (probably) SDL place the picture to the center.

[Sergio Baldoví]

Hmm... Not, not really... because we do not know the geometry of the window that time...

I think we are messing SDL, e.g.,

1. Disable WM size hints at build time: ./configure --with-sdl --disable-wm-aspect-hint
2. Run Fuse with no aspect hint: fuse --no-aspect-hint
3. Now resize the window.
4. SDL does letterboxing so aspect ratio of spectrum screen looks good.
5. Spectrum screen is centered in the window.

I wonder if this code is really needed inside SDL_WINDOWEVENT_RESIZED event:

if( settings_current.aspect_hint
#ifdef USE_WM_ASPECT_X11
    && !sdldisplay_use_wm_aspect_hint
#endif
) {

  int aspect = 3 * event.window.data1 - 4 * event.window.data2;

  if( aspect < 0 ) /* too high */
    SDL_SetWindowSize( sdlwin, event.window.data1, 3 * event.window.data1 / 4 );
  else if( aspect > 0 ) /* too wide */
    SDL_SetWindowSize( sdlwin, 4 * event.window.data2 / 3, event.window.data2 );
}

That move the spectrum screen position to the bottom and doesn't maintain the aspect ratio of the window (i.e., don't prevent letterboxing). I can't see any advantage.

[Gergely Szasz]

I really miss something... :-?

I build fuse with: ./configure --with-sdl --disable-wm-aspect-hint

1. I run fuse with: ./fuse
2. When i resize window (with my WM I click a corner, move, and click again),
3. fuse change window size to keep 4:3 aspect ratio (e.g.: if i drag wider, fuse narrowing it)
4. i maximize the window
5. fuse try to resize, because it wider than 4:3
6. WM not allow to resize, so window fill the whole desktop, but SDL and fuse think: the window resized, so SDL render the picture at the left. But vertically the picture fill the window. (e.g. if I have a 'portrait' monitor, than SDL render the picture at top, but horizontally fill the window...)

So, i never encounter a left-bottom picture in this situation... (just left only)

And fuse try to prevent aspect ratio (for me), and while i didn't maximize the window it was successful..

BTW: i'm not insist on this code too much (just first time it looked like as a simple solution for maintaining aspect ratio without any 'aid')

Screenshots:
01 - start
02 - during resize (I never see two image at 'realtime ;-)
03 - after resize
04 - after maximized

[Sergio Baldoví]

Oh! I see, it is my system behaving differently, again. The soft method doesn't work for me so I think it is acceptable as it could be disabled by command-line.

[Sergio Baldoví]

I think it is time to test this in as many machines as possible. I'll commit it next weekend barring objections.

[Gergely Szasz]

We have one thing left: the scaler stuff.

In the other hand, is resizing down a 3x scaler a wasteful task? That makes me wonder if using a fixed scaler and a variable scaler at the same time is advisable.

Yes, there is no real reason to use fuse scalers before SDL "free" scaler. (except only the TV and PAL TV effect...)

So:

1. We may use a switch: "SDL scaler" or "`Free' scaler", and when user select, fuse ignore the current_scaler and use the native 320x240 image with SDL "virtual size"

2. or we add a dummy scaler ("`Free' scaler" or with something resonable name) to the scalers (now only for SDL UI) and user can select it from the scalers list and ...

I think the 2nd is "more user friendly" :-)

[Sergio Baldoví]

That makes me wonder if using a fixed scaler and a variable scaler at the same time is advisable.

Yes, there is no real reason to use fuse scalers before SDL "free" scaler. (except only the TV and PAL TV effect...)

Thinking more about this, there is also an antialiasing effect (with different degrees). Moreover, scalers allow to choose a proportional size, e.g., scaling to 2x has less pixel distortion than 1.95x. It is difficult to get a proportional size by resizing the window by hand. If we ever remove fixed scalers, it should be an alternative to effects and size selection.

1- We may use a switch: "SDL scaler" or "`Free' scaler", and when user select, fuse ignore the current_scaler and use the native 320x240 image with SDL "virtual size"

IMHO this is not user friendly, options/scalers should do what they are supposed to do, otherwise would bring confusion.

2- or we add a dummy scaler ("`Free' scaler" or with something resonable name) to the scalers (now only for SDL UI) and user can select it from the scalers list and ...

I don't see the point. How should the free scaler be different to normal scaler? Are you thinking on a simplification of data copies (Spectrum -> tmp_screen -> scl_screen -> texture)?

AFAIK, at the current state, the normal scaler acts like a free scaler and other scalers don't allow scaling down. That's good enough for me but I'm open to other solutions.

[Gergely Szasz]

Thinking more about this, there is also an antialiasing effect (with different degrees).

We don't need fuse "software scaling" for antialiasing effect, because SDL2 can do it with the underlaying hardware accelerated scaler.

Additionally some of the filters has (IMHO) a quite ugly effect if superposed on SDL scaler: Tv2x/Tv3x and DotMatrix.
Some others look better, but smoothed and pixelized at the same time: 2xSai, Super 2xSai, and SuperEagle.

I attached a patch to test the SDL "bluring" effect...

./fuse --sdl-scale-quality <#>

where # is one of:

• 0 or nearest - nearest pixel sampling (default)
• 1 or linear - linear filtering (supported by OpenGL and Direct3D)
• 2 or best - anisotropic filtering (supported by Direct3D)

I don't see the point. How should the free scaler be different to normal scaler?

May I was misleading... I think all "old" scalers should be fixed size, only with "free" scaler resized fuse "freely"...

User may select from one of the

• "fixed size" scalers ("old plain" software scalers)
• or SDL2 (free) scaler with predefined size (2x, 3x)

If user select a fixed scaler, than cannot resize the window by "hand".
But if the SDL2 scaler selected, than fuse allow to resize freely (may try to keep aspect ratio...)

So, the filter list may looks like:
A - Normal
B - Double size
...
O - HQ2x
P - HQ3x
Q - 2x (resizeable)
R - 3x (resizeable)

Hmm..?

[Sergio Baldoví]

I think all "old" scalers should be fixed size, only with "free" scaler resized fuse "freely"...

I pretty much agree with that. If SDL scaler distorts (blur) an "old" scaler when resizing the window, then better use the original size of the scaler. On the other hand, the fullscreen mode don't change the desktop resolution so resizing "old" scalers is rather inevitable.

I attached a patch to test the SDL "bluring" effect...

Does SDL allow a sharp (non blurring) scaling? I've found that my preferred scaler in fullscreen mode is "Triple size" because the image is less blurred and has less unique colours than other scalers.

I've found an artifact using SDL scaler when updating dirty rectangles on large windows where the blur effect is noticeable. There are some marks at the border, probably interpolated pixels that have not been refreshed.

[Gergely Szasz]

The artifact caused by rounding error of not integer scaling factor.

With this patch, fuse updates the whole texture, so the artifact disappeared...

Use: fuse --sdl-scale-quality 0 to sdl "sharp -> nearest neighbour" scaler.

[Fredrick Meunier]

For what it's worth on the Mac where I have freely resizable windows using Open GL I've removed the normal scalers - it's just called "None" and corresponds to the 1x standard scaler.

For the scalers that include some post-processing I've kept both the 2x and 3x versions as generally the higher resolution version scales better (particularly with non-multiples of the source image and bilinear filtering).

I allow the user to choose whether they want nearest neighbour or bilinear scaling and have hot keys to quickly select 1x, 2x, 3x, 4x and 5x zoom levels of the window.

[Sergio Baldoví]

The last patch fix most artefacts I've recently seen, except one, after switching fullscreen and pressing some keys the border shows part of the display before going fullscreen (see attachment). It seems a backbuffer that has not been initialized. That only happens on a laptop with Intel graphics card, but works fine on another laptop with ATI graphics card.

Regarding the scaler stuff, I've mixed feelings about the fixed scalers. I think both solutions are viable:

1. Fixed scalers can't be freely resize, as we were previously discussing.
2. Fixed scalers are freely resizeable, as the MacOSX port does.

SDL2 scaler seems too much blurry to me (even with nearest neighbour method). I have changed my mind and would let users choose a fixed scaler as a base before SDL scaling. I would warn in the documentation about the possible ugly effects when superposing fixed and SDL2 scalers, and the predisposition to use more than 16 standard colours too. But that's just my opinion.

Regarding the size of the spectrum display, I agree with Fred about allowing 1x, 2x, 3x, 4x and 5x zoom levels.

[Sergio Baldoví]

I've just created branch 2014-12-28-sdl2 in [r5100] and committed diff.sdl2_09.patch in [r5101] with slight syntax cleanups.

The patch sdl2_intel_gfx_01a.diff fix the artefact seen in the previous post by cleaning the renderer twice (one for each backbuffer) when resizing the window, but this is an ugly hack because the display flickers.

The SDL documentation points the solution:

The backbuffer should be considered invalidated after each present; do not assume that previous contents will exist between frames. You are strongly encouraged to call SDL_RenderClear() to initialize the backbuffer before starting each new frame's drawing, even if you plan to overwrite every pixel.

The patch sdl2_intel_gfx_01b.diff cleans the renderer in every frame. That seems the proper solution. I have not tested the impact on performance, though.

[Sergio Baldoví]

I've committed patch sdl2_intel_gfx_01b.diff in [r5107] as doesn't have a significant impact on performance.

I've tested SDL2 UI performance (see attachments) as code has changed since first patch. In my PC, SDL2 UI is slightly worse than SDL1 with 1x filter, moderately worse with 2x filter and considerably worse with 3x filter.