Re: [openMSX-devel] Video scaling

[Sean Y. <se...@me...>]

Hi,

On Sun, Feb 24, 2019 at 11:23:38AM +0100, Maarten ter Huurne wrote:
> Hi all,
> 
> The way openMSX currently does video scaling has some problems and when 
> the SDL2 transition has stabilized it might be a good time to tackle 
> these. Whether that is before or after releasing with SDL2 is something 
> I don't think we need to answer yet. Rather, I think we should have this 
> discussion to be able to answer that question.
> 
> 
> The problems that I'm aware of:
> 
> Resolutions are diverging a lot. We support handhelds with 320x240 
> screens, while we also have users that want to run openMSX on 4K 
> monitors and TVs. The support for the low-res screens is working fine, 
> but for hi-res screens, our maximum scale factor of 4 is not enough.
> 
> In the CRT world, full-screen mode meant a mode change to a different 
> native resolution: if you'd ask for 640x480, you'd get exactly 640x480 
> on your monitor. In the LCD world, every screen has a native resolution. 
> If you output in a different resolution, your video gets scaled 
> automatically. However, the result of video getting scaled twice isn't 
> as pretty as scaling it once.
> 
> MSX screen modes are designed for 4:3 screens: 256x192 is exactly 4:3, 
> 256x212 isn't that far off. However, most modern screens are 16:9 or at 
> least some other format that is wider than 4:3. openMSX's scaling system 
> is based on a 320x240 base resolution, to which an integer scale factor 
> can be applied. This means that if we were to introduce a scale factor 
> 5, this would result in a 1600x1200 output, which is too many lines for 
> a 1080p screen, even though the display area of the MSX screen is 212 * 
> 5 = 1060 lines.
> 
> Setting the scale factor via the OSD console or Catapult is neither 
> intuitive nor convenient. It would be much simpler for the user if it 
> was possible to resize the openMSX output window and the scale factor 
> would adapt automatically.
> 
> Some scalers only work at certain scale factors. If the user selects an 
> unsupported combination, openMSX falls back to a different scaler. 
> Ideally, every scaler would work at every scale factor, but in practice 
> that may not be an option. However, we still could consider whether a 
> different fallback would be more appropriate, for example falling back 
> to a lower scale factor (bigger borders) or doing a two-step scaling.
> 
> The software scaling code is a pain to maintain:
> - we have to write each scaler twice: once in software and once in pixel 
> shaders
> - for good performance, we use templates than we instantiate for each 
> zoom factor and color depth; this both complicates the code and 
> increases the executable size
> - the number of pixels we'd have to output for 4K is likely more than we 
> can render on a single core (an educated guess; I didn't benchmark it) 
> and scaling on multiple cores would further complicate the code
> 
> Is there anything I missed?
> 
> 
> To solve these, I think we have to redesign how scaling is done in 
> openMSX. At its core, what video scaling does is bridge the gap between 
> the MSX resolution and the native resolution of the screen on which 
> openMSX is displayed.
> 
> Scaling twice isn't going to look as good as scaling once, so we should 
> make openMSX output at the native resolution of the screen. SDL2 has a 
> non-exclusive full-screen mode that fits in this model: instead of a 
> mode switch, it creates a screen-filling window in the native resolution 
> of the screen. This eliminates the delay of the mode switch: the switch 
> to full-screen mode becomes almost instant.
> 
> In non-exclusive full-screen mode, SDL2 scales the application's output 
> to the native resolution of the screen, if necessary. We avoid this 
> second scaling if we make openMSX output in native resolution. However, 
> that means that switching from/to full-screen mode changes the scale 
> factor, which is not compatible with the scale factor being a setting.
> 
> Having a resizable openMSX window on desktops means the scale factor 
> should change as the window becomes larger or smaller. Again, this 
> conflicts with the scale factor being a setting.
> 
> So I think we need to drop the scale factor setting. Instead, we would 
> compute the scale factor based on the output size and some new settings. 
> I don't know yet what these new settings should be, but directly or 
> indirectly they should control two things:
> 
> 1. Which area of the MSX screen is considered essential?
> 
> There are MSX applications that do things in the border, like changing 
> colors, set adjust, vertical overscan. So the interesting part of an MSX 
> screen is more than the 256x212 official mode size. I don't recall what 
> the largest possible size is, but some of Matra's games got 224 visible 
> lines with overscan, so that's a lower bound.
> 
> If we'd want to ensure 224 lines are visible, that would mean we can't 
> use factor 5 on 1080p, since 5 * 224 = 1120. I don't think it's a good 
> idea to force a scale factor of 4 on 1080p while playing for example an 
> MSX1 game just because there is a small amount of software that wouldn't 
> fit on the screen with scale factor 5. So we could have a setting that 
> by default doesn't consider the border area essential, but that can be 
> changed by the user if they do want to see what is normally border area.

That's a great idea.

> 2. What pixel aspect ratio does the user like?
> 
> Personally, I think pixel aspect ratio should be very close to 1:1, or 
> the resulting image will look terrible. However, these are also users 
> that have a different opinion and dislike large borders more than they 
> dislike squished or stretch images. So we need a setting to please both 
> groups.
> 
> Note that some scale algorithms only support integer zoom factors, in 
> which case the pixel aspect ratio must be 1:1 while that algorithm is 
> active, unless a second scaling is done.

In order to correctly overlay the msx video on laserdis, the aspect ratio
has to be different than 1:1. I'm not sure the result has too look terrible,
I assume (not presume) this can be aliased nicely with multi sampling
shaders.

> In a way such settings mimic the knobs on MSX monitors for horizontal 
> and vertical stretch. I guess that validates their existence :)
> 
> Something we could consider while we're redesigning the system is 
> support for non-integer scale factors. Some scale algorithms require 
> integer scale factors, while others do not. Currently we restrict all 
> algorithms to an integer vertical scale factor, while a few algorithms 
> respond to a configurable horizontal stretch. We could support non-
> integer scale factors in both directions though.

Would it be possible to have re-write e.g. RGBTriplet using multisampling
so that it can work at any scale factor?

TBF I'm not entirely clear in my head how this would work.

> As I listed among the problems, software scaling isn't an option for 
> large scale factors. If we want to scale to native resolution and don't 
> want to scale twice, we must use hardware scaling. Is it feasible to 
> force that though?
> 
> Hardware scaling depends on being able to run pixel shaders. Currently 
> we only support that via OpenGL, but there are other APIs that support 
> pixel shaders, such as OpenGL ES, Vulkan, DirectX and Metal. Of that 
> list, I think only OpenGL ES makes sense to support in the short term, 
> since it would allow hardware scaling on Android. Also it would require 
> the least amount of effort, since OpenGL ES is a subset of OpenGL.
> 
> Hardware scaling also depends on there being an actual GPU in the 
> system. Devices like the Dingoo A320 and very old PCs don't have GPUs. 
> In my opinion it would be a shame to exclude such systems altogether, 
> but at the same time I don't want to put in a large amount of 
> maintenance effort for fringe systems. Perhaps a compromise would be to 
> support only 1x and 2x scaling with the "simple" algorithm on such 
> systems. That would allow us to remove a lot of code, but still make 
> ports to GPU-less systems possible.
> 
> 
> So to summarize what a new system would look like:
> - the user controls the output window size, via window resizing and 
> switching from/to full-screen mode; openMSX reacts to this by 
> automatically updating the scale factor
> - the current scale settings (scale_factor and horizontal_stretch) are 
> replaced with new settings that indirectly control the computed 
> horizontal and vertical scale factor
> - software scaling will be used only for special limited cases; for 
> general use we require hardware scaling
> 
> This is of course still only a rough sketch, but I think it's useful to 
> have a broad idea of where we want to go before diving into details.
> 
> Please tell me your ideas and opinions on this topic.

This feature is the one thing I really miss in openmsx. All good stuff!


Sean