Re: [openMSX-devel] Video scaling

[GuyveR800 <guy...@xs...>]

Hi Maarten,

Some of your assumptions are way off, but generally you have the right 
idea. Actually pseudoMSX has worked similar to this for years already. 
Shame that you guys keep redoing work I already did over and over again.

Regards,

Patriek


On 24/02/2019 11:23, Maarten ter Huurne wrote:
> Hi all,
> 
> The way openMSX currently does video scaling has some problems and when
> the SDL2 transition has stabilized it might be a good time to tackle
> these. Whether that is before or after releasing with SDL2 is something
> I don't think we need to answer yet. Rather, I think we should have this
> discussion to be able to answer that question.
> 
> 
> The problems that I'm aware of:
> 
> Resolutions are diverging a lot. We support handhelds with 320x240
> screens, while we also have users that want to run openMSX on 4K
> monitors and TVs. The support for the low-res screens is working fine,
> but for hi-res screens, our maximum scale factor of 4 is not enough.
> 
> In the CRT world, full-screen mode meant a mode change to a different
> native resolution: if you'd ask for 640x480, you'd get exactly 640x480
> on your monitor. In the LCD world, every screen has a native resolution.
> If you output in a different resolution, your video gets scaled
> automatically. However, the result of video getting scaled twice isn't
> as pretty as scaling it once.
> 
> MSX screen modes are designed for 4:3 screens: 256x192 is exactly 4:3,
> 256x212 isn't that far off. However, most modern screens are 16:9 or at
> least some other format that is wider than 4:3. openMSX's scaling system
> is based on a 320x240 base resolution, to which an integer scale factor
> can be applied. This means that if we were to introduce a scale factor
> 5, this would result in a 1600x1200 output, which is too many lines for
> a 1080p screen, even though the display area of the MSX screen is 212 *
> 5 = 1060 lines.
> 
> Setting the scale factor via the OSD console or Catapult is neither
> intuitive nor convenient. It would be much simpler for the user if it
> was possible to resize the openMSX output window and the scale factor
> would adapt automatically.
> 
> Some scalers only work at certain scale factors. If the user selects an
> unsupported combination, openMSX falls back to a different scaler.
> Ideally, every scaler would work at every scale factor, but in practice
> that may not be an option. However, we still could consider whether a
> different fallback would be more appropriate, for example falling back
> to a lower scale factor (bigger borders) or doing a two-step scaling.
> 
> The software scaling code is a pain to maintain:
> - we have to write each scaler twice: once in software and once in pixel
> shaders
> - for good performance, we use templates than we instantiate for each
> zoom factor and color depth; this both complicates the code and
> increases the executable size
> - the number of pixels we'd have to output for 4K is likely more than we
> can render on a single core (an educated guess; I didn't benchmark it)
> and scaling on multiple cores would further complicate the code
> 
> Is there anything I missed?
> 
> 
> To solve these, I think we have to redesign how scaling is done in
> openMSX. At its core, what video scaling does is bridge the gap between
> the MSX resolution and the native resolution of the screen on which
> openMSX is displayed.
> 
> Scaling twice isn't going to look as good as scaling once, so we should
> make openMSX output at the native resolution of the screen. SDL2 has a
> non-exclusive full-screen mode that fits in this model: instead of a
> mode switch, it creates a screen-filling window in the native resolution
> of the screen. This eliminates the delay of the mode switch: the switch
> to full-screen mode becomes almost instant.
> 
> In non-exclusive full-screen mode, SDL2 scales the application's output
> to the native resolution of the screen, if necessary. We avoid this
> second scaling if we make openMSX output in native resolution. However,
> that means that switching from/to full-screen mode changes the scale
> factor, which is not compatible with the scale factor being a setting.
> 
> Having a resizable openMSX window on desktops means the scale factor
> should change as the window becomes larger or smaller. Again, this
> conflicts with the scale factor being a setting.
> 
> So I think we need to drop the scale factor setting. Instead, we would
> compute the scale factor based on the output size and some new settings.
> I don't know yet what these new settings should be, but directly or
> indirectly they should control two things:
> 
> 1. Which area of the MSX screen is considered essential?
> 
> There are MSX applications that do things in the border, like changing
> colors, set adjust, vertical overscan. So the interesting part of an MSX
> screen is more than the 256x212 official mode size. I don't recall what
> the largest possible size is, but some of Matra's games got 224 visible
> lines with overscan, so that's a lower bound.
> 
> If we'd want to ensure 224 lines are visible, that would mean we can't
> use factor 5 on 1080p, since 5 * 224 = 1120. I don't think it's a good
> idea to force a scale factor of 4 on 1080p while playing for example an
> MSX1 game just because there is a small amount of software that wouldn't
> fit on the screen with scale factor 5. So we could have a setting that
> by default doesn't consider the border area essential, but that can be
> changed by the user if they do want to see what is normally border area.
> 
> 2. What pixel aspect ratio does the user like?
> 
> Personally, I think pixel aspect ratio should be very close to 1:1, or
> the resulting image will look terrible. However, these are also users
> that have a different opinion and dislike large borders more than they
> dislike squished or stretch images. So we need a setting to please both
> groups.
> 
> Note that some scale algorithms only support integer zoom factors, in
> which case the pixel aspect ratio must be 1:1 while that algorithm is
> active, unless a second scaling is done.
> 
> 
> In a way such settings mimic the knobs on MSX monitors for horizontal
> and vertical stretch. I guess that validates their existence :)
> 
> Something we could consider while we're redesigning the system is
> support for non-integer scale factors. Some scale algorithms require
> integer scale factors, while others do not. Currently we restrict all
> algorithms to an integer vertical scale factor, while a few algorithms
> respond to a configurable horizontal stretch. We could support non-
> integer scale factors in both directions though.
> 
> 
> As I listed among the problems, software scaling isn't an option for
> large scale factors. If we want to scale to native resolution and don't
> want to scale twice, we must use hardware scaling. Is it feasible to
> force that though?
> 
> Hardware scaling depends on being able to run pixel shaders. Currently
> we only support that via OpenGL, but there are other APIs that support
> pixel shaders, such as OpenGL ES, Vulkan, DirectX and Metal. Of that
> list, I think only OpenGL ES makes sense to support in the short term,
> since it would allow hardware scaling on Android. Also it would require
> the least amount of effort, since OpenGL ES is a subset of OpenGL.
> 
> Hardware scaling also depends on there being an actual GPU in the
> system. Devices like the Dingoo A320 and very old PCs don't have GPUs.
> In my opinion it would be a shame to exclude such systems altogether,
> but at the same time I don't want to put in a large amount of
> maintenance effort for fringe systems. Perhaps a compromise would be to
> support only 1x and 2x scaling with the "simple" algorithm on such
> systems. That would allow us to remove a lot of code, but still make
> ports to GPU-less systems possible.
> 
> 
> So to summarize what a new system would look like:
> - the user controls the output window size, via window resizing and
> switching from/to full-screen mode; openMSX reacts to this by
> automatically updating the scale factor
> - the current scale settings (scale_factor and horizontal_stretch) are
> replaced with new settings that indirectly control the computed
> horizontal and vertical scale factor
> - software scaling will be used only for special limited cases; for
> general use we require hardware scaling
> 
> This is of course still only a rough sketch, but I think it's useful to
> have a broad idea of where we want to go before diving into details.
> 
> Please tell me your ideas and opinions on this topic.
> 
> Bye,
> 		Maarten
> 
> 
> 
> 
> 
> _______________________________________________
> openMSX-devel mailing list
> ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openmsx-devel
>