The "custom" fullscreen mode doesn't use the given (custom) width/height settings for the resolution, resulting in unexpected resolution switching. Descriptive log comparison:
--
vice-2.3 $ src/x64 -config nosuchfile -VICIIhwscale -sdlcustomw 1280 -sdlcustomh 1024 -VICIISDLfullmode 1 -VICIIfull
*** VICE Version 2.3 ***
OS compiled for: Linux
GUI compiled for: SDL
CPU compiled for: AMD64/x86_64
Compiler used: GCC-4.5.4
[...]
Drive: Finished loading ROM images.
SDLVideo: Warning - bitdepth not set for OpenGL, trying 32...
SDLVideo: 1280x1024 32bpp OpenGL (fullscreen)
SDLVideo: 1280x1024 32bpp OpenGL (fullscreen)
--
vice-2.4 $ src/x64 -config nosuchfile -VICIIhwscale -sdlcustomw 1280 -sdlcustomh 1024 -VICIISDLfullmode 1 -VICIIfull
*** VICE Version 2.4 ***
OS compiled for: Linux
GUI compiled for: SDL
CPU compiled for: AMD64/x86_64
Compiler used: GCC-4.5.4
[...]
SDLVideo: Warning - bitdepth not set for OpenGL, trying 32...
SDLVideo: 719x544 32bpp OpenGL (fullscreen)
--
SDL decides on 800x600 on the latter (broken) case, as it would on the "automatic" fullscreen mode. The expected result is using 1280x1024 as the resolution (as in 2.3).
maybe the same as this one?
http://sourceforge.net/tracker/?func=detail&aid=3570771&group_id=223021&atid=1057617
fixed in r27531.