Judging from what I can see, some light-related calculations seem to operate on values with gamma/transfer function pre-applied, resulting in incorrect (darker) images.
The issue of not using linear light in calculations is pronounced when a black-and-white pixel-per-pixel checkerboard pattern is displayed using -VICIIfilter 0 and then -VICIIglfilter 0 (correct brightness) compared to e.g. -VICIIglfilter 2 (too dark).
On the other hand, mpv recommends sigmoid instead of linear upscaling (see --sigmoid-* options in man mpv) due to ringing artifacts. I am not sure whether there might be other possible workarounds for VICE in this regard.
Since converting from and to linear light is computational expensive (relatively speaking), this should be optional.