[kvm-devel] VNC segfault

[Marcelo T. <mto...@re...>]

Hi Anthony,

We're experiencing qemu segfaults when using VNC over high latency
links.

(gdb) bt
#0  0x0000003a8ec838d3 in memcpy () from /lib64/libc.so.6
#1  0x00000000004b9aff in vnc_update_client (opaque=0x3514140) at vnc.c:223
#2  0x000000000040822d in qemu_run_timers (ptimer_head=0x8e9500, current_time=5942450)
    at /root/marcelo/kvm-userspace/qemu/vl.c:1112
#3  0x0000000000413208 in main_loop_wait (timeout=1000)
    at /root/marcelo/kvm-userspace/qemu/vl.c:7482
#4  0x000000000060de86 in kvm_main_loop ()
    at /root/marcelo/kvm-userspace/qemu/qemu-kvm.c:524
#5  0x0000000000413259 in main_loop () at /root/marcelo/kvm-userspace/qemu/vl.c:7506
#6  0x0000000000415d3a in main (argc=21, argv=0x7fff00907dd8)
    at /root/marcelo/kvm-userspace/qemu/vl.c:9369

Problem is that sometimes vs->width and vs->weight are not updated to
reflect the size allocated for the display memory. If they are larger
than whats allocated it segfaults:

(gdb) p vs->old_data_h
$22 = 400
(gdb) p vs->old_data_w
$23 = 720
(gdb) p vs->old_data_depth
$24 = 4

(gdb) p vs->height
$20 = 480
(gdb) p vs->width
$21 = 640
(gdb) p vs->depth
$25 = 4

old_data_h, old_data_w and old_data_depth have been saved from the last
vnc_dpy_resize run. The code relies on the client's "set_encondings"
processing to happen _before_ the vnc_update_client() timer triggers,
which might not always be the case.

I have no clue about correctness of the following though. What do you
say?

diff --git a/qemu/vnc.c b/qemu/vnc.c
index f6ec5cf..5540677 100644
--- a/qemu/vnc.c
+++ b/qemu/vnc.c
@@ -302,7 +302,7 @@ static void vnc_dpy_resize(DisplayState *ds, int w, int h)
     ds->width = w;
     ds->height = h;
     ds->linesize = w * vs->depth;
-    if (vs->csock != -1 && vs->has_resize && size_changed) {
+    if (vs->csock != -1 && size_changed) {
 	vnc_write_u8(vs, 0);  /* msg id */
 	vnc_write_u8(vs, 0);
 	vnc_write_u16(vs, 1); /* number of rects */