Looks like xine-lib bug (maybe it does not validate OSD dirty area coordinates).
I did small change to xineliboutput. Does this happen still with latest xineliboutput and xine-lib ?
If yes, could you produce more verbose log by compiling video_out_vdpau.c with
#define LOG
added to beginning of the file ?