#7 Auto-reconnect after VC restart can break handle reinit

[Alex Davidson]

From time to time, VirtualCenter decides to respond to every PutUpdates request with 'Invalid command entity'. No idea what causes this. Restarting the VC service fixes it.

Managed.VIM then gets stuck in a loop during reconnect, filling the log file with the following:

2010-01-30 10:58:03,627 [3632] INFO Managed.VIM.VimHandleCache - Reinitialising single handle vma-0000-0000-0000
2010-01-30 10:58:03,877 [3632] INFO Managed.VIM.VimHandleCache - Reinitialised handle vma-0000-0000-0000
2010-01-30 10:58:03,940 [3632] ERROR Managed.VIM.VimObjectUpdater - Unhandled exception in message loop
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at Managed.VIM.VimHandleCache.Add(IVimHandle vimHandle) in C:\inetpub\wwwroot\VirtualCloud\Managed.VIM\VimHandleCache.cs:line 85
at Managed.VIM.VimHandleCache.ReinitialiseObject(IVimHandle handle, RenewHandleDelegate renew) in C:\inetpub\wwwroot\VirtualCloud\Managed.VIM\VimHandleCache.cs:line 265
at Managed.VIM.VimObjectUpdater.ProcessHandleFault(IEnumerable`1 handleFaults) in C:\inetpub\wwwroot\VirtualCloud\Managed.VIM\VimObjectUpdater.cs:line 241
at Managed.VIM.VimObjectUpdater.MessagePump() in C:\inetpub\wwwroot\VirtualCloud\Managed.VIM\VimObjectUpdater.cs:line 197

The largest log seen so far was 16GB, from a process that ran for several hours.

This breakage does not appear to prevent the interface being used, but it does prevent shutdown of the message loop when the process tries to exit.

[Alex Davidson]

ReinitialiseObjects(), VimHandleCache.cs:line 247 re-adds a handle to the cache if it fails to reinitialise as part of the reconnect. This allows it to be relinked later.

But at this stage the handle's Disposed handler has been disconnected, since ReinitialiseObjects() discards and rebuilds the cache instead of modifying it. If the handle is seen first by ReinitialiseObject(), the Unlink call in RenewHandle(), VimObjectUpdater.cs:line 116 will not remove it from the cache, causing a key conflict when it gets re-added.