[Mingw-users] `as.exe' hanging intermittently (SOLVED)

[Craig R. <cr...@po...>]

Hi all

I'm running into a rather odd problem with my new mingw+msys install
that has me at my wit's end, and I'm hoping for some help or ideas. Or I 
was when I initially wrote this - see below.

When I compile programs - even trivial test programs - it usually works
fine, but some of the time gcc/g++ hang indefinitely. A bit of
investigation with Process Explorer revealed that the instance of as.exe
exec'd by gcc is always running when the hang occurs. `ps -aW' also 
shows it, but interestingly it has a pid of 0, with only the winpid 
being non 0. Process Explorer suggests that `as.exe' is not using any 
CPU time, nor is its memory allocation changing at all. I strongly 
suspect it's deadlocked somewhere. With more testing I found that the 
hang occurs whether as is run from gcc, or directly from the shell.

Update: while writing this, I appear to have solved the issue. Of all 
the insane things, the Logitech camera monitor that handles the built-in 
webcam on this Acer TravelMate 8200 seems to have somehow been 
interfering with msys, specifically as.exe. It doesn't make sense, but 
appears to be the case. While running a shell script that loops `make && 
make clean' in cmd.exe as described below, I noticed the logitech 
process LVPrcSrv.exe spike in CPU usage, maxing out one core. When it 
became clear it wasn't going to recover I killed it, and now I can't 
reproduce the as.exe hang. How utterly weird. The as.exe hang is not 
typically accompanied by that CPU usage spike; I've only seen it this 
once. I've re-tested by rebooting and running the as.exe tests under (a) 
rxvt, (b) cmd.exe without runfirst, (c) cmd.exe with runfirst, and all 
hung. After killing that process, I can't get any of them to hang 
anymore. The camera was not used at all.

The services snap-in calls the service associated with LVPrcSrv the 
"Logitech Process Monitor" ... which is interesting and, in light of 
these issues, somewhat scary. It's description is "Webcam effects 
helper". If I start the service again (it having been stopped by my 
killing its process), I find that as.exe begins to intermittently hang 
again. Stop the service, and everything is fine.

Killing LVPrcSrv.exe does not cause hung as.exe processes to resume 
execution, but after it is killed new as.exe processes will not hang.

I don't even want to know what that process is doing such that it can 
interfere so severely with totally unrelated processes running on the 
system. Logitech might want to take a good look at this one.

So ... problem solved partly through luck and partly through lots of 
tedious investigation, and what a weird problem it was. I hope this, and 
the following, will be informative for you folks.




My original discussion follows, so Google can find it and in case it 
helps anybody else:

I've seen this hang once with another process that appeared to be
unrelated to as.exe. It was early on, and I can't remember what it was.
Sorry. I'm pretty sure there was a hung as.exe running at the time,
though, or I'd just killed it.

I'm using an Intel Core Duo 2.0GHz laptop with 2GB of RAM running XP SP2
with current security patches. I have the .NET 2.0 runtime and SDK plus
the Windows Platform SDK installed, in case it matters. The version of
as.exe present is 2.15.91 20040904, with gcc 3.4.2 (mingw-special).

I tried to hook gdb up to as.exe, but gdb also hung indefinitely. It
works fine on other tasks. Connecting with windbg was not especially
informative due to the lack of debugging symbols in as.exe, but I've
attached its best-effort stack traces for a hung as.exe and a hung gdb
trying to attach to as.exe in case they're useful.

I'm inexperienced with Windows debugging, especially binaries without
available symbol tables, but I think the windbg stack trace suggests
that `as.exe' may be deadlocked in exit(). This would explain its lack
of a mingw pid.

Interestingly, setting the CPU affinity on `rxvt' and the shell it
spawns before trying to start my build seems to reduce the frequency
with which as.exe hangs, but does not eliminate the problem. It does not
seem to matter whether I run `as.exe' by hand / directly from make or if
I let gcc or g++ invoke it; it'll hang sometimes either way. Setting the
affinity with `runfirst.exe' when invoking a shell script that loops
'make clean && make' from cmd.exe also seems to dramatically reduce the
frequency of hangs.

I can also reproduce this problem working from cmd.exe. Compiling and
assembling a `hello world' test C file with:

gcc -S -o test.S test.c
as -o test.o test.S

works most of the time, but like when it's run in msys will sometimes
just hang in as.exe . At the time `as' hangs there are *no* other
msys/mingw processes running. Under rxvt+bash, control C doesn't affect
the hang - I have to kill as.exe in process explorer - but under cmd.exe
control C kills as.exe .

It typically takes somewhere in the vein of four to eight invocations of
`as.exe' to get a hang, but this does vary.

In addition to the backtraces from windbg, I grabbed thread backtraces
from Process Explorer. Strangely, they're different. Here's the
backtrace from `as.exe' run under rxvt from make, when hung, as reported
be process explorer:

Thread 12156:
ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
ntdll.dll!RtlEnterCriticalSection+0x46
ntdll.dll!LdrGetDllHandleEx+0x12c
ntdll.dll!LdrGetDllHandle+0x18
!GetModuleHandleW+0x57
!GetModuleHandleW+0x16f
!GetModuleHandleW+0x1f
!GetModuleHandleA+0x1f
!strerror+0x2b29
!initterm+0x111
!exit+0x12
!RegisterWaitForInputIdle+0x49

thread 12168:
ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c56
ntkrnlpa.exe!ZwYieldExecution+0x2538
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
ntdll.dll!RtlEnterCriticalSection+0x46
!strerror+0x2b58
!CoGetComCatalog+0x22bc
!CoGetComCatalog+0x2289
!CoGetComCatalog+0x2275
!CoGetComCatalog+0x21db
!CoTaskMemAlloc+0x91
ntdll.dll!LdrInitializeThunk+0x29
ntdll.dll!LdrFindResourceDirectory_U+0x276
ntdll.dll!RtlValidateUnicodeString+0x506
ntdll.dll!LdrLoadDll+0x110
!LoadLibraryExW+0xc8
!LoadLibraryExA+0x1f
!LoadLibraryA+0x2d

thread 9468:
ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
ntdll.dll!RtlEnterCriticalSection+0x46
ntdll.dll!KiUserApcDispatcher+0x7

Before I start trying to rebuild gcc - with a dodgy compiler - to get
debugging symbols, I'm hoping someone has a .pdb file (is that what's
needed on Windows for symbols?) or a version of as or the compiler suite
with debugging symbols that I can drop into place to investigate this
further. Even better, maybe somebody knows what might be going on, or
what more information I can collect to help.

I've also attached a typical process list for this system while running
normally (process_list.txt).

The test project I've been using, which was created while investigating
the exact asm output by auto_ptr use vs manual allocation/deallocation
in simple classes, is:

Makefile:
=========
ALL_ASM=auto.S manual.S
ALL_DUMP=auto.dump manual.dump
ALL_EXE=auto.exe manual.exe
ALL_OBJ=auto.o manual.o

all: $(ALL_ASM) $(ALL_OBJ) $(ALL_EXE) $(ALL_DUMP)
exe: $(ALL_EXE)
asm: $(ALL_ASM)
obj: $(ALL_OBJ)
dump: $(ALL_DUMP)

%.S: %.cpp
	g++ -S $< -o $@

%.o: %.S
	as $< -o $@

%.exe: %.o
	g++ $< -o $@

%.dump: %.exe
	objdump -d -w -S $< > $@

clean:
	rm -f $(ALL_ASM) $(ALL_DUMP) $(ALL_EXE) $(ALL_OBJ) Makefile~

auto.c:
=======
#include <memory>

class Dummy {
	char buf[200];
};

class Sample {

	std::auto_ptr<Dummy> dummy_ptr;
	
public:
	Sample() : dummy_ptr(new Dummy()) { }
	~Sample() { }
};

int main() {
	Sample sa;
}

manual.cpp
==========
class Dummy {
	char buf[200];
};

class Sample {

	Dummy * const dummy_ptr;

public:
	Sample() : dummy_ptr(new Dummy()) { }
	~Sample() { delete dummy_ptr; }
};

int main() {
	Sample sm;
}

I've also confirmed that a trivial hello world in C, compiled with `gcc'
not g++, will exhibit the problem.

--
Craig Ringer