On various occasions my system has been flooded with 15-20 or more svn.exe processes. They continue to use up memory and CPU, and after I kill the processes they reopen, forcing me to reboot each time this happens.
Could this be due to stopping StatSVN during the generation of the report? I'm not sure if StatSVN uses a seperate process for each thread (seperate difficulties prevent me from debugging any further currently), but if that is the case, would ending StatSVN leave the SVN processes active and unable to be terminated?
If it makes a difference, I'm using a Windows Service to call StatSVN, so when I refer to stopping StatSVN, I actually mean stopping the service. I will add the required measures to prevent this when the service is stopped, but I wanted to know if that was the cause of the problem and let you know since I haven't been able to find anything about it on the forums.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've never heard of someone having such problems. However, what you describe sounds like the calls to svn diff are freezing, for some reason. StatSVN launches multiple svn diff in parallel to speed up the query times.
If you run StatSVN at the command line and specify the -debug option, do you see the actual svn diff commands that are executed? I am guessing this problem is intermittent and that it usually works, but randomly stops. I can't see why an svn diff would do that other than to prompt for authentication, but we pass those parameters along when executing the process.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
With the debug option, I Do see all the svn diff calls being executed. However, I have so far been unable to catch the problem occurring and capture the output from it.
You're correct, this has only happened 5 or 6 times over a couple weeks. When I have noticed it occur, I have rebooted right away. However, my logs show that my scheduled runs have been timing out after 10 hours, when the run usually takes less than 1. This happened several times in a row, so it's possible that either it is re-occurring every time now, or it happened once and the processes remained after my service timed out and shut it down.
It's very strange that this problem hasn't been brought up before. It has now occurred for me on my laptop and my development server. If I can do anything to help narrow down the issue, please let me know.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would recommend loading up http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx (Process Explorer) and seeing what command line arguments are given to the frozen SVN executables. Try to run that command directly from the command line and see what happens.
I would not be surprised that your server is blocking you because you are using too many simultaneous connections on their server. Try playing with the various StatSVN thread parameters (see the user manual) to see if you can prevent being blocked out.
Thanks,
Jason
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Okay, I managed to recreate the issue while watching the processes with Process Explorer. The problem does indeed occur when my service is stopped (manually or otherwise unexpectedly), after which the svn processes remain open. I'm still not sure why this is happening, but I'll probably just add some code to my service to close all the processes before stopping.
Unfortunately, now I have ANOTHER issue. I have reverted to various revisions of my service, most of which I know were functioning in the past, and none of them will work now.
Process Explorer shows the service successfully call SVN to create the logfile, then the java call appears with the multiple SVN threads. Once the SVN processes are closed, the java process hangs. (Using 0% CPU and only 25MB of memory.)
As far as I can tell, nothing else about my setup has changed. I've tried cleaning up, updating, and checking out clean copies of the trunk being used. Running the identical process from the command line is successful.
NOTE: After I wrote this, I checked and noticed that the statistics report was in fact successfully generated. Does StatSVN do anything after generating the report that could cause the java process to remain open?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> Once the SVN processes are closed, the java process hangs. (Using 0% CPU and only 25MB of memory.)
It probably expects them to return but since they never do, hangs. I'll forward this note to benoitx for code review.
> Does StatSVN do anything after generating the report that could cause the java process to remain open?
It does not.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
jjcole, debugging this is much easier with a full stacktrace to see which thread(s) are hung, preventing the JVM from exiting. If you're in this state now, open a prompt and do this:
$ jps
<jps output>
You'll see a list of running java processes and their PIDs. Get the PID of the StatSVN one, say it's 5555.
$ jstack 5555
2009-06-16 14:10:55
Full thread dump Java HotSpot(TM) Client VM (14.0-b16 mixed mode, sharing):
"Worker-70" prio=6 tid=0x07910400 nid=0x1648 in Object.wait() [0x0a08f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
[...]
That'll give benoit more info :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Killing a the SVN process that was launched by StatSVN is probably not good... but I do not know how java handles this, it is simply doing the usual:
final Process lastProcess = Runtime.getRuntime().exec(sCommand, null, getWorkingFolder());
This creates 2 stream: the result and the error... I do not know how it would react to the process being killed...
The debug information should show you which command it is launching during the run "FIRING command line:...."
with the thread name.
In doubt I would reduce the number of thread to 1 and see what is happening... Why is your SVN server not responding quickly? Is it an internal server? Are you sure that the firewall is not blocking you? You could have generated a lot of queries and been flagged as a DoS (It happened to me with Apache... once again, sorry guys!)
I'll need to do more investigation but it is going to take a while...
B
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Okay, small update: using 0.5.0 the report generates and then the java process hangs. Using the last version (0.4.1?) the svn diff calls complete, and then the java process hangs before generating the report. I'll post a stacktrace in a minute.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
While I've been able to avoid the multiple SVN processes, I'm still getting the other problems mentioned above. Running the exact same comands in the command prompt works fine, but using my service (running as a C# Process object) the java process will hang and not complete or terminate.
If I terminate the process and view the output, it stops immediately after an SVN diff call. Occasionally, it will reach the final diff call, but it never gets far enough to write any files. (Except the repository cache file.)
I believe this is version 0.4.1 I'm using, but I have had similar problems with the latest version in the past. I will try it again with the new version just to be safe.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Okay, interesting update. I used the latest version of StatSVN and the same problem occurs, the java process hangs as usual. However, I noticed this time that despite having no child SVN processes at the time, when I stopped the service the java process immediately opened the SVN processes. Killing java left the SVN processes orphaned, but they continued to work until completing and terminating.
Is there any reason why the service I'm using to call StatSVN with would prevent java from functioning correctly?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On various occasions my system has been flooded with 15-20 or more svn.exe processes. They continue to use up memory and CPU, and after I kill the processes they reopen, forcing me to reboot each time this happens.
Could this be due to stopping StatSVN during the generation of the report? I'm not sure if StatSVN uses a seperate process for each thread (seperate difficulties prevent me from debugging any further currently), but if that is the case, would ending StatSVN leave the SVN processes active and unable to be terminated?
If it makes a difference, I'm using a Windows Service to call StatSVN, so when I refer to stopping StatSVN, I actually mean stopping the service. I will add the required measures to prevent this when the service is stopped, but I wanted to know if that was the cause of the problem and let you know since I haven't been able to find anything about it on the forums.
I've never heard of someone having such problems. However, what you describe sounds like the calls to svn diff are freezing, for some reason. StatSVN launches multiple svn diff in parallel to speed up the query times.
If you run StatSVN at the command line and specify the -debug option, do you see the actual svn diff commands that are executed? I am guessing this problem is intermittent and that it usually works, but randomly stops. I can't see why an svn diff would do that other than to prompt for authentication, but we pass those parameters along when executing the process.
With the debug option, I Do see all the svn diff calls being executed. However, I have so far been unable to catch the problem occurring and capture the output from it.
You're correct, this has only happened 5 or 6 times over a couple weeks. When I have noticed it occur, I have rebooted right away. However, my logs show that my scheduled runs have been timing out after 10 hours, when the run usually takes less than 1. This happened several times in a row, so it's possible that either it is re-occurring every time now, or it happened once and the processes remained after my service timed out and shut it down.
It's very strange that this problem hasn't been brought up before. It has now occurred for me on my laptop and my development server. If I can do anything to help narrow down the issue, please let me know.
I would recommend loading up http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx (Process Explorer) and seeing what command line arguments are given to the frozen SVN executables. Try to run that command directly from the command line and see what happens.
I would not be surprised that your server is blocking you because you are using too many simultaneous connections on their server. Try playing with the various StatSVN thread parameters (see the user manual) to see if you can prevent being blocked out.
Thanks,
Jason
Okay, I managed to recreate the issue while watching the processes with Process Explorer. The problem does indeed occur when my service is stopped (manually or otherwise unexpectedly), after which the svn processes remain open. I'm still not sure why this is happening, but I'll probably just add some code to my service to close all the processes before stopping.
Unfortunately, now I have ANOTHER issue. I have reverted to various revisions of my service, most of which I know were functioning in the past, and none of them will work now.
Process Explorer shows the service successfully call SVN to create the logfile, then the java call appears with the multiple SVN threads. Once the SVN processes are closed, the java process hangs. (Using 0% CPU and only 25MB of memory.)
As far as I can tell, nothing else about my setup has changed. I've tried cleaning up, updating, and checking out clean copies of the trunk being used. Running the identical process from the command line is successful.
NOTE: After I wrote this, I checked and noticed that the statistics report was in fact successfully generated. Does StatSVN do anything after generating the report that could cause the java process to remain open?
> Once the SVN processes are closed, the java process hangs. (Using 0% CPU and only 25MB of memory.)
It probably expects them to return but since they never do, hangs. I'll forward this note to benoitx for code review.
> Does StatSVN do anything after generating the report that could cause the java process to remain open?
It does not.
jjcole, debugging this is much easier with a full stacktrace to see which thread(s) are hung, preventing the JVM from exiting. If you're in this state now, open a prompt and do this:
$ jps
<jps output>
You'll see a list of running java processes and their PIDs. Get the PID of the StatSVN one, say it's 5555.
$ jstack 5555
2009-06-16 14:10:55
Full thread dump Java HotSpot(TM) Client VM (14.0-b16 mixed mode, sharing):
"Worker-70" prio=6 tid=0x07910400 nid=0x1648 in Object.wait() [0x0a08f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
[...]
That'll give benoit more info :)
Indeed, the stacktraces would help.
Killing a the SVN process that was launched by StatSVN is probably not good... but I do not know how java handles this, it is simply doing the usual:
final Process lastProcess = Runtime.getRuntime().exec(sCommand, null, getWorkingFolder());
This creates 2 stream: the result and the error... I do not know how it would react to the process being killed...
The debug information should show you which command it is launching during the run "FIRING command line:...."
with the thread name.
In doubt I would reduce the number of thread to 1 and see what is happening... Why is your SVN server not responding quickly? Is it an internal server? Are you sure that the firewall is not blocking you? You could have generated a lot of queries and been flagged as a DoS (It happened to me with Apache... once again, sorry guys!)
I'll need to do more investigation but it is going to take a while...
B
Okay, small update: using 0.5.0 the report generates and then the java process hangs. Using the last version (0.4.1?) the svn diff calls complete, and then the java process hangs before generating the report. I'll post a stacktrace in a minute.
I don't have the JDK so I'll have to install that first. Also, here's a part of the standard error output of the java process after I terminate it:
Jun 16, 2009 2:14:33 PM net.sf.statsvn.util.JavaUtilTaskLogger info
INFO: StatSVN - SVN statistics generation
Parsing SVN log 'logfile.log' exclude pattern 'null'
svn diff 1/22 on r4599 (1645 ms.) main
EmployeeImport.csproj, on r4599, +10 -6
EmployeeImport/CsvEmployee.cs, on r4599, +0 -27
EmployeeImport/PepsiCoEmployeeRoutePlan.cs, on r4599, +14 -0
EmployeeImport/PepsiCoEmployeeRoutePlanParsedStatus.cs, on r4599, +10 -0
EmployeeImport/EmployeeImporter.cs, on r4599, +0 -2
EmployeeImport/Program.cs, on r4599, +233 -104
EmployeeImport/PepsiCoEmployeeRoutePlanParser.cs, on r4599, +90 -0
EmployeeImport/ImportTest.cs, on r4599, +172 -168
EmployeeImport/PepsiCoEmployeeRoutePlanParsedEventArgs.cs, on r4599, +12 -0
svn diff 2/22 on r6088 (1092 ms.) main
EmployeeImport/PepsiCoEmployeeRoutePlanParser.cs, on r6088, +4 -2
EmployeeImport/ImportTest.cs, on r6088, +12 -12
The output lists all of the required svn diff calls for this trunk, so it's stopping just after it finishes those calls.
Running jstack gives me: Not enough storage is available to process this command. I have plenty of memory so I'm not sure why it fails.
I have to move on to something else now, I'll try to provide more information when I come back to this.
While I've been able to avoid the multiple SVN processes, I'm still getting the other problems mentioned above. Running the exact same comands in the command prompt works fine, but using my service (running as a C# Process object) the java process will hang and not complete or terminate.
If I terminate the process and view the output, it stops immediately after an SVN diff call. Occasionally, it will reach the final diff call, but it never gets far enough to write any files. (Except the repository cache file.)
I believe this is version 0.4.1 I'm using, but I have had similar problems with the latest version in the past. I will try it again with the new version just to be safe.
Okay, interesting update. I used the latest version of StatSVN and the same problem occurs, the java process hangs as usual. However, I noticed this time that despite having no child SVN processes at the time, when I stopped the service the java process immediately opened the SVN processes. Killing java left the SVN processes orphaned, but they continued to work until completing and terminating.
Is there any reason why the service I'm using to call StatSVN with would prevent java from functioning correctly?