Hi guys, short story how I solved CNTLM freezing problem, when it's used in gateway mode.
Our company has shared connection with company we are part of. When they ugraded older ISA to Forefront 2010 I've decided to change settings to centralized CNTLM as gateway.
Everything was ok when I tried it on my computer, and sadly I found more restriction on the new proxy. Using cntlm as gateway is great (one centralized settings, faster browsing,…), but after I set it for whole company (only 9active users), cntl became unresponsive and it used higher CPU and RAM than usual. This happend on version 0.92.1 and server is Windows Srv 2003 R2 SP2. So I upgraded cntlm to latest 0.93 beta 5. It was much better but after longer time happend the same. Even "restart service" settings in service manager doesn't help because service was probably running correctly (for the service manager, described bellow), but stays unresponsive.
So I was searching logs what happend and I found facebook and twitter domains in it (https wasn't the freezing rule even it seems so at the begining). Fb. and Tw. are blocked by Forefront proxy. "Blocking" out *facebook* and twitter servers with rule NoProxy in ctlm.ini helped and browsing was smoother, but then it happend on google maps. So what to do now?
After some time thinking I realize that service is running under cygwin enviroment. Cygwin environment has no problems only one app inside is unresponsive and this is a problem when it is running as service. Service manager doesn't need to restart service without problems. --This is my thought, guys, correct me if I'm wrong --
Ok, how to solve this problem? How to restart it when it's needed? It is easy solution on Linux - script checking and restarting, harder on Windows - the same but .. you know Windows…
Then I remember on NSSM - alternative service manager (url: nssm.cc ). Long story short, service instaled as cntlm.exe with options: -f -c /proxy/cntlm.ini . Param "f" foreground is needed because fork to background process is probably translated as crash by nssm. Parameter for path is the same as "c:\proxy\cntlm.ini" . It helped a lot! No freezing now and service is restarted in case of problem immediately, because service manager can monitor cntlm activity.
End of story. I had no problems runnning CNTLM on GNU/Linux, it seems related to cygwin env. I'm sharing this because it can safe your time as it saved mine. Sorry for my English grammer :)
Note for CNTLM developer: Pokud by bylo potřeba vysvětlit česky, protože tomu nahoře nebylo rozumět, dejte vědět :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Update - It was much better for a while but after some time it got stucked again (on https connections usually). It's solved by script which is restarting service each 5mins now (awfull hack I know).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2013-02-26
Hmm. I found this behaviour too. Started as a service cntlm will run for a while, then stall. Sometimes it will take minutes or even hours to load one page. Killing then restarting the thing does solve the problem - for a short while …
I wanted to find out what was going on. Killed the service, started cntlm in a cygwin-bash-shell-window using "/cygdrive/c/Programme/cntlm/cntlm.exe -f -c /cygdrive/c/Programme/cntlm/cntlm.ini" -- after some time I thought it shalk stall and I would be able to find out what went wrong with cntlm. But: even after hours the service ran. fast, responsible, reliable. Merde! This way I could not find why it doesn't run this way as windows service!
Second test, this time I started cntlm in a cmd.exe window. But -- same as above! Even after hours no stall, not even a slow down! Responsive, fast, reliable.
But as soon as I started it again as service via cygwinsrv.exe the same: after a few requests the service went unresponsive and stalled.
The only real difference between these where the environment cntlm ran in: cygwin-bash and cmd.exe had nearly the same (cause I like to use cygwin commands in cmd.exe-scripts). But started as service the environment is cut down to something very simple. Parts of the path are missing, not all variables defined. I took some additional time to start a simple cmd.exe shell as a service and then started cntlm within, awaiting it to stall and get unresponsive as if it were directly started -- no success. Again it ran: reliable, fast, stable.
So again: where is the difference? /dev/stdin is redirected from /dev/null, /dev/stdout, as is /dev/stderr are redirected to /var/log/cntlm.log. As stated in the manpage to cygrunsrv. But is it true? I can enter characters into the open cmd.exe window. this *is* interactive and no redirects are notable. Within such an environment cntlm runs rock solid. But in an redirected environment?? Next test: bash shell. Then I issued the command: "/cygdrive/c/Programme/cntlm/cntlm.exe -f -c /cygdrive/c/Programme/cntlm/cntlm.ini </dev/null >/var/log/cntlm.log 2>&1". No change: cntlm was responsive, fast, reliable.
Could be a test case for cygrunsrv.exe. Started via this piece of software cntlm stalls. Started from a shell it doesn't. Now I am a bit clueless about what really is going on there …!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2013-02-26
After some additional tests I found cntlm.exe running stable, reliable and fast if installed with:
The problem seems to come from stderr and stdout. They are not redirected as described and cntlm tries to write to them. If the buffer associated with them is full it stalls. Redirecting them to files solves this problem, but adds an other one in: service.exe waits on some output from the service. This never happens now, because stderr and stdout are redirected. cygrunsrv can take this part if it is advised to watch for a "pid"-File. cntlm creates one, if given the "-P" option followed by a filename. In tune the "-f"-option making cntlm stay in foreground has to be removed.
Now all are happy: service.exe is informed about the successful service start, cntlm has its /dev/stdout and /dev/stderr, cygrunsrv finds the "pid"-File to stop the service … and last, not least me, because the whole thing runs as expected: stable, reliable, fast -- without any crude hacks restarting the service every five minutes … ;-)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Man, this is an awesome work and answer! Thank you! I'll try it when I have some free time, and I also hope this could help somebody else!
Btw, my story had some continue… I configured Squid proxy and made another way out from the network excluding official, but unusable corporate proxy :) Cntlm is restarted each 30mins now and quite stable for its purposes (for updates only).
Thanx again for sharing your solution!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Having seen this last week I reconfigured our (dormant, due to constant crashes and slowdowns) installed cntlm and fired it up in place of our trundling ntlm-aps process.
Since then we've expreienced no crashes or slowdowns using cntlm in gateway mode with about 20 people sharing it.
Thanks for sharing … top marks!!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi guys, short story how I solved CNTLM freezing problem, when it's used in gateway mode.
Our company has shared connection with company we are part of. When they ugraded older ISA to Forefront 2010 I've decided to change settings to centralized CNTLM as gateway.
Everything was ok when I tried it on my computer, and sadly I found more restriction on the new proxy. Using cntlm as gateway is great (one centralized settings, faster browsing,…), but after I set it for whole company (only 9active users), cntl became unresponsive and it used higher CPU and RAM than usual. This happend on version 0.92.1 and server is Windows Srv 2003 R2 SP2. So I upgraded cntlm to latest 0.93 beta 5. It was much better but after longer time happend the same. Even "restart service" settings in service manager doesn't help because service was probably running correctly (for the service manager, described bellow), but stays unresponsive.
So I was searching logs what happend and I found facebook and twitter domains in it (https wasn't the freezing rule even it seems so at the begining). Fb. and Tw. are blocked by Forefront proxy. "Blocking" out *facebook* and twitter servers with rule NoProxy in ctlm.ini helped and browsing was smoother, but then it happend on google maps. So what to do now?
After some time thinking I realize that service is running under cygwin enviroment. Cygwin environment has no problems only one app inside is unresponsive and this is a problem when it is running as service. Service manager doesn't need to restart service without problems. --This is my thought, guys, correct me if I'm wrong --
Ok, how to solve this problem? How to restart it when it's needed? It is easy solution on Linux - script checking and restarting, harder on Windows - the same but .. you know Windows…
Then I remember on NSSM - alternative service manager (url: nssm.cc ). Long story short, service instaled as cntlm.exe with options: -f -c /proxy/cntlm.ini . Param "f" foreground is needed because fork to background process is probably translated as crash by nssm. Parameter for path is the same as "c:\proxy\cntlm.ini" . It helped a lot! No freezing now and service is restarted in case of problem immediately, because service manager can monitor cntlm activity.
End of story. I had no problems runnning CNTLM on GNU/Linux, it seems related to cygwin env. I'm sharing this because it can safe your time as it saved mine. Sorry for my English grammer :)
Note for CNTLM developer: Pokud by bylo potřeba vysvětlit česky, protože tomu nahoře nebylo rozumět, dejte vědět :)
Update - It was much better for a while but after some time it got stucked again (on https connections usually). It's solved by script which is restarting service each 5mins now (awfull hack I know).
Hmm. I found this behaviour too. Started as a service cntlm will run for a while, then stall. Sometimes it will take minutes or even hours to load one page. Killing then restarting the thing does solve the problem - for a short while …
I wanted to find out what was going on. Killed the service, started cntlm in a cygwin-bash-shell-window using "/cygdrive/c/Programme/cntlm/cntlm.exe -f -c /cygdrive/c/Programme/cntlm/cntlm.ini" -- after some time I thought it shalk stall and I would be able to find out what went wrong with cntlm. But: even after hours the service ran. fast, responsible, reliable. Merde! This way I could not find why it doesn't run this way as windows service!
Second test, this time I started cntlm in a cmd.exe window. But -- same as above! Even after hours no stall, not even a slow down! Responsive, fast, reliable.
But as soon as I started it again as service via cygwinsrv.exe the same: after a few requests the service went unresponsive and stalled.
The only real difference between these where the environment cntlm ran in: cygwin-bash and cmd.exe had nearly the same (cause I like to use cygwin commands in cmd.exe-scripts). But started as service the environment is cut down to something very simple. Parts of the path are missing, not all variables defined. I took some additional time to start a simple cmd.exe shell as a service and then started cntlm within, awaiting it to stall and get unresponsive as if it were directly started -- no success. Again it ran: reliable, fast, stable.
So again: where is the difference? /dev/stdin is redirected from /dev/null, /dev/stdout, as is /dev/stderr are redirected to /var/log/cntlm.log. As stated in the manpage to cygrunsrv. But is it true? I can enter characters into the open cmd.exe window. this *is* interactive and no redirects are notable. Within such an environment cntlm runs rock solid. But in an redirected environment?? Next test: bash shell. Then I issued the command: "/cygdrive/c/Programme/cntlm/cntlm.exe -f -c /cygdrive/c/Programme/cntlm/cntlm.ini </dev/null >/var/log/cntlm.log 2>&1". No change: cntlm was responsive, fast, reliable.
Could be a test case for cygrunsrv.exe. Started via this piece of software cntlm stalls. Started from a shell it doesn't. Now I am a bit clueless about what really is going on there …!
After some additional tests I found cntlm.exe running stable, reliable and fast if installed with:
The problem seems to come from stderr and stdout. They are not redirected as described and cntlm tries to write to them. If the buffer associated with them is full it stalls. Redirecting them to files solves this problem, but adds an other one in: service.exe waits on some output from the service. This never happens now, because stderr and stdout are redirected. cygrunsrv can take this part if it is advised to watch for a "pid"-File. cntlm creates one, if given the "-P" option followed by a filename. In tune the "-f"-option making cntlm stay in foreground has to be removed.
Now all are happy: service.exe is informed about the successful service start, cntlm has its /dev/stdout and /dev/stderr, cygrunsrv finds the "pid"-File to stop the service … and last, not least me, because the whole thing runs as expected: stable, reliable, fast -- without any crude hacks restarting the service every five minutes … ;-)
Hello,
When i try to run this code piece on CMD i get the following error. I feel like this code piece is for Linux rather than Windows am i right ?
C:\Program Files (x86)\Cntlm>cygrunsrv -I "Cntlm" -a "-P /var/log/cntlm.pid -c
C:\Program Files (x86)\Cntlm/cntlm.ini" \ -d "Cntlm Authentication Proxy" \ -f
"HTTP Accelerator" \ -0 /dev/null \ -1 /var/log/cntlm.log \ -2 /var/log/cntl
m.err \ -x /var/log/cntlm.pid \ -p C:\Program Files (x86)\Cntlm/cntlm.exe \ -
c C:\Program Files (x86)\Cntlm
cygrunsrv: Trailing commandline arguments not allowed
Try `cygrunsrv --help' for more information.
Man, this is an awesome work and answer! Thank you! I'll try it when I have some free time, and I also hope this could help somebody else!
Btw, my story had some continue… I configured Squid proxy and made another way out from the network excluding official, but unusable corporate proxy :) Cntlm is restarted each 30mins now and quite stable for its purposes (for updates only).
Thanx again for sharing your solution!
This is excellent work.
Having seen this last week I reconfigured our (dormant, due to constant crashes and slowdowns) installed cntlm and fired it up in place of our trundling ntlm-aps process.
Since then we've expreienced no crashes or slowdowns using cntlm in gateway mode with about 20 people sharing it.
Thanks for sharing … top marks!!