Menu
▾
▴
[proxyTools-users] Re: [noCensorship] Re: Fed up with LP
|
From: wayne <wa...@ny...> - 2003-05-05 21:43:33
|
> From: "Madani AL" <mad...@ho...> > To: noc...@fr... > Subject: [noCensorship] Re: Fed up with LP > > Hi wayne > > The following are my findings with regards to SBM and Cyberia In KSA: > > 1. Cyberia: > running the latest firewalls.xml under LP, I realized that subnet had to be > increased to 212.119.64.0/19 as a received the following line from the > localProxy Engine (the 'back end'), version: 4.253: > [LP host (212.119.94.*) was not inside firewall KSA-cyberia]. Done. Good work. > Next, I only got the following proxies in layer 0 of 10080:1(connect): > 204.113.91.64:8001 > 212.119.79.2:80 ( I got this one when I extended the subnet range to 19) > 218.145.25.108:8081 > 24.129.32.175:9631 > > I merged the following working proxies into hosts.xml. These proxies > currently are the only active ones for Cyberia (Ogertel): > 212.119.67.14 :8080 PFFFPFPPPPPPPPFPFPFP 1.9/3.2 via:1.0 cache4.jed > agent:CacheFlow-Proxy/1.0 connBack:212.138.64.174 > 212.119.67.15 :80 PFFFPFPPPPPPPPFPFPFP 1.3/2.7 via:1.0 cache4.jed > agent:CacheFlow-Proxy/1.0 connBack:212.138.64.174 > 212.119.67.16 :8080 PFFFPFPPPPPPPPFPFPFP 1.7/3.0 via:1.0 cache4.jed > agent:CacheFlow-Proxy/1.0 connBack:212.138.64.171 > 212.119.67.17 :8080 PFFFPFPPPPPPPPFPFPFP 1.7/3.0 via:1.0 cache4.jed > agent:CacheFlow-Proxy/1.0 connBack:212.138.64.174 > 212.119.67.80 :80 PFFFPFPPPPPPPPFPFPFP 1.8/3.3 via:1.0 cache4.jed > agent:CacheFlow-Proxy/1.0 connBack:212.138.64.172 Four of those are new, and one already known. I've fudged a statProxy result set to merge these results in, and done it twice, to account for the infrequent reports from the KSA. Please ... it takes a bit of work to fake these result sets for mergeHosts. It's much easier (and less error-prone) for me if you include the statProxy header line, even if it's munged. Just leave enough info for me to know which firewall/ISP the tests were done from (the first two, or three, octets will usually do). > The following proxies should be disabled in hosts.xml at least for now: > 212.119.67.18:8080 > 212.119.67.19:8080 > 212.119.67.20:8080 > 212.119.79.2:80 > 212.119.79.66:8080 > 212.119.79.67:8080 > 212.119.82.61:8080 They get lost forever when I do the next cleanHosts if I do that. I much prefer statProxy(Parallel) tests - several, done over a week or two. So now I'm undecided about what to do. I don't want to risk losing them forever, because i so rarely get up to date info from there, and it will probably be a long time before I get test results allowing mergeHosts to add them again. But at the same time, if I do nothing, I lose the info that you have just provided :-( Later - I've compromised. I've fudged a statProxy result set with timeouts on all those connections, and merged it three times. That's enough to disable ones that were on the way out anyway, but not kill completely those I thought were ok up to now. One more test result from there indicating they are dead will disable them. As it's turned out, none got disabled, because the last time I had any good reports from there (probably from you!), they were ok. In the meantime, LP will try not to use them, because they have such low uptime ratio scores. If they are changing often, the we need reports more often too. :-) > Conclusion: Cyberia is working fine with the original > config-KSA-cyberia.xml. > Problem solved. Great. > 2. SBM: Ahh .. the *real* problem. > I noticed that [master.xml] is not consistent whenever I run the Autoconfig > option in LP. That's true. If your network is not consistent, master can't be either. For example, it must wait a full TCP timeout for each connection. If sometimes, a response comes back at <TCP timeout>+1, then master says it's blocked. That timeout is about 21 seconds. Not much I can do to ensure more consistency in an inconsistent network, except possibly to repeat each test 'several' times. Is that worthwhile? Because each test response is nearly at the timeout, the master tests would take approx 'several' times as long. > Examples: > > Mapping the firewall rules, please wait ... > Warning: filtering detected (probably transparent proxy) on port 25 > Warning: filtering detected (probably transparent proxy) on port 110 > Warning: filtering detected (probably transparent proxy) on port 8083 This bothers me. It means that master didn't get the random 'garbage' characters on each connection to these ports at login.icq.com. I've only seen that on port 80 before (where there *is* a transparent proxy). I may be making an unwarranted assumption there. > port 21 is open > port 22 is open > port 23 is open > port 25 is blocked > port 80 is blocked > port 81 is blocked > port 82 is open > port 85 is blocked > port 110 is blocked > port 119 is blocked > port 443 is blocked > port 563 is open > port 666 is open > port 800 is open > port 880 is open > port 1080 is blocked > port 2009 is open > port 3128 is blocked > port 6588 is blocked > port 6801 is open > port 7000 is open > port 7021 is open > port 7033 is open > port 7070 is open > port 7137 is open > port 7175 is open > port 7475 is open > port 8000 is blocked > port 8001 is open > port 8002 is open > port 8003 is open > port 8004 is open > port 8005 is open > port 8006 is open > port 8007 is open > port 8008 is open > port 8080 is blocked > port 8081 is open > port 8082 is open > port 8083 is blocked > port 8084 is open > port 8085 is open > port 8086 is open > port 8087 is open > port 8088 is blocked > port 8089 is blocked > port 8090 is open > port 8091 is open > port 8141 is open > port 8180 is open > port 8421 is open > port 8616 is blocked > port 8888 is blocked > port 8965 is open > port 9001 is open > port 9081 is open > port 9274 is blocked > port 9589 is open > port 10080 is open > port 12345 is open > port 14000 is open > port 20034 is open > Saving the results in master.xml > > Please note that port 8083 is blocked here while it is open in another test: > > Mapping the firewall rules, please wait ... > Warning: filtering detected (probably transparent proxy) on port 25 > Warning: filtering detected (probably transparent proxy) on port 110 > Warning: filtering detected (probably transparent proxy) on port 9589 So, even this test (last time it indicated 8083), is vulnerable to the same things. I see this sort of thing here too, and proxyTools was designed with that knowledge in mind. It's not a problem with proxyTools, but it must be taken account of by anything that uses the results from master/statProxy etc (for example, LP). It's the same with proxy scanners. Most are a *lot* worse than statProxy. It's a compromise between speed and reliability. Master is to be used while the user is waiting, so it compromises on the side of speed (and is unreliable), while statProxy compromises on the side of reliability (and is slow). My latest statProxyParallel makes the assumption that you're testing more than a few proxies, so it can wait as long as it needs for each response (while accepting all the other responses). I think that's a much better approach, though it makes the tests look very slow if you test just one proxy. BTW, I never called statProxy a proxy scanner (it was more of a proxy analyzer), but statProxyParallel combines the best of both worlds and I will call it a scanner now. Very soon (it still doesn't do Socks tests, or tests via CONNECT proxies like statProxy does - soon), it will just replace statProxy in the CVS. > port 21 is open > port 22 is open > port 23 is open > port 25 is blocked > port 80 is blocked > port 81 is blocked > port 82 is open > port 85 is blocked > port 110 is blocked > port 119 is blocked > port 443 is blocked > port 563 is open > port 666 is open > port 800 is open > port 880 is open > port 1080 is blocked > port 2009 is open > port 3128 is blocked > port 6588 is blocked > port 6801 is open > port 7000 is open > port 7021 is open > port 7033 is open > port 7070 is open > port 7137 is open > port 7175 is open > port 7475 is open > port 8000 is blocked > port 8001 is open > port 8002 is open > port 8003 is open > port 8004 is open > port 8005 is open > port 8006 is open > port 8007 is open > port 8008 is open > port 8080 is blocked > port 8081 is open > port 8082 is open > port 8083 is open > port 8084 is open > port 8085 is open > port 8086 is open > port 8087 is open > port 8088 is blocked > port 8089 is blocked > port 8090 is open > port 8091 is open > port 8141 is open > port 8180 is open > port 8421 is open > port 8616 is open > port 8888 is blocked > port 8965 is open > port 9001 is open > port 9081 is open > port 9274 is open > port 9589 is blocked > port 10080 is open > port 12345 is open > port 14000 is open > port 20034 is open > Saving the results in master.xml > > Mapping the firewall rules, please wait ... > Warning: filtering detected (probably transparent proxy) on port 110 > Warning: filtering detected (probably transparent proxy) on port 25 > Warning: filtering detected (probably transparent proxy) on port 8087 > Warning: filtering detected (probably transparent proxy) on port 20034 > Warning: filtering detected (probably transparent proxy) on port 800 > Warning: filtering detected (probably transparent proxy) on port 21 > port 21 is blocked > port 22 is open > port 23 is open > port 25 is blocked > port 80 is blocked > port 81 is blocked > port 82 is open > port 85 is blocked > port 110 is blocked > port 119 is blocked > port 443 is blocked > port 563 is open > port 666 is open > port 800 is blocked > port 880 is open > port 1080 is blocked > port 2009 is open > port 3128 is blocked > port 6588 is blocked > port 6801 is open > port 7000 is open > port 7021 is open > port 7033 is open > port 7070 is open > port 7137 is open > port 7175 is open > port 7475 is open > port 8000 is blocked > port 8001 is open > port 8002 is open > port 8003 is open > port 8004 is open > port 8005 is open > port 8006 is open > port 8007 is open > port 8008 is open > port 8080 is blocked > port 8081 is open > port 8082 is open > port 8083 is open > port 8084 is open > port 8085 is open > port 8086 is open > port 8087 is blocked > port 8088 is blocked > port 8089 is blocked > port 8090 is open > port 8091 is open > port 8141 is open > port 8180 is open > port 8421 is open > port 8616 is open > port 8888 is blocked > port 8965 is open > port 9001 is open > port 9081 is open > port 9274 is open > port 9589 is open > port 10080 is open > port 12345 is open > port 14000 is open > port 20034 is blocked > > I have more of these results. Results are not consistent. It looks like the > timeout value for port testing need to be increased. Sure, that's the issue. The timeout now is 20 seconds for the whole lot. Ok, I've changed it to 40 seconds. > Running LP using SBM fire wall > > This is localProxy Engine (the 'back end'), version: 4.253 > Building configuration KSA-sbm > Please wait, this may take a few minutes > (you may avoid this in future by using the 'last' config)... > I'm assuming this IP address is 212.46.37.* > Waiting for the front end to connect ... connected > Importing globals.xml version 4.12 > Importing hosts.xml version 4.1664 > Importing commStrats.xml version 4.7 > Importing services.xml version 4.23 > Importing firewalls.xml version 4.110 > Importing config-KSA-sbm.xml version 4.1 > No (or incomplete) credentials found in environment > Using firewall: KSA-sbm > DNS addresses from o/s: 212.46.32.35, 212.46.32.45 > Name servers (o/s and firewall info): 212.46.32.35, 212.46.32.45, > 212.46.32.65, 212.46.32.33 > Accessible subnets: 212.46.32.0/19, 212.46.32.0/19 > DnsTimeout initialized: 5.5 > Sorting hosts (uses DNS, please connect)... > Filled layer 10119.1.*.- (1/10), (max,min) score was (0.4, 0.4) > Filled layer 10080.0.*.- (0/10), (max,min) score was (-1, -1) > > Disabling 10080 (non-censoring HTTP proxy - standard).0 - no layer 0 hosts > available > Filled layer 10080.1.*.- (3/10), (max,min) score was (0.4, 0.00032) > Filled layer 10080.1.-.* (10/10), (max,min) score was (1, 0.072) > Filled layer 10080.2.*.- (10/10), (max,min) score was (0.4, 0.0076) > Filled layer 10082.0.*.- (1/10), (max,min) score was (0.052, 0.052) > Filled layer 10082.1.*.- (3/10), (max,min) score was (0.4, 0.00032) > Filled layer 10082.1.-.* (10/10), (max,min) score was (0.18, 0.061) > Filled layer 10082.2.*.- (6/10), (max,min) score was (0.4, 0.00023) > Filled layer 10076.0.*.- (10/10), (max,min) score was (0.4, 0.016) > Filled layer 10076.1.*.- (3/10), (max,min) score was (0.4, 0.00032) > Filled layer 10076.1.-.* (10/10), (max,min) score was (0.6, 0.17) > Filled layer 10076.2.*.- (10/10), (max,min) score was (0.4, 0.016) > Checking all layer 0 hosts for connectivity... > Results from isConnectable: 00101011000000 > Unable to connect to 66.250.69.1:8572 204.113.91.64:8001 168.9.253.251:3347 > 216.126.204.21:8303 216.206.18.12:8002 209.2 > 05.129.2:8851 200.23.144.129:8001 205.205.143.254:8002 206.129.0.18:8281 > 218.145.25.108:8081 > Couldn't connect to 10 proxies > Don't panic, I can probably work ok without them > > LocalProxy is offering services on the following ports: > 10076:non-censoring HTTPS proxy > 10078:HTTP zapped advertisement proxy > 10079:proxy autoconfiguration > 10080:non-censoring HTTP proxy - standard > 10081:localProxy control > 10082:anonymous, non-censoring HTTP proxy > 10119:Usenet news - ccnews > > > For web browser proxy service, please configure your web browser > http proxy entry to localhost:10080 and the > https proxy entry to localhost:10076 now > > Only accepting connections from 127.0.0.1, 212.46.37.* > Executed getConfig(), result (truncated): <perldata> > <hash> > ... > Making test request: GET http://www.mhho.com/Links/Random.html > ---------------------- > The following are the only proxies selected in: 10080,1:connect, layer 0 > 204.113.91.64:8001 > 218.145.25.108:8081 > 216.20.10.43:8616 > 24.129.32.175:9631 > Also, non of SBM proxies were selected in layer 0 of the 2:direct, modified > > Conclusion: This ISP is ?@#$@%#$... Do not know that Is needed to make LP > identify the proxies in the subnet! I found it! I found it! :-) There was a bug which would have affected any of these firewalls which were inaccessable from outside (the 'notPanix' acl thingy). And only those I had already noticed, and tagged as such. I've fixed that, and also made it easier to trace these things. Around line 6086, just uncomment each line and fill in the type of trace you want - basically whatever you know about what you want to trace. I already had this debug code there, but it wasn't as easy to use before. :-) > 3. Open proxies with KSA network: > statProxy v4.156 report from 212.46.37.*(KSA-sbm): > 212.100.203.194 :3128 PFFFFFFPFFFFPFFPFPFP 4.3/38.6 via:1.0 cache10.ruh, > 1.1 d1proxy (NetCache 4.1R6), 1.0 setup (NetCache NetApp/5.2.1) Here we go again. This is an Awalnet proxy. If you did this test from SBM, then I have to assume all Awalnet proxies are accessible from SBM. I asked you about this before, IIRC. This time I see (below) what looks like a full the statProxy result, so I'm making the changes to incorporate the info. That will make a major increase in the number of proxies accessable from SBM. The results of this will swamp the extra proxies found for sbm by fixing the bug above. Please let me know if this report was edited in some way - ok? > agent:BlueCoat-Security-Appliance connBack:212.138.47.11 > 212.100.205.226 :8080 PFFFTRFPFFFFPPFPFPFP 2.8/16.1 via:1.0 cache10.ruh, > 1.0 netcache2 (NetCache NetApp/5.2.1R3), 1.0 SERVER2 > agent:BlueCoat-Security-Appliance connBack:212.138.47.17 > 212.100.207.10 :8080 PFFFFFFPFFFFPPFPFPFP 1.7/3.6 via:1.0 cache8.ruh, 1.0 > netcache1 (NetCache NetApp/5.3.1R2), 1.0 LINDORAPC > agent:BlueCoat-Security-Appliance connBack:212.138.47.11 > 212.100.207.22 :3128 PFFFFFFPFFFFPPFPFPFP 1.7/3.3 via:1.0 cache10.ruh, 1.0 > netcache1 (NetCache NetApp/5.3.1R2), 1.0 USTSRV > agent:BlueCoat-Security-Appliance connBack:212.138.47.12 > 212.102.11.254 :8080 PFFFFFFPFFFFFFFPFFFP 2.1/4.1 via:1.0 cache4.ruh, 1.0 > New-Proxy2 (NetCache NetApp/5.3.1R2) agent:NetCache NetApp/5.3.1R2D5 OMG! a whole new isp (Shabakah) at 212.102.0.0/19. Added to firewalls.xml, but I have no idea what ISPs can access this subnet (except for sbm/212.46.37.* of course). > connBack:212.26.70.40 > 212.102.12.180 :3128 PFFFFFFPFFFFFFFPFFFP 2.4/5.1 via:1.0 New-proxy3 > (NetCache NetApp/5.3.1R2D5) agent:NetCache NetApp/5.4R2 > connBack:212.138.47.14 > 212.102.2.122 :8080 PFFFFFFPPFFFFPFPFFFP 3.9/13.8 via:1.0 New-proxy3 > (NetCache NetApp/5.3.1R2D5) agent:NetCache NetApp/5.3.1R2D5 > connBack:212.26.70.41 > 212.102.2.229 :80 PFFFFFFPPFFFFPFPFPFP 2.3/4.7 via:1.0 new-proxy1 > (NetCache NetApp/5.4R2) agent:NetCache NetApp/5.4R2 connBack:212.26.70.40 > 212.102.3.66 :80 PFFFFFFPFFFFPPFPFPFP 2.3/5.4 via:1.0 New-proxy3 > (NetCache NetApp/5.3.1R2D5) agent:NetCache NetApp/5.3.1R2D5 > connBack:212.26.70.41 > 212.107.101.146 :80 PFFFFFFPFFFFPPFPFPFP 3.9/9.0 via:1.0 cache1.jed, 1.0 > SAUDISTEELBDC agent:BlueCoat-Security-Appliance connBack:212.119.67.17 A new subnet for Trinet. That means also for cyberia? I've added these too. > 212.107.97.2 :8080 PFFFFFFPFFFFFFFPFPFP 2.4/5.1 via:1.0 cache3.jed > agent:BlueCoat-Security-Appliance connBack:212.119.67.17 > 212.107.99.130 :8080 PFFFPFPPPPPPPPFPFPFP 3.7/13.9 via:1.0 cache4.jed > agent:BlueCoat-Security-Appliance connBack:212.119.67.17 > 212.11.177.235 :8080 PFFFFFPPFPFPFPFPFPFP 1.9/8.9 via:1.0 cache4.ruh, 1.0 > cache-01 (NetCache NetApp/5.3R2) agent:BlueCoat-Security-Appliance More subnets for sps too. Done. > connBack:212.11.191.16 > 212.116.194.195 :8080 PFFFPFPPPPPPPPFPFPFP 1.7/3.2 via:1.0 cache7.ruh, 1.0 > netcache3 (NetCache NetApp/5.2.1R3) agent:BlueCoat-Security-Appliance > connBack:212.138.47.11 More subnets for awalnet > 212.33.168.229 :80 PFFFPFPPPPPPPPFPFPFP 4.7/7.0 via:1.0 cache10.ruh, 1.0 > C3100-0033602110 (NetCache NetApp/5.2.1R1D4), 1.0 TMISERVER > agent:BlueCoat-Security-Appliance connBack:212.33.160.3 sol ... > 212.62.100.205 :8080 PFFFFFFPFFFTPPFPFPFP 3.9/14.6 via:1.0 cache1.jed, 1.0 > NetCache3100 (NetCache NetApp/5.3.1R2), 1.0 IBM iccnet - another new one for firewalls.xml > agent:BlueCoat-Security-Appliance connBack:212.138.64.174 > 212.76.68.49 :8080 PFFFPFPPPPPPPPFPFPFP 3.0/7.9 via:1.0 cache7.ruh > agent:BlueCoat-Security-Appliance connBack:212.76.68.204 > 212.76.70.39 :8080 PFFFFFFPFFFFPPFPFPFP 9.8/30.4 via:1.0 cache1.ruh, 1.0 > SERVER agent:BlueCoat-Security-Appliance connBack:212.76.68.203 > 212.76.75.103 :8080 PFFFPFFPPPPPFFFPFPFP 3.0/4.5 via:1.0 cache1.ruh > agent:BlueCoat-Security-Appliance connBack:212.76.68.203 > 212.76.75.50 :80 PFFFPFFPPPPPPPFPFPFP 2.2/3.7 via:1.0 cache10.ruh > agent:BlueCoat-Security-Appliance connBack:212.76.68.203 > 212.76.76.11 :8080 PFFFPFPPPPPPFPFPFPFP 2.3/4.8 via:1.0 cache9.ruh > agent:BlueCoat-Security-Appliance connBack:212.76.68.204 > 212.76.76.16 :80 PFFFPFPPPPPPPPFPFPFP 2.0/4.2 via:1.0 cache9.ruh > agent:BlueCoat-Security-Appliance connBack:212.76.68.203 > 212.76.76.42 :8080 PFFFFFFPFFFFPPFPFPFP 2.2/3.9 via:1.0 cache1.ruh, 1.0 > SERVER agent:BlueCoat-Security-Appliance connBack:212.76.68.203 212.62.99.52 > :8080 PFFFFFFPFFFFPPFPFPFP 1.6/5.6 via:1.0 cache4.ruh, 1.0 NetCache3100 > (NetCache NetApp/5.3R2D4), 1.0 SALAH agent:BlueCoat-Security-Appliance > connBack:212.138.47.14 sahara ... > 213.184.160.152 :80 PFFFFFFPFFFFPPFPFPFP 1.8/24.2 via:1.0 cache8.ruh, 1.0 > netcache2 (NetCache NetApp/5.2.1R3), 1.0 MISHWAR_MAIL > agent:BlueCoat-Security-Appliance connBack:212.138.47.20 > 213.184.164.63 :80 PFFFFFFPFFFFPPFPFPFP 7.9/12.1 via:1.0 cache1.ruh, 1.0 > netcache4 (NetCache NetApp/5.3.1R2) agent:BlueCoat-Security-Appliance > connBack:212.138.47.29 awalnet. > Please NOTE: the above open proxies are not fixed/stable. Today are open > tomorrow might be closed. The ISPs are switching their proxies very often That's why LP is the way it is. That doesn't matter to it. But whole subnets/ISPs changing certainly affects it. If one ISP allows another ISPs users to access it's proxies, for example, then LP's build should change quite dramatically. > Best regards > madani I've made lotsa changes based on your info; all good, I think. Please ignore extra warnings from mergeHosts for the moment, I'm still evaluating a new routine (isAccessible). Please tell me about any problems you see - I'm sure I've broken something as well :-( Good info, as usual. Thanks. -- wa...@ny... http://proxytools.sourceforge.net/ |
