Web page monitoring

[Ronin]

Hi Paul;

First, I would like to thank you for your help with the ET rules for Snort it worked great!!!!

I do have two questions.

1. Is it possible to monitor web surfing activities using NST.  I would be insterested in which users went to which sites and for how long.  I saw that squid was included in NST but i don't think it was configured in this way.

2. Is it possible to use Snort inline as an IPS with the NST configuration?

Thank you in advance

Patrick

[Paul Blankenbaker]

Hi Patrick:

I'm glad to hear that the ET rules for Snort are working for you.

1. As far as using squid to monitor web traffic:

It is easy to start the squid (proxy) server on your NST system, run the command:

service squid start

If you want squid started whenever the system reboots, also run the command:

chkconfig squid on

By default, the proxy server will start up accepting connections on port 3128. Users will not be required to login to use the proxy server. You should be able to try it out by configuring a web browser on your network to use the NST as its proxy server (you'll need to know the IP address of the NST system).

Any request which goes through squid should create a log entry in the file: /var/log/squid/access.log. This will include time information, the machine that made the request and the web site which it went to. You can then use a "squid log analyzer" to process the results of the log file into a more human readable form.

Here's an example of using the calamaris analyzer:

yum install calamaris
cat /var/log/squid/access.log | calamaris -R -1 | less

The output will be long and show statistics about the domains (google.com, cnn.com, …) visited by each system using the proxy server. Here's a portion of the report after having two systems connected to the proxy server (tortilla-e.linux.bogus and cayenne-e.linux.bogus) for a short time and hitting a couple of web sites:

# Incoming TCP-requests by host
host / target                      request   hit-%   Byte    hit-% sec   kB/sec 
--------------------------------- --------- ------ -------- ------ ---- ------- 
tortilla-e.linux.bogus                  306   1.96  2487559   0.20    0   44.77 
 *.turner.com                           104   0.00  1237071   0.00    0   81.24 
 *.google.com                            78   7.69   447276   1.13    0   56.92 
 *.cnn.com                               27   0.00   193357   0.00    0   29.58 
 *.chitika.net                           12   0.00    53060   0.00    0   15.83 
 *.squid-cache.org                       11   0.00    64930   0.00    0   39.70 
 *.doubleclick.net                       11   0.00    49710   0.00    0   16.45 
 aplawrence.com                           9   0.00   187768   0.00    0  101.08 
 *.googleusercontent.com                  6   0.00    49518   0.00    0   26.48 
 *.atdmt.com                              6   0.00    49595   0.00    0   23.41 
 *.googlesyndication.com                  5   0.00    22567   0.00    0   49.30 
 *.adnxs.com                              4   0.00     8446   0.00    0   11.36 
 *.mybloglog.com                          3   0.00     4329   0.00    0    7.76 
 *.intermundomedia.com                    2   0.00    30829   0.00    0   41.13 
 *.adtechus.com                           2   0.00     1617   0.00    0    2.75 
 *.revsci.net                             2   0.00     7836   0.00    0   11.92 
 *.facebook.com                           2   0.00      888   0.00    0    1.35 
 *.mozilla.com                            2   0.00     1388   0.00    0    4.12 
 *.quantserve.com                         2   0.00     3048   0.00    0    3.58 
 *.2mdn.net                               2   0.00    11718   0.00    0   35.87 
 *.blogspot.com                           2   0.00     7325   0.00    0   21.35 
 *.google-analytics.com                   1   0.00      452   0.00    0    2.60 
 <error>                                  1   0.00      555   0.00    0    6.23 
 *.facebook.net                           1   0.00      364   0.00    0    0.77 
 *.images-amazon.com                      1   0.00    29225   0.00    0   81.08 
 *.scorecardresearch.com                  1   0.00      620   0.00    0    1.48 
 *.yahoo.com                              1   0.00      504   0.00    1    0.42 
 *.dl-rms.com                             1   0.00      398   0.00    1    0.73 
 *.bbc.co.uk                              1   0.00      682   0.00    0    3.68 
 *.nprove.com                             1   0.00     3607   0.00    0   13.71 
 *.pulse360.com                           1   0.00     3624   0.00    0    7.15 
 *.bbci.co.uk                             1   0.00    10155   0.00    0  110.19 
 *.amazon.com                             1   0.00     1374   0.00    1    1.24 
 *.feedburner.com                         1   0.00     1876   0.00    0    7.07 
 *.fbcdn.net                              1   0.00     1847   0.00    0   15.03 
cayenne-e.linux.bogus                    50   0.00   243886   0.00    0   20.60 
 *.redali.com                            18   0.00    91453   0.00    0   21.43 
 *.nflximg.com                           15   0.00    45007   0.00    0   21.20 
 *.netflix.com                            6   0.00    96906   0.00    0   46.66 
 *.google.com                             6   0.00      938   0.00    0    2.14 
 *.microsoft.com                          2   0.00     2596   0.00    0    7.35 
 *.atdmt.com                              1   0.00     5481   0.00    2    3.26 
 *.public-trust.com                       1   0.00     1187   0.00    1    2.26 
 *.brightedge.com                         1   0.00      318   0.00    0    0.87 
--------------------------------- --------- ------ -------- ------ ---- ------- 
Sum                                     356   1.69  2731445   0.19    0   40.53

Notice that this report shows the web servers each system using the proxy server went to. I'm not sure how easy it would be to get it to map to individual users. Also, if you aren't happy with calamaris output, you might want to google a bit on "squid log analyzer" as there are quite a few options available.

I don't typically set up squid to require user login, but maybe if you figured out how to do that, then maybe there would be information in the log file to tie the requests to individual users. If you choose to pursue this route, I'd be interested in hearing how you make out.

Now, the other problem is forcing traffic through the proxy server. If you aren't planning on using the NST system as a gateway on your network, you will need to prevent users from accessing web sites directly. This usually involves going to your firewall/router and setting up rules such that connections to port 80 and maybe 443 are only permitted if they come from the IP address of your proxy server (the NST system running squid).

Finally, you may want to look at adding the package "squidGuard". I haven't used it, but I know it is intended to be used in conjunction with squid to redirect/block certain web traffic.

2. As far as using Snort inline as a IPS with the NST configuration, it's funny you should ask. Ron is currently working on updating the snort package to the 2.9.0 release and he hinted to me that IPS functionality will be included. I'm not exactly sure what his plans are at the moment, but I would expect to see something in the near future.

Have a good day,
Paul

[Ronin]

Hi Paul;

Thanks for the quick response!!!

The suggestion for squid are great and I will be pursuing the different squid options as you mentioned…and if you like I can give you updates as to what worked and what didn't work.  I can do it here on this forum incase others are interested in doing the same thing.

As for the 2.9.0 version that would be fantastic and including IPS functionality would be awsome!!….Way to go Ron and Paul..

Keep up the great work and I love this product!!!!

Patrick