Extended BitTorrent Client and Tracker

Hi, I've got a few questions about xbt tracker... Ok, here they come ;) 1) Is there a possibility to restrict statistics to a given IP address only? I'd like to do this so only I have access to them and others wouldn't lurk around. 2) I'd like to install xbt in /usr/bin and its config file in /etc. Is there a switch in xbt which can be used to specify the location of the config file? Something like xbt_tracker --config /etc/xbt_tracker.conf 3) I've had serious problems with another tracker I was using (opentracker). After running it for 56 hours straight, it had consumed almost 500 MB of RAM (my server only got 2.5 GB), even though its total peers amount and torrent tracking stayed the same (peers: ~30000 - torrents: ~ 3500) and the consumption linearly increased with time so I was forced to restart the tracker every 2 days or so through a cron job. How's the memory consumption of xbt ? 3) Can xbt be made a bit more "secure"? By "secure" I mean to poison and inject into its peer list randomly generated IPs for the purpose of plausible deniability. Opentracker, which I used to run, supports and does this. 4) Is there a list somewhere of all the options one can use in its config file? I saw some on the xbt website, but I was wondering if this is a complete list of them or if there's more. Thanks in advance :)

I've made some documentation for XBT Tracker that can be found on http://visigod.com/xbt-tracker Regarding the XBTT vs OpenTracker. I can say that I have xbt running for almost 2 years and never had to restart it (except for upgrades). The memory consumption is minimum and it's probably the best tracker around

Thanks for the reply, visigod... I looked at your options explanation and it looks good so far ;) I'd still very much would like an answer to the config file path specification and restricting statistic to an IP so if someone else can offer an answer, I'd be very happy ;) On a similar note, while I was running opentracker and had problems with it, its author only seemed to reply to mails which are positive in nature or somehow praise his program. If, on the other hand, someone sends him a mail about problems or something that looks odd (I'm looking at you, bloated memory consumption) then he never replies back. At least this was my experience so far with him...

1. Only by editing the source. 2. --conf_file <file> (IIRC) 3. No complaints.

Why are there two questions numbered 3? :p 3. Only by editing the source.

Hi Olaf, Yeah, sorry for the wrong numbering. Was pretty tired here while writing and I didn't proofread. Thanks for the hint on --config_file, I'll look into it My tracker has been running so far for 7 hours and it has only consumed 5.5 MB so far which is very good (if this was opentracker, it'll be close to one hundred MB by now, if not more) so xbt will be my new tracker from now on Another question..... My tracker runs on tcp port 8990 while my web server on 8080. I've noticed that often Azureus clients (try to) connect to the web server itself as is shown from the server's logs below. URL Redirection has not been enabled in the config of xbt so I wonder why these clients go there? In the server's logs I see things like... [08/Dec/2009:11:03:32 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:11:16:29 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_17" [08/Dec/2009:11:36:02 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:11:36:02 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:11:48:33 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_17" [08/Dec/2009:12:08:09 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:12:08:09 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:12:20:39 +0100] "GET / HTTP/1.1" 200 4097 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_17" [08/Dec/2009:12:40:14 +0100] "GET / HTTP/1.1" 200 4098 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:12:40:14 +0100] "GET / HTTP/1.1" 200 4098 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_15" [08/Dec/2009:12:52:40 +0100] "GET / HTTP/1.1" 200 4098 "-" "Azureus 4.3.0.4;Windows XP;Java 1.6.0_17" Is there an way to prevent this?

You should ask the Azureus devs, I've no idea. Or try to reproduce it with Azureus and a network sniffer.

I don't think this has something to do with Azureus itself. While running opentracker, I never saw such things in the web server's log. Only after switching to xbt, did they start to appear. Also, if I enable URL redirection so that those who access the tracker in a browser will be redirected to port 8080 instead, I see a flood of torrent clients being redirected to the web server as if it was the tracker, not xbt, and in the logs there's even the GET ?info_hash=blablablabla stuff clients are trying to get off the web server (I'm running lighttpd btw)

I don't see any other client in your log. If redirect_url isn't set, the tracker never returns a redirect (how would it know the port of your web server?). If it is set, it returns a redirect if the response would otherwise be empty / not found. This might be caused by a wrong path.

That's because I only posted a piece of the log which concerns Azureus as it's the only one which seems to connect to the web server while redirects are off (other clients don't do it).... If I set as option in the config file redirect_url = http://my.website.net:8080 Then after a while, the server's logs get flooded with stuff like this [08/Dec/2009:04:39:08 +0100] "GET /\x00/?info_hash=H1-%de%b4%5d%c5%cc%ad8%d4%92d%f4%99%c1%df%14%95%89 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:10 +0100] "GET /\x00/?info_hash=%05%ff%0c%e5%7d%c3%da-%25%97%9d%7c%d2BH%80%a6%87%5b%d3 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:21 +0100] "GET ?info_hash=%05%ff%0c%e5%7d%c3%da-%25%97%9d%7c%d2BH%80%a6%87%5b%d3 HTTP/1.1" 301 0 "-" "Java/1.6.0_11" [08/Dec/2009:04:39:21 +0100] "GET ?info_hash=H1-%de%b4%5d%c5%cc%ad8%d4%92d%f4%99%c1%df%14%95%89 HTTP/1.1" 301 0 "-" "Java/1.6.0_11" [08/Dec/2009:04:39:21 +0100] "GET /\x00/?info_hash=%05%ff%0c%e5%7d%c3%da-%25%97%9d%7c%d2BH%80%a6%87%5b%d3 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:21 +0100] "GET /\x00/?info_hash=dk%eb%5c%9a%8dak%dd%fb%e6%f4%27%a5%c9_%86%fbc%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:22 +0100] "GET /\x00/?info_hash=H1-%de%b4%5d%c5%cc%ad8%d4%92d%f4%99%c1%df%14%95%89 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:23 +0100] "GET ?info_hash=H1-%de%b4%5d%c5%cc%ad8%d4%92d%f4%99%c1%df%14%95%89 HTTP/1.1" 301 0 "-" "Java/1.6.0_11" [08/Dec/2009:04:39:25 +0100] "GET ?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_07" [08/Dec/2009:04:39:25 +0100] "GET /\x00/?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:26 +0100] "GET ?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_07" [08/Dec/2009:04:39:26 +0100] "GET /\x00/?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:27 +0100] "GET ?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_07" [08/Dec/2009:04:39:27 +0100] "GET /\x00/?info_hash=H1-%de%b4%5d%c5%cc%ad8%d4%92d%f4%99%c1%df%14%95%89 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:27 +0100] "GET /\x00/?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:28 +0100] "GET ?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_07" [08/Dec/2009:04:39:28 +0100] "GET /\x00/?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:29 +0100] "GET ?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_07" [08/Dec/2009:04:39:30 +0100] "GET /\x00/?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:30 +0100] "GET ?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_07" [08/Dec/2009:04:39:31 +0100] "GET /\x00/?info_hash=%cc%98%af%cc%80%8d%ae%c4%a6%ca%9d%b0%97M%83%84%eb%e4%e2%15 HTTP/1.1" 400 349 "-" "-" [08/Dec/2009:04:39:44 +0100] "GET ?info_hash=dk%eb%5c%9a%8dak%dd%fb%e6%f4%27%a5%c9_%86%fbc%15 HTTP/1.1" 301 0 "-" "Java/1.6.0_11" [08/Dec/2009:04:39:52 +0100] "GET /\x00/?info_hash=dk%eb%5c%9a%8dak%dd%fb%e6%f4%27%a5%c9_%86%fbc%15 HTTP/1.1" 400 349 "-" "-" There's a lot of Java in there so I highly suspect these are again Azureus clients. I've also seen BTWebClient which is the Web seed part of uTorrent

The last ones are malformed. The leading slash is missing or there's a null byte.

Note that you should include a trailing slash in your redirect_url...

Well, even with trailing slash in redirect_url = http://my.website.net:8080/, I still get the same, an example... [08/Dec/2009:15:37:49 +0100] "GET /?info_hash=%95w%ff%ffH%a7z%c2%ab%bb%89%14%8b%29%a0%7d%db%90%b2%0d HTTP/1.1" 200 4098 "-" "Java/1.6.0_03" This time the server returns a success (code 200) I got like 20 requests within half a minute from different IPs, and they all are Java clients which again smells of Azureus. Removing redirect_url seems to limit this behavior a lot, though as said, every now and then Azureus clients connect. It's not a big problem for me when redirect_url is disabled, but thought you should know about this....

Maybe Azureus caches the redirect somehow? You could log the requests that trigger redirects in xbt/Tracker/connection.cpp, then you should quickly find out the cause.

What am I looking at in connection.cpp and what to do?

Put these two lines after the s.empty check, after { Then recompile and restart static std::ofstream f("xbt_tracker_raw.log"); f << m_server->time() << '\t' << inet_ntoa(m_a.sin_addr) << '\t' << ntohs(m_a.sin_port) << '\t' << v << std::endl;

it seems this board messes up with the code. Can you paste it to pastebin and provide the link to it?

Yeah, damn SF! It should look like this: http://pastebin.com/m11dc33fa

Alright, I'll do that and let you know. But now, I gotta go to the stores or they'll soon close here in Belgium :S

OK, so I did a few further testings. I switched back to opentracker for a while and noticed that there were still a few Azureus clients always connecting to my web server, thinking it is my tracker instead. Two IPs in particular where connecting to my server like every 2-4 minutes, even though redirection is on in opentracker. Since I couldn't stop them, I just banned these IPs with iptables and now they're gone I also found out what was causing this huge memory consumption when using opentracker. It was the userspace bandwidth shaper (trickle) that I was using. To eliminate doubts, I ran opentracker without it for 4 hours and it always stayed at 2 MB (with trickle, it would have consumed about 40 MB in these four hours). After that, I switched to xbt and ran it with trickle. xbt showed the exact same behavior. RAM consumption linearly increased when trickle was involved while without it, xbt had only consumed 5.5 MB in 10 hours. Under trickle and running for 4 hours, xbt was at 55 MB and climbing.... Another question about xbt: does xbt accept custom IPs sent by clients? If not, this is an official part of the bittorrent protocol for trackers and it should support this ;)

Hehe, blaming opentracker while it was your own fault... :p Yes, but only if the connection is from a LAN IPA. Did you enable the logging?

Actually, it was the fault of trickle. I don't know why it would force an app to consume so much memory while shaping it. I've also seen a similar behavior while shaping xbt but this time using tc htb egress shaping only. No, I haven't enabled logging yet. I'm planning to run xbt for a week then do some other testings. Comparing xbt to opentracker, both have nice things, but opentracker seems to be above it for the following reasons (which I like) 1) it poisons the swarm with random IPs for the purpose of plausible deniability. This is very useful for public trackers and I like that. Tracker operators don't want to play cops for companies that lurk in tracker swarms and then try to sue people just by getting their IPs from there and claiming it is 100% valid information... xbt doesn't have it 2) it accepts custom IPs sent by clients no matter where they are 3) it can restrict stats to a custom IP only so others wouldn't lurk around. Currently, my opentracker gets its stats from 127.0.0.1 so no one is able to get them from there but me.... xbt can't do that 4) doesn't need any DB which is a plus for various reasons, like reduces wear & tear of disk as it doesn't write anything to disk and minimizes possibility of data loss or other problems in case something goes wrong with the DB 5) it supports compressed gzip full scrapes (does xbt support that?) 6) it can sync in a cluster. Though a plus feature, it's not interesting for me on the other hand, comparing bandwidth of opentracker and xbt, it seems xbt beats it. While running opentracker, it tends to download ~20-30 KB/s and upload ~10-20 KB/s. Switching to xbt, download is ~10-20 (most of the time its ~10-15) KB/s and upload is ~5-15 KB/s. both trackers have an announce and full scrape interval of 45 minutes (2700 secs). This is all on a swarm of ~35000 peers tracking ~4800 torrents

2) When is that useful? 3) What stats are so confidential? Did you disable full scrape as well? 4) mysql_host = - 5) Yes

for 2) it's a cheating technique. if someone harvests IPs from the tracker, if a client sends to the tracker a custom IP, that's what the tracker will report to the harvester, afaik... no? for 3) I don't know for xbt (haven't looked much at what it reports as stats) but opentracker can display the top10 of torrent hashes it tracks (top10 based on peers and seeders).... No, my xbt has full scrape on about your 4) answer... I don't get it, do you mean that just by setting mysql_host = -, it does not require a DB at all? about 5) nice. thanks ;)

oh, about the code you gave me. I'll patch and recompile xbt in a week or so, then enable redirection again and see/log if I get clients trying to connect to the web server instead, and wtf is going on :)