Posted: 3/7/2007 7:19 PM
Post subject: New BitTorrent tracker (WIP)
Quote
Bisqwit
Editor, Active player (297)
Joined: 3/8/2004
Posts: 7469
Location: Arzareth
Earlier this week I noticed that the BitTorrent tracker of Tasvideos is starting to reach noticeable CPU load levels. It is the standard BitTorrent tracker with only minor changes (doesn't forget obsolete movies, remembers useragents). So I wrote my own. http://bisqwit.iki.fi/source/bqtrack.html This tracker was written with two goals in mind: ­-- Compatible with the site code (including the way the "dstate" file is loaded and saved) -- It is as fast as possible. In particular, I couldn't care less about the http administration/statistics features, which seem to be what most other trackers seem to concentrate at. I think I accomplished both goals suitably well. However, it has less features than the standard BitTorrent tracker. In particular, it has no NAT detection schemes whatsoever. I don't know how much that matters. It has also been observed, that it reports significantly higher swarm member counts than the old tracker did -- likely somehow inflated. I have a theory why it could happen (doesn't detect abruptly dropped peers except from a timeout of 1 hour), but I don't know how the standard tracker avoids it. Also, it occassionally dies from a "too many open file descriptors" error. I don't know how to effeciently avoid that either. (No, it does not leak resources.) The tracker comes with General Public License, so if someone wants to try a hand at improving it, patches are welcome :)
Posted: 3/7/2007 8:15 PM
Post subject: Re: New BitTorrent tracker (WIP)
Quote
Tub
Joined: 6/25/2005
Posts: 1377
someone has been busy :)
Bisqwit wrote:
Also, it occassionally dies from a "too many open file descriptors" error. I don't know how to effeciently avoid that either. (No, it does not leak resources.)
I didn't check, but you could try increasing /proc/sys/fs/file-max ..
m00
Posted: 3/8/2007 1:37 AM
Quote
ZeXr0
Joined: 8/27/2006
Posts: 883
I don't know if it was that tracker, but it had been reported a couple of time that the tracker was either down, or users was unable to connect to the tracker. Still I was connected to other users but not the tracker itself.
Posted: 3/8/2007 2:44 AM
Quote
DeHackEd
Emulator Coder, Site Developer, Former player
Joined: 11/6/2004
Posts: 833
As root, 
ulimit -n 2048
You're using libevent, nice.
Posted: 3/8/2007 8:22 AM
Quote
Bisqwit
Editor, Active player (297)
Joined: 3/8/2004
Posts: 7469
Location: Arzareth
DeHackEd wrote:
You're using libevent, nice.
Yeah, libevent is nice. I'm surprised you know about it.
ZeXr0 wrote:
I don't know if it was that tracker, but it had been reported a couple of time that the tracker was either down, or users was unable to connect to the tracker. Still I was connected to other users but not the tracker itself.
The same has also been reported for the actual site. I suspect connection problems. On the other hand, during development I have restarted the tracker several times with several minute long pauses.
Posted: 3/8/2007 1:47 PM
Quote
DeHackEd
Emulator Coder, Site Developer, Former player
Joined: 11/6/2004
Posts: 833
Bisqwit wrote:
Yeah, libevent is nice. I'm surprised you know about it.
Of course I do. It's needed for memcached, and microstorage uses it as of a few months ago to cache compressed data (the poor P166 couldn't keep up). You suggested I do this, didn't you? One big thing about trackers is they tend to get a lot of hung connections, given that clients with dumb upload rate settings will make communication really slow. The default tracker uses poll()/select() which is O(connections). libevent uses Linux's epoll(), which is O(interesting events). That should help a little bit.
Posted: 3/8/2007 1:50 PM
Quote
Bisqwit
Editor, Active player (297)
Joined: 3/8/2004
Posts: 7469
Location: Arzareth
DeHackEd wrote:
Of course I do. It's needed for memcached, and microstorage uses it as of a few months ago to cache compressed data (the poor P166 couldn't keep up). You suggested I do this, didn't you?
It's just that I didn't expect you to remember a particular library memcached happens to use. EDIT: Okay, added NAT checking to the tracker now.
Posted: 3/22/2007 2:03 AM
Post subject: Tracker problems
Quote
Carnildo
Joined: 3/22/2007
Posts: 7
I'm seeing about an order of magnitude more seeds than normal (and a higher peer count). I've also got three torrents that are giving unusual responses from the tracker: For superpunchout-tas-adelikat.avi, Azureus is reporting -153 peers. For supermarioworld-timeattack-star96-viper7.avi and supermario64-tas-16stars-spezzafer.avi, Azureus is reporting "Error (Requested download is not authorized for use with this tracker.)"
Posted: 3/22/2007 2:15 AM
Post subject: Re: Tracker problems
Quote
Tub
Joined: 6/25/2005
Posts: 1377
Carnildo wrote:
For supermarioworld-timeattack-star96-viper7.avi and supermario64-tas-16stars-spezzafer.avi, Azureus is reporting "Error (Requested download is not authorized for use with this tracker.)"
Both of these movies were obsoleted and are not tracked any more.
m00
Posted: 3/22/2007 2:19 AM
Quote
DeHackEd
Emulator Coder, Site Developer, Former player
Joined: 11/6/2004
Posts: 833
Those AVIs are old and have been dropped.
Posted: 3/23/2007 11:19 PM
Quote
slo_bro
Joined: 5/17/2006
Posts: 34
I'm seeing unusually high numbers both for seeds and downloaders. Dark Fulgore's SNES Killer Instinct is reporting 76 downloaders. For several days btfriend.py has been sharing the same six videos, and only rarely have any of them actually been being downloaded from my computer. Hopefully this just means that no videos are in dire need of seeds right now. (3/27/2007) Looks like btfriend.py is working fluidly today. In fact, right now six videos are being shared from my PC.
Posted: 4/10/2007 3:00 PM
Quote
Deep_Loner
Joined: 12/26/2006
Posts: 256
Location: United States of America
Bisqwit wrote:
EDIT: Okay, added NAT checking to the tracker now.
Sorry for bringing up an old topic (and please pardon my ignorance), but what happens to a BitTorrent download if the tracker detects an NAT router? Is the downloaded slowed down or blocked? If so, can certain exceptions be made? I'm just trying to find ways to work around my ISP's prejudice against BitTorrent.
Posted: 4/10/2007 3:01 PM
Quote
DeHackEd
Emulator Coder, Site Developer, Former player
Joined: 11/6/2004
Posts: 833
It means that the tracker will not send your IP address to other users as a potential client to connect to. It has nothing to do with prejudice or anything like that. If you're NAT'd, you can't be connected to and any attempt would waste other users' resources, so you're taken off the list.
Posted: 4/10/2007 3:07 PM
Quote
Deep_Loner
Joined: 12/26/2006
Posts: 256
Location: United States of America
...so if I can't be connected to, then it's impossible for other clients to share file pieces with me, which defeats the purpose... is my understanding of this right?
Posted: 4/10/2007 3:08 PM
Quote
DeHackEd
Emulator Coder, Site Developer, Former player
Joined: 11/6/2004
Posts: 833
No, you can be provided with a list of other users, but that's the only way to find anybody. If everybody fails a nat check, you're genuinely stuck. If only 1 or 2 users pass a nat check, they're the only people who can be connected to and performance stinks because they're responsible for all the bandwidth. You'd be amazed at what the NAT ratio is on some torrents.
Posted: 4/10/2007 3:11 PM
Quote
Bisqwit
Editor, Active player (297)
Joined: 3/8/2004
Posts: 7469
Location: Arzareth
If the tracker determines that your client cannot be connected to, it will not offer you as a peer to other peers, for it would be futile for them to try to connect to your client as well. You will still receive list of other peers (but only those who don't have that same cumbersome problem), so transfer should be possible. This is how I understand what the NAT checking feature does. I might be wrong. PS: The tracker that is currently running on the server (for the last 11 days) is not the one I posted about; it is the official BitTorrent tracker once again.
Posted: 4/10/2007 3:17 PM
Quote
Deep_Loner
Joined: 12/26/2006
Posts: 256
Location: United States of America
OK. So it is possible to download Torrents from behind an NAT router; it's just not fair to the other users. But it's a start. Thanks for the help.
Posted: 4/10/2007 3:19 PM
Quote
Bisqwit
Editor, Active player (297)
Joined: 3/8/2004
Posts: 7469
Location: Arzareth
Deep Loner wrote:
OK. So it is possible to download Torrents from behind an NAT router; it's just not fair to the other users. But it's a start.
No, of any two peers, if either one has a connectable port, the transfer can work in both directions. If neither one has, then they cannot exchange pieces.