bittorrent blues
When Eclipse 3.1 was released, the eclipse.org website slowed to a crawl within hours. One key cause of the saturation was that we failed to allow our mirror sites time to sync up before publishing the links. It hurt and we learned.
During the 3.1 chaos, I must have received about 10 e-mails from (understandably) disgruntled downloaders all wondering why I didn’t post torrents. Commens ranged from “P2P - it seems so obvious” to “who’s the idiot”. So why didn’t we?
I tried to use BitTorrent in the past with little success. Slow downloads, no peers and no seeds left me dangling with a fast cable connection and nothing coming in. So when Eclipse 3.1 was released, I couldn’t imagine why one would want to use BitTorrent to download it. But these e-mails made me question myself. Has it changed? All ten of them can’t be wrong? Has the power of P2P outclassed dozens of server-class computers with big disks connected to large Internet pipes throughout the world, managed by skilled SysAdmins?
Shortly after 3.1, I set out to download Fedora Core 4. It weighs in at about 2.5 GB. I spotted the FC4 torrent links and I figured, “it’s a sign, let’s give this another try”. With a new attitude, I go to download Azureus, arguably the best BitTorrent client out there. Clicking on the Linux X86_64 link, I am presented with a list of mirror sites. Hrm. “Mirrors, how passé” I think to myself. With Azureus installed, I then click on the shiny Fedora Core 4 torrent links and voilà - I’m downloading FC4 at the blazing speed of — 90kB/sec.
What is this? I can easily get just shy of 600 kB/sec from eclipse.org and here I am, with the power of the P2P world at my feet, yawning at 90k with only 30 peers. “Maybe I need to give it some time to get to like me” I think so I go out for a while.
Several hours later, I return to my Azureus screen, only to see 18 kB/sec - 4%. Ouch, only 12 peers. Granted, I’m giving a bit back, uploading at - hang on - 5kB/sec, but at this rate, it would be faster to drive to Raleigh and ask someone at Red Hat for the CDs (Ottawa-Raleigh is about a 28-hour round-trip, for about 2.5GB - grossing 26 kB/sec. I did the math).
Just for fun, I return to the Fedora page, hit the download link and click, *shudder*, mirrors. I browse to my favorite mirror site and click the CD1 ISO. Boom. 490 kB/sec. Within a few hours I had all 5 ISO’s, ready to burn, having picked 5 different mirrors for each CD. I quietly shut down Azureus, now at 12%, and went to bed.
So what have I learned from all this? Mirror sites still rock - many thanks are due to the mirror maintainers worldwide. For the next big Eclipse Release, we’ll allow time for our mirrors to sync up before publishing links, and we’ll publish links to mirror sites that host torrents. Will we make torrents available on eclipse.org? I don’t think it’s necessary.
Posted August 5th, 2005 by Denis Roy in category: Uncategorized
You can skip to the end and leave a response. Pinging is currently not allowed.
13 Responses to “bittorrent blues”
Leave a Reply
You must be logged in using your Eclipse Bugzilla account to post a comment.
Anonymous Says:
August 5th, 2005 at 12:50 pm
I am going to have to disagree. For me, bittorrent is a much faster way to receive popular files. I am often frustrated by mirrors and the slow speed I download files from them. Bittorrent allows me to get any file quickly and easily without a hassle of finding a mirror that has the file and the bandwith.
Joakim E Says:
August 5th, 2005 at 1:27 pm
Lets see if I get this right.
Eclipse.org had an event that crushed the servers with more requests and bandwidth than it could handle.
People said, “why not use bitorrent to alleviate the crush”.
You proceed to test bitorrent out on a file that only 30 people worldwide want.
Not exactly an apples to apples type of test.
Go hit a torrent with over 200 seeds, maybe Fedora (Stentz) i386 binary instead -
http://torrent.fedoraproject.org/stentz-binary-i386.torrent
Also, were you behind a firewall that masked your incoming port? The load balancing algorithms within the swarm will automatically downgrade your connection if you don’t participate equally.
-JoakimE
Anonymous Says:
August 5th, 2005 at 1:39 pm
So why don’t you let your servers actually seed the torrents? That way you get the speed of of your server plus the speed of the uploaders.
Seems to be a good think to get the extra bandwith for free.
RefuX Says:
August 5th, 2005 at 2:22 pm
Bitorrent is not a total slam dunk for every distribution scenario. However what is was designed to do it does extremely well:
It is designed to deal with the scenario where a lot of people want to same file at the same time.
Bitorrent is one of the greatest downloading innovations in the last few years; to disregard it simply because you have tried it a few times and it didn’t work out is silly.
If you want to experience the power of Bittorrent, wait until the next big game demo / patch is released and see which gets you your download faster.
There is also the aspect of cost, Blizzard’s World of Warcraft uses Bittorrent for the distribution system for patches/updates this saves them tens of thousands of dollars a month! (I wish I could find the article that talks about this). Of course WoW falls into the perfect Bittorrent scenario; everyone wants the same file at the same time!
This is exactly the same scenario with a new Eclipse release; everyone wants the same file at the same time.
I don’t know how the whole mirror network works and who pays for what, but you would think it would be in everyone’s interest to reduce bandwidth load on servers?
Denis Roy Says:
August 5th, 2005 at 3:07 pm
Interesting comments, thanks.
Joakim, I tried your suggestion, and indeed it yielded better results. With FC4/i386 I’m getting about 210kB/sec after 30 minutes. I’m still far from my usual 550, but it’s somewhat respectable and I’m also giving back to the comunity by allowing people to upload. Unfortunately, i386 is not what I want to download, so I’m back to square 1. I do concede that torrents will help the win32 crowd, but the rest are pretty much up the creek.
Refux, I’m not going to disregard it, because as you say, it would be silly. Oh and silly I am not! You must have misread the last paragraph of my blurb.
Anonymous Says:
August 5th, 2005 at 3:55 pm
Denis, did you also follow joakim’s suggestion of making sure you upload ports are open? Makes a difference.
I believe we’re going to see some web-enabled client programs (browsers, and others) which are going to come bit-torrent enabled, meaning clicking on a torrent link will be handled by the app, just like a typical download is today.
One downside of BT is that it’s impossible to get d/l counts. No more million download contests!
– pmuellr
Anonymous Says:
August 5th, 2005 at 6:25 pm
I strongly disagree. There must be a reason why all mayor Multiplayer Games (HL2, WOW) are turning towards P2P distribution of patches.
I´d never get 600 kb/s from almost any US server (I´m in Germany) - and definitely not from Eclipse.org. Bittorrent is a lot more responsive with heavy load - check e.g. the distribution of a 350mb heavy episode of “Lost” only an hour after it´s aired.
Eclipse 3.1 release was a real joke compared to the excellent work done in it itself.
I´d like to see a comparision of P2P stability vs DOS as well, as I guess multiple trackers are far easier to keep online that single sites.(The download counter on the http releases of Eclipse 3.1 on different mirrors didn´t help decrease the main server´s load either…)
Brad Reynolds Says:
August 5th, 2005 at 7:49 pm
With the logic applied in the post a Porsche in the shop is better than the 1990 Honda Accord that never fails and it’s the day of the big interview. I really don’t mean to sound like a smart ass but 18kB/sec is better than 0kB/sec. It’s not about the download speed, it’s about having a download. I’m glad you all at eclipse learned a lesson, I hope things go better next time.
Denis Roy Says:
August 5th, 2005 at 8:30 pm
pmueller, the million download challenge was indeed another element that complicated release 3.1. I honestly hope we never do a live download counter again.
Brad, If I were to give you an 18 kB/sec download for the next release, I’m convinced you won’t send me praise for providing you a download. Instead, you’re going to tell me it’s hella slow. What I would really like if for the next release to go without a hitch. A couple of our mirror sites volunteered to post torrents for the main files, so we’ll likely take them up on that offer.
Anonymous Says:
August 8th, 2005 at 7:17 am
I can’t really comment on the torrent stuff, I never really used it myself.
Just a tad of input on the download stats handling, Mozilla uses a software called bouncer, http://osuosl.org/projects/bouncer/, which gives good perf for the almost 80M downloads of firefox, with many different mirrors and binaries (at least two products, 30+ locales, a bunch of versions, upgrades vs downloads).
We did have our share of release download hickups, and AFAICT, the last one was the first one we survived well. So there has to be a lesson that the guys at OSUOSL learned, I bet they tell it, if you ping them nicely.
Anonymous Says:
August 8th, 2005 at 8:14 am
Nice post but you should have think that Fedora 4 has been out for a while already.
You should have tried when it got out.
That’s the same thing with Eclipse 3.1
If there was a torrent file the moment of the release, people would have choose it. (the ones who believe in it)
Mariano Kamp Says:
August 12th, 2005 at 6:56 am
Well, if the bandwidth dedicated from the mirrors would be available for bittorrent than it would be almost as fast as an old school download. This is obviously also true for non-windows binaries.
Taking into account that the downloaders are also uploading at the same time than it will beat the old school downloads.
Even better the mirrors just need to subscribe to new torrents available from eclipse.org and the distribution would go smoothly and without the overhead of setting up rsync etc. The problems with broken links like with eclipse 3.1 would not appear.
Btw. Try downloading Knoppix for example. You’ll see how fast that goes. And not only i386 goes fast.
Mariano Kamp Says:
August 17th, 2005 at 4:32 am
I believe a better example for the proper use of bittorrent is the distribution of the Knoppix DVD: http://torrent.unix-ag.uni-kl.de/index.html