[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [eclipse-mirrors] mirror causing issues?
|
James,
Indeed, hosting Eclipse is problematic even for us :) I've always
considered the behaviour you've seen to be normal. Here's a glimpse at
the CLOSE_WAIT count per IP on one of the download.eclipse.org servers.
$ netstat -an | awk '/CLOSE_WAIT/ {split($5,c,":"); print c[1]}' | sort
| uniq -c | sort -nr
24 60.176.x.x
16 118.114.x.x
10 123.232.x.x
7 59.62.x.x
7 58.247.x.x
etc..
Keep in mind that download.eclipse.org is setup for 4,000 apache
children (five servers @ 800 each), and these numbers are taken from a
single server.
The code 206's are typically an artifact of download managers or
proxy/cache mechanisms. It's odd that the range seems to fluctuate, but
bear in mind that the IP you've listed is in China, where the
person-to-IP ratio is much, much higher than what we see here in North
America.
SO how can we solve the problem? Good question. I'm sure you have
KeepAlive's off, as well as DNS lookups. Raising the Apache limit is
fine until you run out of RAM. Switching to a different Apache MPM could
be useful if you can do it. Like many, we use prefork for maximum PHP
compatibility. Lowering the tcp wait time could work, or looking for an
Apache module to limit the number of connections per IP. At Eclipse we
have a perl script that stays resident and monitors activity like this.
If a single IP occupies more than X sockets, we block the IP for a few
minutes with iptables just to keep things under control.
Another alternative would be to host the Eclipse mirror on a different
machine, or installing a lightweight http server (like lighttpd) on your
box and using it for the Eclipse mirror instead, where it is free to hit
limits you set without impacting the rest of your site.
Does anyone else have ideas?
Denis
James Craig wrote:
Lately I have noticed that my eclipse mirror is creating problems
for the rest of my website. I was hoping someone else may have had
similar problems or even better: a solution.
Currently my apache server allows up to 512 child processes to handle
web traffic. Before we became an eclipse mirror we needed 50 and were
pretty happy. What started to happen is all 50 children were getting
used up serving eclpise. ok... so we went to 512 (after trying 100,
200, etc) and everything seemed good.
Now we are getting hit again, and I am starting to look at WHAT is
happening.
I am not sure what to make of this yet;
netstat -na shows me a lot of this:
129.21.30.99.80 58.208.50.191.38146 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38354 64083 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38360 65535 2903 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38362 65535 2903 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38411 45207 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38449 65535 2903 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38450 65535 5807 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38451 65535 11615 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38452 65535 2903 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38614 48967 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38624 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38655 65535 7259 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38678 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38679 65535 2903 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38682 65535 5807 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38683 63487 5807 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38911 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38919 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.38926 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39100 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39129 65535 4355 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39153 65535 2903 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39373 65535 3215 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39385 65535 3215 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39439 65535 3216 49368 0
CLOSE_WAIT
129.21.30.99.80 58.208.50.191.39464 65535 3216 49368 0
CLOSE_WAIT
my access logs show this:
grep 58.208.50.191 today_access
58.208.50.191 - - [19/Mar/2009:10:32:48 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 11171588
58.208.50.191 - - [19/Mar/2009:10:32:21 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 119971811
58.208.50.191 - - [19/Mar/2009:10:31:17 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 115139299
58.208.50.191 - - [19/Mar/2009:10:31:31 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 65060335
58.208.50.191 - - [19/Mar/2009:10:31:32 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 92568818
58.208.50.191 - - [19/Mar/2009:10:31:32 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 156356524
58.208.50.191 - - [19/Mar/2009:10:31:38 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 135074239
58.208.50.191 - - [19/Mar/2009:10:31:41 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 52516385
58.208.50.191 - - [19/Mar/2009:10:31:41 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 29904541
58.208.50.191 - - [19/Mar/2009:10:31:41 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 121384147
58.208.50.191 - - [19/Mar/2009:10:31:41 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 23746375
58.208.50.191 - - [19/Mar/2009:10:31:57 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 146362170
58.208.50.191 - - [19/Mar/2009:10:31:57 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 19919071
58.208.50.191 - - [19/Mar/2009:10:31:59 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 8110847
58.208.50.191 - - [19/Mar/2009:10:32:03 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 63776997
58.208.50.191 - - [19/Mar/2009:10:32:03 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 47505213
58.208.50.191 - - [19/Mar/2009:10:32:03 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 16153317
58.208.50.191 - - [19/Mar/2009:10:32:03 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 7420077
58.208.50.191 - - [19/Mar/2009:10:32:22 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 83651354
58.208.50.191 - - [19/Mar/2009:10:32:24 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 146357058
58.208.50.191 - - [19/Mar/2009:10:32:24 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 77465420
58.208.50.191 - - [19/Mar/2009:10:32:48 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 83648762
58.208.50.191 - - [19/Mar/2009:10:32:56 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 16102831
58.208.50.191 - - [19/Mar/2009:10:32:56 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 100707732
58.208.50.191 - - [19/Mar/2009:10:33:23 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 89899285
58.208.50.191 - - [19/Mar/2009:10:33:23 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 98696426
58.208.50.191 - - [19/Mar/2009:10:33:26 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 44874890
58.208.50.191 - - [19/Mar/2009:10:33:28 -0400] "GET
/mirrors/eclipse/eclipse/downloads/drops/R-3.4.2-200902111700/eclipse-SDK-3.4.2-win32.zip
HTTP/1.1" 206 128740909
I notice that it seems to be the case for eclipse-SDK-3.4.2-win32.zip
160193808
and
eclipse-jee-ganymede-SR2-win32.zip 171861283
rarely do they get the whole file, but they leave their connections open,
holding on to a child processes, and tying up resources.\
There are quite a few sites (few resolve via dig and nslookup) that are
doing
this to us.
Does anyone else see this?
Anyone have any thoughts?
thank you,
Jim Craig
_______________________________________________
eclipse-mirrors mailing list
eclipse-mirrors@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipse-mirrors
