[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [p2-dev] Mirror ranking
- From: "Kirill Balod" <Kirill.Balod@xxxxxxxxxxx>
- Date: Thu, 10 Dec 2009 15:51:13 -0000
- Delivered-to: email@example.com
- Thread-index: Acp5QnuUJg3DKvTYRLWGFhT4CbytZQAa7GNA
- Thread-topic: [p2-dev] Mirror ranking
One more notice - CompositeRepository is used for a
"mirror task" marks a repo as bad if download of one IU was failed.
I guess - it is optimisation for remote sites (but not
sure it is always correct) but it strange solution for local
For example - some zipped archive sites now don't have
"jar" entries but only "jar.pack.gz". "jar" entries were manually removed before
zip but "artifacts.xml" still contains references to two artifacts for each
IU. In spite of ability to fetch all IU from such site - mirror task can
not mirror it.
I'm sure this algorithm can be
improved. Please enter a bug report for it. I like the idea of resetting or
lowering the failure count if we later have a successful transfer. How it
behaved on the most recent transfer is generally going to be much more
interesting than historical behaviour.
12/09/2009 01:40 PM
P2 developer discussions
|P2 developer discussions
mirrored Helios today and it basically took forever. After a few
was beginning to wonder what was going on and luckily, the
process ran in a
debugger. I found that the top ranked mirror was the
one at eclipse.org.
That surprised me since I know that I have a fast
mirror in Sweden that
serves up a copy of Helios.
First I checked if this mirror was included
in the list served up by the
mirror request to Eclipse.org. It was. Next, I
stopped the debugger and
patched the URL for entry number zero in my mirrors
list with the URL of
that mirror. I resumed and now the processing went very
much faster. So
the mirror was actually OK.
So why did
download.eclipse.org move to the top of the list? It's
supposed to be right
at the bottom. The algorithm for sorting the
mirrors looks like
public int compareTo(Object o)
if (!(o instanceof
MirrorInfo that = (MirrorInfo)
//less failures is
if (this.failureCount !=
return this.failureCount - that.failureCount;
//faster is better
(this.bytesPerSecond != that.bytesPerSecond)
return (int) (that.bytesPerSecond -
initial rank indicates geographical proximity
return this.initialRank - that.initialRank;
A failure count of one will deem the mirror forever worse
then a failure
count the zero, no matter if that mirror is a hundred times
think that was what caused my problem. All mirrors in the list
failureCount of 1 and a byte-count of -1, except two,
download.eclipse.org (initialRank = 55) and one other (initialRank=10)
because after some initial failure, they were never given a second
My guess is that something went wrong at the very beginning that
all mirrors except download.eclipse.org and node number 10 to fail.
sure what that was. That however, moved download.eclipse.org to the top
and node number 10 to second place. And although I have mirrors 100
times faster close by, they are never consulted again. I'm downloading
about 3.800 artifacts.
Mirrors may have temporary and fairly short
outages. They may be
incomplete in some respect, or just be under very heavy
load for a short
period of time. I think the algorithm could be improved by
periodic retry on mirrors with an initialRank value that indicates
it is geographically close. I also think that we should have a ratio
between high transfer rate and failure count. Let's say that 5 times
higher transfer rate is worth one failure. Perhaps a successful transfer
should reset the failure count, or at least cut it in half so that
failures are forgiven by subsequent good behavior.
One question that
I don't know the answer to at this point is what
happens when an artifact is
missing although it should be there
according to the artifact repository.
Will the mirror get punished by
that? If that's the case, then it's not so
good. The same will be true
on all mirrors but the best one will be
What do you think?
to report this email as spam.
This message has been scanned for viruses by MailController.