| Re: [epp-dev] Files for release mirrors to have time |
Jonah,
The problem from
https://bugs.eclipse.org/bugs/show_bug.cgi?id=548497#c29 is
happening again:
$curl -O
http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 429 100 429 0 0 1618 0 --:--:-- --:--:--
--:--:-- 1618
$curl -I
http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
0 429 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 Mar 2020 13:06:13 GMT
Content-Type: application/x-java-archive
Content-Length: 429
Connection: keep-alive
Last-Modified: Fri, 06 Mar 2020 14:22:22 GMT
ETag: "1ad-5a03061c70fb5"
X-NodeID: download2
X-Proxy-Cache: STALE
Accept-Ranges: bytes
When I do this it bypasses the cache:
$curl -I --header "X-Cache-Bypass: true"
http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0HTTP/1.1 404 Not Found
Server: nginx
Date: Wed, 18 Mar 2020 13:09:15 GMT
Content-Type: text/html
Connection: keep-alive
Vary: Accept-Encoding
X-NodeID: download2
But a subsequent call still returns it as existing so it does not
flush the cache:
$curl -I
http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
0 429 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 Mar 2020 13:09:21 GMT
Content-Type: application/x-java-archive
Content-Length: 429
Connection: keep-alive
Last-Modified: Fri, 06 Mar 2020 14:22:22 GMT
ETag: "1ad-5a03061c70fb5"
X-NodeID: download2
X-Proxy-Cache: STALE
Accept-Ranges: bytes
Both compositeContent.jar and compositeArtifacts.jar need to yield 404 for a HEAD request...
Note how it says "X-Proxy-Cache: STALE" in each case.
Several calls like this:
$curl -I
http://download.eclipse.org/technology/epp/packages/2020-03/content.jar
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
0 77058 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 Mar 2020 13:15:15 GMT
Content-Type: application/x-java-archive
Content-Length: 77058
Connection: keep-alive
Last-Modified: Tue, 17 Mar 2020 11:19:04 GMT
ETag: "12d02-5a10b1a79ee91"
X-NodeID: download2
X-Proxy-Cache: HIT
Accept-Ranges: bytes
Show "X-Proxy-Cache: HIT".
I think both those files were deleted two hours ago.
Regards,
Ed
I believe everything is now in place for the release except that I and CI user cannot touch the directory so that mirrors are working for p2. See Bug 561215
Thanks,Jonah
On Tue, 17 Mar 2020 at 09:55, Denis Roy <denis.roy@xxxxxxxxxxxxxxxxxxxxxx> wrote:
_______________________________________________For completeness... Ed's assessment of file timestamps vs. directory is spot on, since p2 typically uses the directory as the mirrorsURL parameter.
Denis
On 2020-03-17 1:20 a.m., Ed Merks wrote:
Markus,
I believe that the server's general approach via download.php is to consider the time stamp of the local file versus its timestamp on the mirror. If they match, the mirror is a good one and will be returned in the list. For the download folder, I expect the timestamp is not so important because only individual files will be downloaded from it. E.g., for this link, the timestamp of the file is important, but not the timestamps of any of the parent folders:
Of course that's just a "theory"...
But for the repository itself, this URL is used by p2:
And here the timestamp of the 2020-03 folder itself is definitely important.
That all being said, an install will only need to download 4 artifacts from the EPP repo, while needing to download 100s from the train repo, so if there were no mirrors for the EPP repo, it would not cause such a huge problem as it does when there are no mirrors for the train repo. That repo already exists and has many mirrors:
In less than 24 hours, this also appears to be fully mirrored:
Regards,
Ed
On 16.03.2020 21:31, Markus Knauer wrote:
You are right. I used to be very careful with the modification times of files and directories, especially the download directories are important. I got used to using my TIMESTAMP files that can be found here and there on the download server, usually in commands like
touch -r . TIMESTAMPtouch -r TIMESTAMP .
to "store" the modification times of the current working directory, and to use it afterwards to copy the very same modification time back to it.
Regards,Markus
On Mon, 16 Mar 2020 at 17:23, Ed Merks <ed.merks@xxxxxxxxx> wrote:
_______________________________________________For last year's March release, the installer was downloaded almost 3 million times, and the packages themselves in total more than 1 million times.
https://www.eclipse.org/downloads/packages/release/2019-03/r
Of course it's not clear over which time period that happened, but there's always quite a bit of activity the day of and the days after... The mirror failures right after the most 2019-12 release certainly brought that download server to its knees enough to take note that there was an ongoing problem. And that's in December, close to Christmas time, when you wouldn't expect quite so much either.
That's all to say that I very much appreciate Jonah being conservative on this front.
Note too that I see that the mirrors are back now!
https://www.eclipse.org/downloads/download.php?format=xml&file=/releases/2020-03/202003181000&countryCode=us&timeZone=1&format=xml
https://www.eclipse.org/downloads/download.php?format=xml&file=/technology/epp/packages/2020-03/RC2a&countryCode=us&timeZone=1&format=xmlThere are not so many mirrors for this yet:
https://www.eclipse.org/downloads/download.php?file=/oomph/epp/2020-03/R/eclipse-inst-win64.exe
But that's after only a few hours.
Now that I'm thinking about it though, we should be careful what happens to the timestamp of the EPP folder and the train folder when the content/artifacts change just minutes before the release. If the folder's timestamp changes, I believe all the mirrors will be disabled again for a while...
Regards,
Ed
On 16.03.2020 16:42, Markus Knauer wrote:
Full agreement... 24h is typically enough.Sometimes, if it was really urgent in the past, I double-checked with Denis and the webmasters in order to make them aware that there may be a download peak. On the other hand, the peaks are not that high (especially for a March release) than they used to be.
And yes, I would definitely vote for the RC2a version (without aeri).
Regards,Markus
On Mon, 16 Mar 2020 at 16:07, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:
_______________________________________________
On Mon., Mar. 16, 2020, 10:54 Ed Merks, <ed.merks@xxxxxxxxx> wrote:
Jonah,
I believe 24 hours is normally more than enough, though the problem in https://bugs.eclipse.org/bugs/show_bug.cgi?id=561125 is still a problem so sooner is probably better than later. I feel that it's safe to assume that RC2a versus RC2 will be good because the difference is really very trivial and I've confirmed that the Java, JEE and RCP packages installed from this basis via the installer start up fine...
Thanks Ed for confirming. I will do the downloads this evening (Ottawa time)
Note that if you're going to transform the EPP repo into a simple repository, you should get the binary/features/plugins folders copied up into the 2020-03 folder in preparation as well.
I will place all the binary/features/plugins files in place too, leaving just the artifacts and content files to be updated Wed morning.
I will ensure that the composites are retuning 404 in time for the release too.
Thanks,Jonah
_______________________________________________Thanks for all the heavy lifting!!
Regards,
Ed
On 16.03.2020 15:42, Jonah Graham wrote:
Hi folks,
When do the files need to be in place for release day? I assume today is better than tomorrow.
In that case we need to have RC2a vs RC2 decision ASAP. If I don't hear anything further I will be making the RC2a files the R files.
Thanks,Jonah
_______________________________________________ epp-dev mailing list epp-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________ epp-dev mailing list epp-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________ epp-dev mailing list epp-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________ epp-dev mailing list epp-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev--
Denis Roy
Director, IT Services | Eclipse Foundation, Inc.
Eclipse Foundation: The Platform for Open Innovation and Collaboration
Twitter: @droy_eclipse
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________ epp-dev mailing list epp-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev