Description Alex Blewitt 2011-01-26 10:25:01 EST

I was attempting to mirror the Helios P2 repository via:

eclipsec -Declipse.p2.mirror=false -noSplash -consoleLog -application org.eclipse.eqinox.p2.artifact.repository.mirrorApplication -source http://download.eclipse.org/releases/helios

However, it takes a significant amount of time to do this. Partially that's due to the incoming network but it's demonstrably slower towards the end than at the beginning.

What appears to happen is that it downloads an artifact, re-generates artifacts.jar, downloads another artifact, re-generates artifacts.jar, downloads another artifact, re-generates artifacts.jar ...

The fact that this file is being re-written after each download, rather than at the end when all the artifacts are completely downloaded, causes a significant amount of time to be wasted as it keeps getting re-written.

Is there a valid technical reason why we can't wait until all artifacts have been mirrored before (re)generating artifacts.jar? It seems little point to capture a work-in-progress, since there's no reason why an artifact that you have downloaded may depend on one you haven't yet; as such, the artifacts.jar is always going to be incomplete until the job has finished anyway.

This can be mitigated in writing to a memory-mapped filesystem (which negates the disk spinning though it's still CPU burning) until the whole mirror is complete, at which point you can then copy it off to disk.