Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [dash-dev] optimizing an Athena build?
If you list the IUsToInstall separated by "+" instead of "," they're all installed in a single step. 

Requires Eclipse 3.5+ as the base, though, not 3.4.

Example from a recent build:
[exec] [p2.dir] Command-line arguments: -application org.eclipse.equinox.p2.director -consoleLog -flavor tooling -roaming -profile SDKProfile -installIU org.eclipse.sdk.feature.group -installIU org.eclipse.sdk.ide -installIU org.eclipse.core.net -installIU org.eclipse.equinox.common -installIU org.eclipse.core.runtime -installIU org.eclipse.debug.core -installIU org.eclipse.rcp.feature.group -installIU org.eclipse.jst.server.generic.core -installIU org.eclipse.wst.ws_core.feature.feature.group -installIU org.eclipse.wst.web_ui.feature.feature.group -installIU org.eclipse.wst.ws_wsdl15.feature.feature.group -installIU org.eclipse.wst.xml_ui.feature.feature.group -installIU org.eclipse.wst.common_ui.feature.feature.group -installIU org.eclipse.wst.common_core.feature.feature.group -installIU org.eclipse.xsd.feature.group -installIU org.eclipse.gef.feature.group -installIU org.eclipse.gmf.runtime.diagram.ui -installIU org.eclipse.emf.compare.match -installIU org.eclipse.emf.compare.diff -installIU org.eclipse.emf.compare.ui -installIU org.mozilla.xpcom -installIU org.jboss.tools.vpe.resref -installIU org.jboss.tools.jst.web.ui -installIU org.jboss.ide.eclipse.as.core -installIU org.jboss.ide.eclipse.archives.webtools -installIU org.jboss.tools.jmx.feature.feature.group -destination /home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse -bundlepool /home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse -metadataRepository file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/, -artifactRepository file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/, -profileProperties org.eclipse.update.install.features=true -os linux -ws gtk -arch x86 

As run from this releng project [1]:

[1]https://anonsvn.jboss.org/repos/jbosstools/trunk/bpel/releng/

Which simply says:

repositoryURLs=\
http://download.jboss.org/jbosstools/updates/nightly/trunk/,\
http://download.eclipse.org/releases/galileo/

IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+org.eclipse.core.net+org.eclipse.equinox.common+org.eclipse.core.runtime+org.eclipse.debug.core+org.eclipse.rcp.feature.group+\
org.eclipse.jst.server.generic.core+\
org.eclipse.wst.ws_core.feature.feature.group+org.eclipse.wst.web_ui.feature.feature.group+org.eclipse.wst.ws_wsdl15.feature.feature.group+\
org.eclipse.wst.xml_ui.feature.feature.group+org.eclipse.wst.common_ui.feature.feature.group+org.eclipse.wst.common_core.feature.feature.group+\
org.eclipse.xsd.feature.group+org.eclipse.gef.feature.group+\
org.eclipse.gmf.runtime.diagram.ui+\
org.eclipse.emf.compare.match+org.eclipse.emf.compare.diff+org.eclipse.emf.compare.ui+\
org.mozilla.xpcom+org.jboss.tools.vpe.resref+org.jboss.tools.jst.web.ui+org.jboss.ide.eclipse.as.core+org.jboss.ide.eclipse.archives.webtools+org.jboss.tools.jmx.feature.feature.group
In this case, even pulling from the Galileo and JBT nightly update sites, the provisioning step only took 4 minutes (248411 ms).
If you source from locally cached p2 repo zips such as these [2], it's even faster. 

[2]http://download.eclipse.org/athena/repos/

Nick

David Carver wrote:
There are ways to do this. And you might be able to do it in your own customTargets.xml file. Basically you would do a two stage build, either all in one Job or in two separate jobs. The first one would be to build and create the target platform and provision it. The second would be to create the necessary tests, and run them based on the input from the first provisioning.
I've done this in the past by using a combination of the ZIP files, combining them into a working target platform, and then deploying the tests and running them. I do agree that the P2 provisioning process seems to take way longer to get setup than just a straight ZIP provisioning.
The main issue I see right now is that with the P2 director steps, it is done in a loop, instead of being able to provide a list of all the items to be provisioned in one provisioning step. Each plugin or feature is provisioned separately. If we could find a way to provide a list of items to be provisioned and have it done in one provisioning step (i.e. not executing P2 director multiple times), I think that alone would go a long way to speeding up the build.
Also, you may want to look at an alternative cleanUp step if you have a large directory that is created during the build with many sub directories. I've found that trying to avoid wildcard searches and deletions with FileSets greatly speeds up the deletion and cleanup process. 

Nicolas Bros wrote:
In fact, I was not only thinking of re-using the build target platform for testing, but also re-using a previously snapshotted build target platform instead of provisioning a new one for each build. My idea was to be able to skip all the work done in the "postSetup" target in customTargets.xml, if nothing changed in the build configuration since the last build. Would that be doable ?
On Wed, Dec 9, 2009 at 3:24 AM, Nick Boldt <nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx>> wrote: 

    The previously provisioned target used to build is reused when
    creating the test target - it's added to the list of repos from
    which IUs are installed.

    This means (in theory) that instead of refetching from Galileo,
    IUs will simply be copied from the local disk. So the only thing
    that slows the process down is file i/o.

    Yes, we could even go a step further to use the existing eclipse
    instead of creating a fresh, clean one, but that means a "dirty"
    test, and I don't like that approach. Having this behaviour be
    optional is certainly doable, but I'm sure people like Dave C will
    object violently to the idea of setting up tests w/ chunks of
    leftover compilation artifacts in them, instead of a clean
    "provision the target, install the built runtime, install the
    tests, and run them" process.

    It may be slower, but it's a more valid test IMO.

    If you want an optional override to reuse the build environment
    instead of creating a new one, please open a bug and propose the
    override property name, eg., "useDirtyBuildEnvironmentForTesting" :)

    Nicolas Bros wrote:

        Hi,

        I'd like my build to be faster. I have noticed that Athena
        installs a whole new Eclipse with all the dependencies
        specified in the build.properties twice for each build (once
        for the build and a second time for tests). That obviously
        takes quite some time.
        Would it be possible to save the Eclipse install once the
        dependencies are deemed stable, and re-use it for future
        builds? There could be a property in the build.properties file
        that indicates the path of an existing Eclipse install to use
        for the build for example.
        If not, what would prevent this?
        --         Nicolas Bros
        R&D
        tel: 06 75 09 19 88
        nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
        <mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>
        nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
        <mailto:nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>>

        Mia-Software, 410 clos de la Courtine
        93160 Noisy-le-Grand
        http://www.mia-software.com
        .: model driven agility :.
------------------------------------------------------------------------ 

        _______________________________________________
        dash-dev mailing list
        dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
        https://dev.eclipse.org/mailman/listinfo/dash-dev


    --     Nick Boldt :: http://nick.divbyzero.com
    Release Engineer :: Eclipse Modeling & Dash Athena
    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev




--
Nicolas Bros
R&D
tel: 06 75 09 19 88
nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
Mia-Software, 410 clos de la Courtine
93160 Noisy-le-Grand
http://www.mia-software.com
.: model driven agility :.
------------------------------------------------------------------------

_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dash-dev


_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dash-dev


--
Nick Boldt :: http://nick.divbyzero.com
Release Engineer :: Eclipse Modeling & Dash Athena

Back to the top