Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [dash-dev] optimizing an Athena build?
Sadly, at this point it's trial an error... wait for the build fail with a message like
Bundle org.foo_0.0.0 failed to resolve.: Missing required plug-in org.bar_0.0.0.
And then go looking for the plugin or feature to add to the list of IUs you need. Chances are good you can just add the plugin and the Galileo repo and you're done... but in some cases, you have to dig deeper - especially if you want to be able to run the build offline (eg., using repo zips).
Or use PDE (or the PDE viz plugin) to show the plugins you depend on. Then you just need to map those plugins to their containing projects (eg. EMF, GEF, WebTools WST / JST, XSD)
What I do is grab an update site zip, unpack it, then iterate through the feature jars, unpacking them. From there you can iterate through the unjarred features' feature.xml files looking for plugin names that match the one you need. For example: 

unzip -d wst-buildrepo-R-3.1.1-20090917225226.zip{_,}
cd wst-buildrepo-R-3.1.1-20090917225226.zip_/features/
for f in $(find . -maxdepth 1 -type f -name "*.jar"); do unzip -d ${f}{_,}; done 
find.sh . feature.xml wst.sse
find.sh . feature.xml org.eclipse.wst.ws_ui.feature
Note that find.sh is a wrapper for find + grep which I wrote (WTFPL license), and is available here [1].) 

[1]http://divbyzero.com/linux/find.in.files.sh.txt
I'm looking for a better, more discovery-based approach (eg., using Buckminster or Tycho) which would be able to crawl though the sources and discover what features/plugins are needed, then use that generated list as input to the IUsToInstall property (if not otherwise set). If you have any ideas, I'm open to 'em. 

N

Nicolas Bros wrote:

Wow! That looks useful.
But how do you get the names of the IUs you must put in IUsToInstall?
On Wed, Dec 9, 2009 at 8:31 PM, Nick Boldt <nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx>> wrote: 

    If you list the IUsToInstall separated by "+" instead of "," they're
    all installed in a single step.

    Requires Eclipse 3.5+ as the base, though, not 3.4.

    Example from a recent build:

        [exec]    [p2.dir] Command-line arguments:  -application
    org.eclipse.equinox.p2.director -consoleLog -flavor tooling -roaming
    -profile SDKProfile -installIU org.eclipse.sdk.feature.group
    -installIU org.eclipse.sdk.ide -installIU org.eclipse.core.net
    <http://org.eclipse.core.net> -installIU org.eclipse.equinox.common
    -installIU org.eclipse.core.runtime -installIU
    org.eclipse.debug.core -installIU org.eclipse.rcp.feature.group
    -installIU org.eclipse.jst.server.generic.core -installIU
    org.eclipse.wst.ws_core.feature.feature.group -installIU
    org.eclipse.wst.web_ui.feature.feature.group -installIU
    org.eclipse.wst.ws_wsdl15.feature.feature.group -installIU
    org.eclipse.wst.xml_ui.feature.feature.group -installIU
    org.eclipse.wst.common_ui.feature.feature.group -installIU
    org.eclipse.wst.common_core.feature.feature.group -installIU
    org.eclipse.xsd.feature.group -installIU
    org.eclipse.gef.feature.group -installIU
    org.eclipse.gmf.runtime.diagram.ui -installIU
    org.eclipse.emf.compare.match -installIU
    org.eclipse.emf.compare.diff -installIU org.eclipse.emf.compare.ui
    -installIU org.mozilla.xpcom -installIU org.jboss.tools.vpe.resref
    -installIU org.jboss.tools.jst.web.ui -installIU
    org.jboss.ide.eclipse.as.core -installIU
    org.jboss.ide.eclipse.archives.webtools -installIU
    org.jboss.tools.jmx.feature.feature.group -destination
    /home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse
    -bundlepool
    /home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse
    -metadataRepository
    file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,
    -artifactRepository
    file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,
    -profileProperties org.eclipse.update.install.features=true -os
    linux -ws gtk -arch x86

    As run from this releng project [1]:

    [1]https://anonsvn.jboss.org/repos/jbosstools/trunk/bpel/releng/

    Which simply says:

    repositoryURLs=\
    http://download.jboss.org/jbosstools/updates/nightly/trunk/,\
    http://download.eclipse.org/releases/galileo/

    IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+org.eclipse.core.net
    <http://org.eclipse.core.net>+org.eclipse.equinox.common+org.eclipse.core.runtime+org.eclipse.debug.core+org.eclipse.rcp.feature.group+\
    org.eclipse.jst.server.generic.core+\
    org.eclipse.wst.ws_core.feature.feature.group+org.eclipse.wst.web_ui.feature.feature.group+org.eclipse.wst.ws_wsdl15.feature.feature.group+\
    org.eclipse.wst.xml_ui.feature.feature.group+org.eclipse.wst.common_ui.feature.feature.group+org.eclipse.wst.common_core.feature.feature.group+\
    org.eclipse.xsd.feature.group+org.eclipse.gef.feature.group+\
    org.eclipse.gmf.runtime.diagram.ui+\
    org.eclipse.emf.compare.match+org.eclipse.emf.compare.diff+org.eclipse.emf.compare.ui+\
    org.mozilla.xpcom+org.jboss.tools.vpe.resref+org.jboss.tools.jst.web.ui+org.jboss.ide.eclipse.as.core+org.jboss.ide.eclipse.archives.webtools+org.jboss.tools.jmx.feature.feature.group

    In this case, even pulling from the Galileo and JBT nightly update
    sites, the provisioning step only took 4 minutes (248411 ms).

    If you source from locally cached p2 repo zips such as these [2],
    it's even faster.

    [2]http://download.eclipse.org/athena/repos/

    Nick


    David Carver wrote:

        There are ways to do this.  And you might be able to do it in
        your own customTargets.xml file.  Basically you would do a two
        stage build, either all in one Job or in two separate jobs.  The
        first one would be to build and create the target platform and
        provision it.   The second would be to create the necessary
        tests, and run them based on the input from the first provisioning.

        I've done this in the past by using a combination of the ZIP
        files, combining them into a working target platform, and then
        deploying the tests and running them.   I do agree that the P2
        provisioning process seems to take way longer to get setup than
        just a straight ZIP provisioning.

        The main issue I see right now is that with the P2 director
        steps, it is done in a loop, instead of being able to provide a
        list of all the items to be provisioned in one provisioning
        step.   Each plugin or feature is provisioned separately.   If
        we could find a way to provide a list of items to be provisioned
        and have it done in one provisioning step (i.e. not executing P2
        director multiple times), I think that alone would go a long way
        to speeding up the build.

        Also, you may want to look at an alternative cleanUp step if you
        have a large directory that is created during the build with
        many sub directories.  I've found that trying to avoid wildcard
        searches and deletions with FileSets greatly speeds up the
        deletion and cleanup process.

        Nicolas Bros wrote:

            In fact, I was not only thinking of re-using the build
            target platform for testing, but also re-using a previously
            snapshotted build target platform instead of provisioning a
            new one for each build. My idea was to be able to skip all
            the work done in the "postSetup" target in
            customTargets.xml, if nothing changed in the build
            configuration since the last build. Would that be doable ?

            On Wed, Dec 9, 2009 at 3:24 AM, Nick Boldt
            <nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx>
            <mailto:nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx>>>
            wrote:

               The previously provisioned target used to build is reused
            when
               creating the test target - it's added to the list of
            repos from
               which IUs are installed.

               This means (in theory) that instead of refetching from
            Galileo,
               IUs will simply be copied from the local disk. So the
            only thing
               that slows the process down is file i/o.

               Yes, we could even go a step further to use the existing
            eclipse
               instead of creating a fresh, clean one, but that means a
            "dirty"
               test, and I don't like that approach. Having this
            behaviour be
               optional is certainly doable, but I'm sure people like
            Dave C will
               object violently to the idea of setting up tests w/ chunks of
               leftover compilation artifacts in them, instead of a clean
               "provision the target, install the built runtime, install the
               tests, and run them" process.

               It may be slower, but it's a more valid test IMO.

               If you want an optional override to reuse the build
            environment
               instead of creating a new one, please open a bug and
            propose the
               override property name, eg.,
            "useDirtyBuildEnvironmentForTesting" :)

               Nicolas Bros wrote:

                   Hi,

                   I'd like my build to be faster. I have noticed that
            Athena
                   installs a whole new Eclipse with all the dependencies
                   specified in the build.properties twice for each
            build (once
                   for the build and a second time for tests). That
            obviously
                   takes quite some time.
                   Would it be possible to save the Eclipse install once the
                   dependencies are deemed stable, and re-use it for future
                   builds? There could be a property in the
            build.properties file
                   that indicates the path of an existing Eclipse
            install to use
                   for the build for example.
                   If not, what would prevent this?
                   --         Nicolas Bros
                   R&D
                   tel: 06 75 09 19 88
                   nbros@xxxxxxxxxxxxxxxx
            <mailto:nbros@xxxxxxxxxxxxxxxx>
            <mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>
                   <mailto:nbros@xxxxxxxxxxxxxxxx
            <mailto:nbros@xxxxxxxxxxxxxxxx>
            <mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>>
                   nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
            <mailto:nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>>
                   <mailto:nbros.mia@xxxxxxxxx
            <mailto:nbros.mia@xxxxxxxxx> <mailto:nbros.mia@xxxxxxxxx
            <mailto:nbros.mia@xxxxxxxxx>>>

                   Mia-Software, 410 clos de la Courtine
                   93160 Noisy-le-Grand
                   http://www.mia-software.com
                   .: model driven agility :.
------------------------------------------------------------------------ 

                   _______________________________________________
                   dash-dev mailing list
                   dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
            <mailto:dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>>
                   https://dev.eclipse.org/mailman/listinfo/dash-dev


               --     Nick Boldt :: http://nick.divbyzero.com
               Release Engineer :: Eclipse Modeling & Dash Athena
               _______________________________________________
               dash-dev mailing list
               dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
            <mailto:dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>>
               https://dev.eclipse.org/mailman/listinfo/dash-dev
-- Nicolas Bros 
            R&D
            tel: 06 75 09 19 88
            nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
            <mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>
            nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
            <mailto:nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>>
            Mia-Software, 410 clos de la Courtine
            93160 Noisy-le-Grand
            http://www.mia-software.com
            .: model driven agility :.
            ------------------------------------------------------------------------

            _______________________________________________
            dash-dev mailing list
            dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
            https://dev.eclipse.org/mailman/listinfo/dash-dev


        _______________________________________________
        dash-dev mailing list
        dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
        https://dev.eclipse.org/mailman/listinfo/dash-dev
-- Nick Boldt :: http://nick.divbyzero.com 
    Release Engineer :: Eclipse Modeling & Dash Athena
    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev




--
Nicolas Bros
R&D
tel: 06 75 09 19 88
nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
Mia-Software, 410 clos de la Courtine
93160 Noisy-le-Grand
http://www.mia-software.com
.: model driven agility :.


------------------------------------------------------------------------

_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dash-dev


--
Nick Boldt :: http://nick.divbyzero.com
Release Engineer :: Eclipse Modeling & Dash Athena

Back to the top