| Re: [dash-dev] optimizing an Athena build? |
Sadly, at this point it's trial an error... wait for the build fail with a message like
Bundle org.foo_0.0.0 failed to resolve.: Missing required plug-in org.bar_0.0.0.
And then go looking for the plugin or feature to add to the list of IUs you need. Chances are good you can just add the plugin and the Galileo repo and you're done... but in some cases, you have to dig deeper - especially if you want to be able to run the build offline (eg., using repo zips).
Or use PDE (or the PDE viz plugin) to show the plugins you depend on. Then you just need to map those plugins to their containing projects (eg. EMF, GEF, WebTools WST / JST, XSD)
What I do is grab an update site zip, unpack it, then iterate through the feature jars, unpacking them. From there you can iterate through the unjarred features' feature.xml files looking for plugin names that match the one you need. For example:
unzip -d wst-buildrepo-R-3.1.1-20090917225226.zip{_,}
cd wst-buildrepo-R-3.1.1-20090917225226.zip_/features/
for f in $(find . -maxdepth 1 -type f -name "*.jar"); do unzip -d ${f}{_,}; done
find.sh . feature.xml wst.sse
find.sh . feature.xml org.eclipse.wst.ws_ui.feature
Note that find.sh is a wrapper for find + grep which I wrote (WTFPL license), and is available here [1].)
[1]http://divbyzero.com/linux/find.in.files.sh.txt
I'm looking for a better, more discovery-based approach (eg., using Buckminster or Tycho) which would be able to crawl though the sources and discover what features/plugins are needed, then use that generated list as input to the IUsToInstall property (if not otherwise set). If you have any ideas, I'm open to 'em.
N
Nicolas Bros wrote:
Wow! That looks useful.
But how do you get the names of the IUs you must put in IUsToInstall?
On Wed, Dec 9, 2009 at 8:31 PM, Nick Boldt <nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx>> wrote:<http://org.eclipse.core.net> -installIU org.eclipse.equinox.common
If you list the IUsToInstall separated by "+" instead of "," they're
all installed in a single step.
Requires Eclipse 3.5+ as the base, though, not 3.4.
Example from a recent build:
[exec] [p2.dir] Command-line arguments: -application
org.eclipse.equinox.p2.director -consoleLog -flavor tooling -roaming
-profile SDKProfile -installIU org.eclipse.sdk.feature.group
-installIU org.eclipse.sdk.ide -installIU org.eclipse.core.net<http://org.eclipse.core.net>+org.eclipse.equinox.common+org.eclipse.core.runtime+org.eclipse.debug.core+org.eclipse.rcp.feature.group+\
-installIU org.eclipse.core.runtime -installIU
org.eclipse.debug.core -installIU org.eclipse.rcp.feature.group
-installIU org.eclipse.jst.server.generic.core -installIU
org.eclipse.wst.ws_core.feature.feature.group -installIU
org.eclipse.wst.web_ui.feature.feature.group -installIU
org.eclipse.wst.ws_wsdl15.feature.feature.group -installIU
org.eclipse.wst.xml_ui.feature.feature.group -installIU
org.eclipse.wst.common_ui.feature.feature.group -installIU
org.eclipse.wst.common_core.feature.feature.group -installIU
org.eclipse.xsd.feature.group -installIU
org.eclipse.gef.feature.group -installIU
org.eclipse.gmf.runtime.diagram.ui -installIU
org.eclipse.emf.compare.match -installIU
org.eclipse.emf.compare.diff -installIU org.eclipse.emf.compare.ui
-installIU org.mozilla.xpcom -installIU org.jboss.tools.vpe.resref
-installIU org.jboss.tools.jst.web.ui -installIU
org.jboss.ide.eclipse.as.core -installIU
org.jboss.ide.eclipse.archives.webtools -installIU
org.jboss.tools.jmx.feature.feature.group -destination
/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse
-bundlepool
/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/testing/target/eclipse
-metadataRepository
file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,
-artifactRepository
file:/home/hudson/hudson_workspace/workspace/jbosstools-cbi-bpel/build/N-SNAPSHOT/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,http://download.jboss.org/jbosstools/updates/nightly/trunk/,http://download.eclipse.org/releases/galileo/,
-profileProperties org.eclipse.update.install.features=true -os
linux -ws gtk -arch x86
As run from this releng project [1]:
[1]https://anonsvn.jboss.org/repos/jbosstools/trunk/bpel/releng/
Which simply says:
repositoryURLs=\
http://download.jboss.org/jbosstools/updates/nightly/trunk/,\
http://download.eclipse.org/releases/galileo/
IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+org.eclipse.core.net<mailto:nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx>>>
org.eclipse.jst.server.generic.core+\
org.eclipse.wst.ws_core.feature.feature.group+org.eclipse.wst.web_ui.feature.feature.group+org.eclipse.wst.ws_wsdl15.feature.feature.group+\
org.eclipse.wst.xml_ui.feature.feature.group+org.eclipse.wst.common_ui.feature.feature.group+org.eclipse.wst.common_core.feature.feature.group+\
org.eclipse.xsd.feature.group+org.eclipse.gef.feature.group+\
org.eclipse.gmf.runtime.diagram.ui+\
org.eclipse.emf.compare.match+org.eclipse.emf.compare.diff+org.eclipse.emf.compare.ui+\
org.mozilla.xpcom+org.jboss.tools.vpe.resref+org.jboss.tools.jst.web.ui+org.jboss.ide.eclipse.as.core+org.jboss.ide.eclipse.archives.webtools+org.jboss.tools.jmx.feature.feature.group
In this case, even pulling from the Galileo and JBT nightly update
sites, the provisioning step only took 4 minutes (248411 ms).
If you source from locally cached p2 repo zips such as these [2],
it's even faster.
[2]http://download.eclipse.org/athena/repos/
Nick
David Carver wrote:
There are ways to do this. And you might be able to do it in
your own customTargets.xml file. Basically you would do a two
stage build, either all in one Job or in two separate jobs. The
first one would be to build and create the target platform and
provision it. The second would be to create the necessary
tests, and run them based on the input from the first provisioning.
I've done this in the past by using a combination of the ZIP
files, combining them into a working target platform, and then
deploying the tests and running them. I do agree that the P2
provisioning process seems to take way longer to get setup than
just a straight ZIP provisioning.
The main issue I see right now is that with the P2 director
steps, it is done in a loop, instead of being able to provide a
list of all the items to be provisioned in one provisioning
step. Each plugin or feature is provisioned separately. If
we could find a way to provide a list of items to be provisioned
and have it done in one provisioning step (i.e. not executing P2
director multiple times), I think that alone would go a long way
to speeding up the build.
Also, you may want to look at an alternative cleanUp step if you
have a large directory that is created during the build with
many sub directories. I've found that trying to avoid wildcard
searches and deletions with FileSets greatly speeds up the
deletion and cleanup process.
Nicolas Bros wrote:
In fact, I was not only thinking of re-using the build
target platform for testing, but also re-using a previously
snapshotted build target platform instead of provisioning a
new one for each build. My idea was to be able to skip all
the work done in the "postSetup" target in
customTargets.xml, if nothing changed in the build
configuration since the last build. Would that be doable ?
On Wed, Dec 9, 2009 at 3:24 AM, Nick Boldt
<nickboldt@xxxxxxxxx <mailto:nickboldt@xxxxxxxxx><mailto:dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>>
wrote:
The previously provisioned target used to build is reused
when
creating the test target - it's added to the list of
repos from
which IUs are installed.
This means (in theory) that instead of refetching from
Galileo,
IUs will simply be copied from the local disk. So the
only thing
that slows the process down is file i/o.
Yes, we could even go a step further to use the existing
eclipse
instead of creating a fresh, clean one, but that means a
"dirty"
test, and I don't like that approach. Having this
behaviour be
optional is certainly doable, but I'm sure people like
Dave C will
object violently to the idea of setting up tests w/ chunks of
leftover compilation artifacts in them, instead of a clean
"provision the target, install the built runtime, install the
tests, and run them" process.
It may be slower, but it's a more valid test IMO.
If you want an optional override to reuse the build
environment
instead of creating a new one, please open a bug and
propose the
override property name, eg.,
"useDirtyBuildEnvironmentForTesting" :)
Nicolas Bros wrote:
Hi,
I'd like my build to be faster. I have noticed that
Athena
installs a whole new Eclipse with all the dependencies
specified in the build.properties twice for each
build (once
for the build and a second time for tests). That
obviously
takes quite some time.
Would it be possible to save the Eclipse install once the
dependencies are deemed stable, and re-use it for future
builds? There could be a property in the
build.properties file
that indicates the path of an existing Eclipse
install to use
for the build for example.
If not, what would prevent this?
-- Nicolas Bros
R&D
tel: 06 75 09 19 88
nbros@xxxxxxxxxxxxxxxx
<mailto:nbros@xxxxxxxxxxxxxxxx>
<mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>
<mailto:nbros@xxxxxxxxxxxxxxxx
<mailto:nbros@xxxxxxxxxxxxxxxx>
<mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>>
nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
<mailto:nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>>
<mailto:nbros.mia@xxxxxxxxx
<mailto:nbros.mia@xxxxxxxxx> <mailto:nbros.mia@xxxxxxxxx
<mailto:nbros.mia@xxxxxxxxx>>>
Mia-Software, 410 clos de la Courtine
93160 Noisy-le-Grand
http://www.mia-software.com
.: model driven agility :.
------------------------------------------------------------------------
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx><mailto:dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>>
https://dev.eclipse.org/mailman/listinfo/dash-dev
-- Nick Boldt :: http://nick.divbyzero.com
Release Engineer :: Eclipse Modeling & Dash Athena
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/dash-dev
-- Nicolas Bros
R&D
tel: 06 75 09 19 88
nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx><mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>
nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
<mailto:nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>>
Mia-Software, 410 clos de la Courtine
93160 Noisy-le-Grand
http://www.mia-software.com
.: model driven agility :.
------------------------------------------------------------------------
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/dash-dev
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/dash-dev
-- Nick Boldt :: http://nick.divbyzero.com
Release Engineer :: Eclipse Modeling & Dash Athena
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/dash-dev
--
Nicolas Bros
R&D
tel: 06 75 09 19 88
nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
Mia-Software, 410 clos de la Courtine
93160 Noisy-le-Grand
http://www.mia-software.com
.: model driven agility :.
------------------------------------------------------------------------
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dash-dev
--
Nick Boldt :: http://nick.divbyzero.com
Release Engineer :: Eclipse Modeling & Dash Athena
_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dash-dev