Community
Participate
Working Groups
Currently this is optimized for the gef case of docs/ examples/ features/ plugins/ but really we should work with any case. We can look for feature.xml files and plugin.xml/fragment.xml files and create a proper directory structure that way.
See also bug 251929 - build from pre-built source zip. We need to build from: a) a workspace of flat plugins/features b) a cvs/svn tree dump (with nesting of features/ and plugins/ subfolders) c) a zip of sources in either of the above structures
It should be noted that to build from a local cache of sources, you will also need o.e.test and o.e.a.o.junit on disk... at least until we make the build smarter so it's fetched independently? Here's how to hackaround the missing o.e.test plugin problem [1], until we write code to convert a standard cvs/svn map from "artifacts :: repo path or URL" to "artifacts :: local path", and does <copy> instead of <cvs> to get the files. http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started#Build_From_Local_Sources Dave, care to write a map-to-copy-script xslt for us? :) In case (a), the $localSourceCheckoutDir would be ~/workspace/ and any paths in the map, like org.eclipse.gef/plugins/org.eclipse.zest, *would be collapsed* eg., zest plugin would be found in ~/workspace/org.eclipse.zest In case (b), the $localSourceCheckoutDir would be ~/workspace/org.eclipse.gef, and paths *would be maintained* eg., zest plugin would be found in ~/workspace/org.eclipse.gef/plugins/org.eclipse.zest In case (c), we'd need to unpack the zip, then inspect the contents to see if it's in form (a), archived workspace or (b), archived repo tree.
(In reply to comment #2) > Dave, care to write a map-to-copy-script xslt for us? :) Well might need a intermediate step that takes the map entries, which are just text entries, and converts them to XML first. If we weren't limited to XSLT 1.0 and could run Saxon it would be easy to do in XSLT 2.0 script as it has support for reading plain text files and then tokenizing them into a ResultSet. Now..if you want a PSF to Copy script that can be done easily.
Okay, looks like ANT has a LoadFile task that will load a text file into a property variable. Which means I can load the Map file into a property and then pass that as a param to the XSLT. I'll investigate this tomorrow.
Created attachment 128645 [details] Map2Copy.xsl Draft 1 The attached file tries to address options (a) and (b). Given the following formatted map file: !*************** xsl.map ! This map file is for ! incubating XSL component. ! These bundles are currently being developed and released ! from the HEAD branch feature@org.eclipse.wst.xsl.feature=v200902112333,:pserver:anonymous@dev.eclipse.org:/cvsroot/webtools,,incubator/sourceediting/features/org.eclipse.wst.xsl.feature feature@org.eclipse.wst.xsl_sdk.feature=v200808310406,:pserver:anonymous@dev.eclipse.org:/cvsroot/webtools,,incubator/sourceediting/features/org.eclipse.wst.xsl_sdk.feature feature@org.eclipse.wst.xsl_tests.feature=v200902122343,:pserver:anonymous@dev.eclipse.org:/cvsroot/webtools,,incubator/sourceediting/features/org.eclipse.wst.xsl_tests.feature The XSL will accept three parameters passed to it: localSourceCheckoutDir - the fullpath to the workspace or location files are to be checkedout (copied) to. collapse.options - a numeric value corresponding to: 0 - collapse plugins into the localSourceCheckoutDir 1 - maintain the same structure and copy into the localSourceCheckoutDir When if collapse.options is not specified, it is assumed to be collapse. When option collapse is run, and the workspace location is set to ~/workspace, the following output will be generated: <?xml version="1.0" encoding="UTF-8"?> <project name="map2cpy"> <copy todir="~/workspace/org.eclipse.wst.xsl.feature"> <fileset dir="incubator/sourceediting/features/org.eclipse.wst.xsl.feature"/> </copy> <copy todir="~/workspace/org.eclipse.wst.xsl_sdk.feature"> <fileset dir="incubator/sourceediting/features/org.eclipse.wst.xsl_sdk.feature"/> </copy> <copy todir="~/workspace/org.eclipse.wst.xsl_tests.feature"> <fileset dir="incubator/sourceediting/features/org.eclipse.wst.xsl_tests.feature"/> </copy> </project> When collapse.option = 1 the following output will be generated: <?xml version="1.0" encoding="UTF-8"?> <project name="map2cpy"> <copy todir="~/workspace/incubator/sourceediting/features/org.eclipse.wst.xsl.feature"> <fileset dir="incubator/sourceediting/features/org.eclipse.wst.xsl.feature"/> </copy> <copy todir="~/workspace/incubator/sourceediting/features/org.eclipse.wst.xsl_sdk.feature"> <fileset dir="incubator/sourceediting/features/org.eclipse.wst.xsl_sdk.feature"/> </copy> <copy todir="~/workspace/incubator/sourceediting/features/org.eclipse.wst.xsl_tests.feature"> <fileset dir="incubator/sourceediting/features/org.eclipse.wst.xsl_tests.feature"/> </copy> </project>
The XSLT does make use of the tokenize() EXSLT extension function. So you will need to run with an XSLT processor that supports this. Both Xalan and Saxon support it, and it has been tested with Xalan. The map file will be needed to be passed in as a parameter. Use Ant's loadfile task to load up a property and then pass in the property as an expression to the XSLT.
Oh and the input to the XSLT, can be any XML file, even the map2copy.xsl file itself can be the input, it doesn't care.
So, the workflow here is: a) user fetches CVS/SVN tree to disk, eg., using a .psf file. Structure could be anything. b) user configures .releng project to tell Athena where to find checked out sources in ${localSourceCheckoutDir} c) user kicks a build. --- The build will then: a) load mapfiles into a variable b) run transform using .xsl, passing in: ${localSourceCheckoutDir}, loaded map variable ${map}, and ${collapse.options}=0 (I see no use for =1 option); All feature@.... lines will be <copy>'d into ${buildDirectory}/features/; everything else will be <copy>'d into ${buildDirectory}/plugins/ c) run generated script to copy from ${localSourceCheckoutDir} into ${buildDirectory}/plugins/ and ${buildDirectory}/features/ d) build. --------- I wonder if we could accomplish the same thing by a) search in ${localSourceCheckoutDir} for dirs named "feature*": copy all child folders into ${buildDirectory}/features/ b) search in ${localSourceCheckoutDir} for dirs named "*feature": copy all those dirs into ${buildDirectory}/features/ c) search in ${localSourceCheckoutDir} for dirs with plugin.xml, fragment.xml, or META-INF/MANIFEST.MF files in them: copy all those dirs into ${buildDirectory}/plugins/ This is more or less what we have already in o.e.dash.common.releng/builder/all/customTargets.xml#copyLocalSourceCheckout, but without the ability to recurse deeper into any arbitrary dir structure. Perhaps a <flattenmapper> would work? -------- Currently, your XSL only handles features. It needs to handle plugins & fragments too. I have to reject your patch because of this.
We'll need sample map files that contain features, fragments, bundles, etc. Will see what I can do to update the XSL. It is after all only a draft. :)
In Fedora we have a PDE Build builder called "package-build" that only differs from the stock PDE Build build.xml in that it has a customTargets.xml that creates a features/ and plugins/ directory structure from a flat CVS/SVN checkout. It's hack-tacular but what it does is call a perl script which looks for feature.xml and plugin.xml/fragment.xml files and makes symlinks under features/ and plugins/ respectively.
(In reply to comment #9) > We'll need sample map files that contain features, fragments, bundles, etc. Will this do? http://dev.eclipse.org/viewcvs/index.cgi/releng/trunk/org.eclipse.linuxtools.releng/maps/linuxtools.map?revision=21930&root=Technology_LINUXTOOLS&view=markup
(In reply to comment #11) > (In reply to comment #9) > > We'll need sample map files that contain features, fragments, bundles, etc. See also https://bugs.eclipse.org/bugs/show_bug.cgi?id=265847#c5
*** Bug 275763 has been marked as a duplicate of this bug. ***
> This won't work. We need to be able to copy features into > ${buildDir}/eclipse/features/ for PDE to find them. > > If you want to support a flat scheme, then we would need to recursively check > each dir under ${localSourceCheckoutDir} for a feature.xml file and copy those > feature dirs into ${buildDir}/eclipse/features. > Okay...so basically the problem comes down to the fact that PDE build is expecting a certain directory structure and isn't flexible enough in it's searching. I've noticed in several places with in the non-Athena build scripts hard coding of particular directory names instead of relying on the properties to provide those areas. Anyways, there are work arounds that we can do, but in general a workspace build is going to be a flattened structure that shouldn't require another checkout of files. I'll see what I can do to help debug and setup this situation.
I've got a new method for converting some localSourceCheckoutDir into the format that PDE needs. To test it, I did this: mkdir ~/workspace/org.eclipse.linuxtools.tree; \ cd ~/workspace/org.eclipse.linuxtools.tree; \ for d in autotools changelog eclipse-build libhover oprofile p2-update-checker profiling releng rpm rpmstubby sdktests systemtap valgrind; do \ svn co https://dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/$d/trunk $d; \ done which results in about 95M of files in a non-standard dir format. Then I copied everything over into the correct format using <ant target="convertRawCheckoutToFeaturesAndPluginsDirs" antfile="${helper}"> <property name="localSourceCheckoutDir" value="${localSourceCheckoutDir}" /> <property name="targetDir" value="${buildDirectory}" /> </ant> which parses for feature.xml, MANIFEST.MF, plugin.xml, fragment.xml, and for folders ending in .releng or .updatesite. The feature.xml dirs get copied into ${targetDir}/features, the rest into ${targetDir}/plugins, and the .releng and .updatesite stuff goes into a ${targetDir}/releng folder, just in case it's needed. The resulting ${targetDir} tree is about 30M, because the .svn folders are omitted from the copy. This will be committed into buildAllHelper.xml in HEAD shortly; I'm also working on bug 273518 at the same time, so attaching a patch would be confusing. There's one minor problem, and that's that feature ids and foldernames have to match (as is generally true for plugins) - see bug 276611. That said, I plan to keep removing the "-feature" suffix when copying from the source tree into the ${buildDirectory} folder, but NOT removing ".feature".
Released to HEAD.
