Bug 375807 - Need to get auto tagging working on 4.2 primary build
Summary: Need to get auto tagging working on 4.2 primary build
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 4.2   Edit
Hardware: PC Linux
: P1 normal (vote)
Target Milestone: 4.2 M7   Edit
Assignee: Paul Webster CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on: 366279 376000 376229
Blocks: 355430 376460
  Show dependency tree
 
Reported: 2012-04-01 17:08 EDT by David Williams CLA
Modified: 2012-04-25 16:00 EDT (History)
5 users (show)

See Also:


Attachments
showing "missing" doc bundles in R4_HEAD (27.18 KB, image/png)
2012-04-05 00:40 EDT, David Williams CLA
no flags Details
end of log from latest failure (7.06 KB, text/plain)
2012-04-05 19:54 EDT, David Williams CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Williams CLA 2012-04-01 17:08:38 EDT
Paul and I looked at this a little at EclipseCon, and got "closer" but seemed to pick up "old" versions of things, or something. 

Plus, there was something "off" about where the expected git clones are. 

The script that does the main work, is in 
eclipse.platform.releng.eclipsebuilder/scripts/git-release.sh

Just thought I'd open this bug to track this specific issue/task.
Comment 1 David Williams CLA 2012-04-02 23:46:25 EDT
One thing that Paul and I didn't understand while "team programming" was why the maps were not already where git-release.sh expected them to be. 

It expected them to be in $gitCache
(which is basically $supportDir/gitClones, or more exactly (now) 
.../eclipse4/build/supportDir/gitClones

From looking at the masterBuild.sh script I inherited :) it appears to be putting them in "commonrepo" which is under $builddirecotry this translates into 
.../eclipse4/build/supportDir/src/commonrepo/
or more exactly as far as where the maps really are, it goes down from there to include "repo" and "project", namely
.../eclipse4/build/supportDir/src/commonrepo/eclipse.platform.releng.maps/org.eclipse.releng

where is where you see the familiar directories

apiexclude
clickThroughs
components
maps
tagging

So ... does it matter? "commonrepo" is used in more places, 
and the advantage of it being under src is that it would be (presumably) removed for a fresh build, which is I guess that's desired? But, I can see advantages to keeping cloned repos separate from "builddirectory".

There seems to be nothing else in "commonrepo" 

FYI, related bug is the one about scmCache, bug 375788.

Any recommendations? If left to my own devices, I think I'd move all git clones to "gitClones" ... but ... not sure what "hidden assumptions" that'd bump into.
Comment 2 David Williams CLA 2012-04-03 02:33:03 EDT
This is a good illustration of what I don't know (and/or forget), but the "tagRepo" is called _before_ the build is done, so "commonrepo" doesn't even exist at the time tagRepo is called. Makes my head spin. 

So, maybe our own "gitCache" is fine to use? In which case, I'm not sure what "commonrepo" is used for ... I guess to reconcile those that do "auto tagging" and those that do not?
Comment 3 David Williams CLA 2012-04-03 02:40:05 EDT
I wanted to mention, whenever there seems a safe opportunity, I've been renaming actual directories to match up with their variables ... so $gitCache now ends with .../gitCache instead of .../gitClones. Helps to keep straight what I see in the scripts, and what I see on the file system.
Comment 4 David Williams CLA 2012-04-03 23:00:02 EDT
FYI, I got this working enough to produce output! 

The main *.txt files are in .../eclipse4/build/supportDir if anyone (Paul) knows how to tell if they look right. 

I disabled the final "push and tag" so as to not touch the actual maps repo. 

I even got it to send be a "build submission" email (just to me, for now) which reads as follows. It sort of seems close to right ... but, hard for me to tell. 
(FYI, I've "hard coded" the reference build to the last known good 4.2 I build, for now). 
 

The build contains the following changes:
+ Bug 204192 - [WorkbenchParts] NullPointerException in WorkbenchWindow.hardClose (FIXED)
+ Bug 242144 - [Progress] Progress view does not shrink (FIXED)
+ Bug 320851 - NPE in WorkbenchSourceProvider (FIXED)
+ Bug 355430 - change primary builder to 4.2 (NEW)
+ Bug 362532 - [CSS] Re-enable dynamic CSS pseudo-classes (NEW)
+ Bug 364738 - Content Types preference page: default encoding is a text field that is not checked against the supported encodings (FIXED)
+ Bug 374597 - [Compatibility] ClassCastException when adding pure 4.x view to 3.x perspective running in compatibility mode (FIXED)
+ Bug 374671 - Open resource dialog open context menu button is unlabeled (FIXED)
+ Bug 374795 - Open Element dialog (CDT) has text field disabled in 3.8M6 (FIXED)
+ Bug 374938 - Generate javadoc in 4.2 build (FIXED)
+ Bug 374942 - Copy keybinding always brings up key binding selection popup (NEW)
+ Bug 374952 - CCE importing preference in new workspace (FIXED)
+ Bug 375066 - [Compatibility] Show Views dialog is always empty (FIXED)
+ Bug 375069 - [CSS] Elements in parented shells are selected with their parent (FIXED)
+ Bug 375175 - Minor improvements for E4 DI debugging (FIXED)
+ Bug 375219 - [CSS Spy] CSS values sometimes don't seem to take effect (FIXED)
+ Bug 375223 - [CSS] Themes can't load local or http-based resources (NEW)
+ Bug 375264 - E4 Contacts Demo's theme toolbars and menus grow after each execution (FIXED)
+ Bug 375269 - Menus items from fragments multiply after restart (MCommand persistence is wrong) (FIXED)
+ Bug 375271 - [CSS] Deprecation warnings should be more helpful (FIXED)
+ Bug 375280 - [CSS] Add support for CSSEngine to re-apply stylesheets to document roots (NEW)
+ Bug 375282 - [CSS] Add support for Composite.setBackgroundMode() (FIXED)
+ Bug 375583 - [CSS] Cannot set foreground colour on a CTabItem (NEW)
+ Bug 375651 - [Compatibilty] NPE on shutdown in WorkbenchPage#getPerspectiveDesc() (FIXED)
+ Bug 375658 - [CSS] gradient not removed when setting plain colour (FIXED)

The following projects have changed:
com.ibm.icu.base
.gitignore
git_stream_exclude_commits.txt
master
master-ecf
master-equinox
master-equinox-p2
master-equinox-weaving
master-jetty
org.eclipse.ant.core
org.eclipse.ant.launching
org.eclipse.ant.optional.junit
org.eclipse.ant.tests.core
org.eclipse.ant.tests.ui
org.eclipse.ant.ui
org.eclipse.cvs
org.eclipse.e4.core.di
org.eclipse.e4.demo.contacts
org.eclipse.e4.ui.css.core
org.eclipse.e4.ui.css.swt
org.eclipse.e4.ui.css.swt.theme
org.eclipse.e4.ui.model.workbench
org.eclipse.e4.ui.tests.css.core
org.eclipse.e4.ui.tests.css.swt
org.eclipse.e4.ui.workbench.swt
org.eclipse.equinox.sdk
org.eclipse.help-feature
org.eclipse.license
org.eclipse.pde.api.tools.ee.fragments
org.eclipse.pde.junit.runtime.addon
org.eclipse.pde.junit.runtime.standalone
org.eclipse.pde.tools.versioning
org.eclipse.platform
org.eclipse.platform.doc.isv
org.eclipse.platform-feature
org.eclipse.rcp
org.eclipse.releng.tests
org.eclipse.releng.tools
org.eclipse.sdk
org.eclipse.sdk.examples
org.eclipse.sdk.examples-feature
org.eclipse.sdk.tests
org.eclipse.test
org.eclipse.test-feature
org.eclipse.test.performance.data
org.eclipse.test.performance.win32
org.eclipse.ui.ide
org.eclipse.ui.workbench
org.eclipse.update.configurator
org.eclipse.update.core
org.eclipse.update.core.linux
org.eclipse.update.core.win32
org.eclipse.update.examples
org.eclipse.update.scheduler
org.eclipse.update.tests.core
org.eclipse.update.ui
Comment 5 David Williams CLA 2012-04-04 00:58:35 EDT
I should mention, there are lots of error? messages in logs that are as follows. Maybe 30 or them. 
 
Is this a sign something is wrong with scripts? Or, maybe just that many things in not tagged for I20120321-0610 build? 


fatal: ambiguous argument 'I20120321-0610': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
fatal: ambiguous argument 'I20120321-0610..integration': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
fatal: ambiguous argument 'I20120321-0610': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
fatal: ambiguous argument 'I20120321-0610..integration': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
fatal: ambiguous argument 'I20120321-0610': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
fatal: ambiguous argument 'I20120321-0610..integration': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
Comment 6 Paul Webster CLA 2012-04-04 08:42:12 EDT
(In reply to comment #5)
> I should mention, there are lots of error? messages in logs that are as
> follows. Maybe 30 or them. 
> 
> Is this a sign something is wrong with scripts? Or, maybe just that many things
> in not tagged for I20120321-0610 build? 

Yes, the 3.8 repos weren't tagged with I20120321-0610 or even the 3.8 I build tag.  When the report script is trying to get logs from the repo, it can't figure out what we're asking and that's why it doesn't return anything useful.

PW
Comment 7 David Williams CLA 2012-04-04 09:55:02 EDT
(In reply to comment #6)
> (In reply to comment #5)

> 
> Yes, the 3.8 repos weren't tagged with I20120321-0610 or even the 3.8 I build
> tag.  

And the solution is left as an exercise for the reader? :)
Comment 8 Paul Webster CLA 2012-04-04 10:26:07 EDT
(In reply to comment #7)
> 
> And the solution is left as an exercise for the reader? :)

I was hoping to keep it a secret, but ... :-)

It'll go away after there's at least one complete auto-tagging build, as all of the repos will be tagged with the last build input that should be in the Ibuild.properties file.

Option 2: figure out the build input commits that were used for the last 3.8 I build, and tag them with I20120321-0610 to give autotagging/report generation step a good starting state.

PW
Comment 9 David Williams CLA 2012-04-04 11:52:05 EDT
(In reply to comment #8)
> (In reply to comment #7)
> 
> It'll go away after there's at least one complete auto-tagging build, as all of
> the repos will be tagged with the last build input that should be in the
> Ibuild.properties file.
> 
> Option 2: figure out the build input commits that were used for the last 3.8 I
> build, and tag them with I20120321-0610 to give autotagging/report generation
> step a good starting state.
> 

How about a compromise (which mostly just tests my understanding of what you are saying and (poor) understand of git and the way the platform team works) ... I could a note to eclipse-dev (or platform-releng-dev?)  saying that if teams leads wanted an accurate "first run", they should work off the last 3.8 I build commits, (its tagged I20120320-1400, right?) and they they should also tag it with the last 4.2 I-build tag (I20120321-0610). Say, by noon Thursday. If they do, fine, the first "auto tagged run" will make more sense? But, if not, that's fine too and the second auto-tagged run will make sense? 

Is that a correct interpretation? Think that's overkill and we should just turn on and run it twice? (Or, do you know some "git magic" where _you_ could add I20120321-0610 tag to things that had 20120320-1400 tag?)
Comment 10 John Arthorne CLA 2012-04-04 12:38:36 EDT
I'd like to avoid asking committers to manually tag. It is error prone and most likely the tag they come up with won't match the tags computed by auto-tagging, which could cause some bundles to decrement version between builds. Clearly I'm not understanding something but I thought no matter what the current state we could just run the tagging scripts (either manually or scheduled) to get the map files into shape.
Comment 11 David Williams CLA 2012-04-04 13:20:45 EDT
(In reply to comment #10)
> ... Clearly I'm
> not understanding something but I thought no matter what the current state we
> could just run the tagging scripts (either manually or scheduled) to get the
> map files into shape.

Well, if anyone is not understanding, it's probably me :) 
Maybe Paul was just talking about the "Error messages" but would not have in practical impact. 

John, did you finish coordinated "master" and R4_HEAD branches as you mentioned yesterday? If so, I'm fine "turning it completely on" and seeing what it does.
Comment 12 Paul Webster CLA 2012-04-04 13:55:17 EDT
(In reply to comment #11)
> Well, if anyone is not understanding, it's probably me :) 
> Maybe Paul was just talking about the "Error messages" but would not have in
> practical impact. 

Correct, it has no practical bearing on the auto-tagging/updating the map files.  It only effects what's included in the email from comment #4 ... the ones that were listed were repos that were tagged with I20120321-0610 (which are only the repos actually built as part of the last 4.2 short build).

The first real auto-tagging should tag all of the repos with the new build id, and from then on, the report should include deltas from build to build.

PW
Comment 13 Paul Webster CLA 2012-04-04 14:04:46 EDT
(In reply to comment #4)
> FYI, I got this working enough to produce output! 
> 
> The main *.txt files are in .../eclipse4/build/supportDir if anyone (Paul)
> knows how to tell if they look right. 
> 
> I disabled the final "push and tag" so as to not touch the actual maps repo. 
> 

I had a quick look through the files and the git repos in gitCache, and they all seem quite reasonable.

PW
Comment 14 John Arthorne CLA 2012-04-04 15:01:18 EDT
(In reply to comment #11)
> John, did you finish coordinated "master" and R4_HEAD branches as you mentioned
> yesterday? If so, I'm fine "turning it completely on" and seeing what it does.

Yes that is finished (see bug 375993 for commit pointers). It was a massive merge so it is possible I missed something but if we do a tagged build and can do a sanity check.
Comment 15 David Williams CLA 2012-04-04 23:35:53 EDT
FYI. I tried "turning on" autotagging, and got the error below. 

My guess is this "doc" is missing from 
gitroot/platform/eclipse.platform.common.git R4_HEAD
 

BUILD FAILED
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:51: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:225: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:91: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/master/customTargets.xml:18: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/master/allElements.xml:16: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/genericTargets.xml:111: No custom build file found in /opt/public/eclipse/eclipse4/build/supportDir/src/plugins/org.eclipse.jdt.doc.user/build.xml. If you want to generate a new build file, temporarily disable the 'custom' property.
Comment 16 David Williams CLA 2012-04-05 00:40:56 EDT
Created attachment 213613 [details]
showing "missing" doc bundles in R4_HEAD

Well, it's pretty clear the JDT doc bundles have not been branched into R4_HEAD, and try as I might, I couldn't figure out how to do it. (not even sure I have commit rights there) ... but ... seemed everything I could find was explaining how to "branch a repository" and nothing about how to "fix up" one or two projects that are not in that branch? 

Perhaps there's better solutions? 

Git experts?
Comment 17 David Williams CLA 2012-04-05 02:12:19 EDT
adding Dani to CC. Dani, any ideas about those JDT doc bundle in R4_HEAD branch? Were they there and then deleted somehow? Recently? Or ... just never merged in?
Comment 18 David Williams CLA 2012-04-05 08:48:33 EDT
(In reply to comment #17)
> ... Were they there and then deleted somehow? Recently? Or ... just never
> merged in?

I'm pretty sure something changed. Not sure if just coincidence to the tagging tool running, or tagging tool somehow to blame? But now even completely disabling "tagRepo" the same failure occurs, "No custom build file found in
/opt/public/eclipse/eclipse4/build/supportDir/src/plugins/org.eclipse.jdt.doc.user/build.xml." Which was not the case early yesterday.
Comment 19 Dani Megert CLA 2012-04-05 08:49:07 EDT
(In reply to comment #17)
> adding Dani to CC. Dani, any ideas about those JDT doc bundle in R4_HEAD
> branch? Were they there and then deleted somehow? Recently? Or ... just never
> merged in?

This happened during the migration (probably on purpose), because so far the 4.2 build consumed the common stuff like JDT and PDE from the 3.8 build. JDT and PDE won't split streams, so, the build should simply consume what's in 'master' for N-builds and what's in 'integration' for I-builds.

BTW, the JDT doc bundles aren't completely empty. It's mainly the missing '.project' file, that gives you the closed projects. For details see e.g.
http://git.eclipse.org/c/platform/eclipse.platform.common.git/tree/bundles/org.eclipse.jdt.doc.isv?h=R4_HEAD
Comment 20 Dani Megert CLA 2012-04-05 08:51:44 EDT
This is e.g. the auto-generated commit (during the migration) that deleted the stuff in JDT ISV:
http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/bundles/org.eclipse.jdt.doc.isv/.project?h=R4_HEAD&id=df2ab66493df29123925add698b2964fa786bc27

Hard to tell whether this was a bug in the migration or happened because the migration script told to do so.
Comment 21 David Williams CLA 2012-04-05 09:20:20 EDT
(In reply to comment #19)
> (In reply to comment #17)

> JDT
> and PDE won't split streams, so, the build should simply consume what's in
> 'master' for N-builds and what's in 'integration' for I-builds.
> 

Paul or John may know better, but since all docs are in eclipse.platform.common I would think once you are happy with "master" you need to merge that into integration, and into R4_HEAD, since the repositories.txt file uses R4_HEAD for that repository, for 4.2 builds.
Comment 22 Dani Megert CLA 2012-04-05 09:23:59 EDT
(In reply to comment #21)
> (In reply to comment #19)
> > (In reply to comment #17)
> 
> > JDT
> > and PDE won't split streams, so, the build should simply consume what's in
> > 'master' for N-builds and what's in 'integration' for I-builds.
> > 
> 
> Paul or John may know better, but since all docs are in eclipse.platform.common
> I would think once you are happy with "master" you need to merge that into
> integration, and into R4_HEAD, since the repositories.txt file uses R4_HEAD for
> that repository, for 4.2 builds.

Just change that entry to point to 'integration' for JDT and PDE doc.
Comment 23 David Williams CLA 2012-04-05 09:31:12 EDT
(In reply to comment #22)
> (In reply to comment #21)
> > (In reply to comment #19)
> > > (In reply to comment #17)

> Just change that entry to point to 'integration' for JDT and PDE doc.

I'm still learning, but there is only one entry: 

.../gitroot/platform/eclipse.platform.common.git R4_HEAD

I think the "auto tagging" works repo by repo?
Comment 24 Paul Webster CLA 2012-04-05 09:34:00 EDT
(In reply to comment #23)
> .../gitroot/platform/eclipse.platform.common.git R4_HEAD
> 
> I think the "auto tagging" works repo by repo?

John and I are looking at this.  We've pushed new commits to eclipse.platform.git and eclipse.platform.common.git R4_HEAD that take all non-4.2 bundles from origin/master for the moment.

We're looking at eclipse.platform.releng.git now.

PW
Comment 25 Dani Megert CLA 2012-04-05 09:34:43 EDT
(In reply to comment #24)
> (In reply to comment #23)
> > .../gitroot/platform/eclipse.platform.common.git R4_HEAD
> > 
> > I think the "auto tagging" works repo by repo?

Ooops!
Comment 26 Paul Webster CLA 2012-04-05 10:03:03 EDT
(In reply to comment #24)
> 
> We're looking at eclipse.platform.releng.git now.

We've updated eclipse.platform.releng.git R4_HEAD.

You'd have to do another auto-tagging before your next build attempt.

PW
Comment 27 David Williams CLA 2012-04-05 10:20:07 EDT
(In reply to comment #26)
> (In reply to comment #24)

> 
> You'd have to do another auto-tagging before your next build attempt.
> 

Jut to be clear, are you saying something would "break" if I did a build attempt before auto-tagging? Or just that if I didn't do it first, the first build would not be accurate, but the subsequent one would be?
Comment 28 Paul Webster CLA 2012-04-05 10:57:44 EDT
(In reply to comment #27)
> 
> Jut to be clear, are you saying something would "break" if I did a build
> attempt before auto-tagging? 

yes, it would still be broken without having autotagging update the map files.

PW
Comment 29 David Williams CLA 2012-04-05 11:16:41 EDT
(In reply to comment #28)
> (In reply to comment #27)

> yes, it would still be broken without having autotagging update the map files.

Not that I doubt you, but for my education ... the "tagRepo" normally runs _before_ the build ... so, I'm wonder what it is that would cause the build to break "if maps not updated first". I think I'm missing some concept, and want to be sure I understand.
Comment 30 Paul Webster CLA 2012-04-05 11:40:07 EDT
(In reply to comment #29)
> (In reply to comment #28)
> > (In reply to comment #27)
> 
> > yes, it would still be broken without having autotagging update the map files.
> 
> Not that I doubt you, but for my education ... the "tagRepo" normally runs
> _before_ the build ...

Sorry, I forgot that we re-enabled that steps as part of the build.  So autotagging will be run at the beginning of the build, and that's fine.

Sorry for the confusion :-)

PW
Comment 31 David Williams CLA 2012-04-05 11:56:25 EDT
(In reply to comment #30)
> (In reply to comment #29)
> > (In reply to comment #28)
> > > (In reply to comment #27)
> 
> Sorry, I forgot that we re-enabled that steps as part of the build.  So
> autotagging will be run at the beginning of the build, and that's fine.
> 

Well, it comes and goes :) 

But, I have done the "manual" step, and if anyone wants to sanity check, results here: 

http://git.eclipse.org/c/platform/eclipse.platform.releng.maps.git/commit/?id=a8ebfe9112a68b0327e1af2f8b89a14f2ceabdfd 

And again, not that I doubt, you, but I think you are saying that of the 5 projects in eclipse.platform.common, its fine if some of them "don't exist" in 
R4_HEAD branch (no .project file) ... that some of those projects are autotagged, based on contents of R4_HEAD, and some not? Still magic to me.
Comment 32 Paul Webster CLA 2012-04-05 12:32:19 EDT
(In reply to comment #31)
> 
> And again, not that I doubt, you, but I think you are saying that of the 5
> projects in eclipse.platform.common, its fine if some of them "don't exist" in 
> R4_HEAD branch (no .project file) ... that some of those projects are
> autotagged, based on contents of R4_HEAD, and some not? Still magic to me.

It used to be fine that they weren't in R4_HEAD, when we did the 4.2 short build.

Now, however, all repos with an R4_HEAD are split repos.  They repo can only offer up one branch at auto-tagging time/build time, so all of the content has to be in R4_HEAD.  The last changes that John and I made should have made that happen (no missing .project files).

PW
Comment 33 David Williams CLA 2012-04-05 14:31:34 EDT
Getting further ... but now fails with message below (and, this is, I think, after its already built "the master feature"?)

Its large, if necessary, but you can see full output in 
/shared/eclipse/eclipse4/fullmasterBuildOutput.txt


I see this map entry in jdtcore.map

plugin@org.eclipse.jdt.core.tests.binaries=GIT,tag=v20120112-1458,repo=git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.binaries.git,path=org.eclipse.jdt.core.tests.binaries

But, do not see tag v20120112-1458 in the git repo. 
Could it be this merely needs to be updated to "most recent" tag? 
(v201204...something) 

Oh, I just noticed the most recent is versioned as 
1.0.1.qualifier (not 1.0.0) if that matters. 

I'm guess this is a part of the build that is creating "test packages" ... but, honestly, do not know what its trying to do. 

= = = = 

generateScript:
[eclipse.buildScript] Some inter-plug-in dependencies have not been satisfied.
[eclipse.buildScript] Bundle org.eclipse.jdt.apt.tests:
[eclipse.buildScript]   Missing required plug-in org.eclipse.jdt.core.tests.binaries_1.0.0.
[eclipse.buildScript] Bundle org.eclipse.jdt.core.tests.performance:
[eclipse.buildScript]   Missing required plug-in org.eclipse.jdt.core.tests.binaries_1.0.0.

BUILD FAILED
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:59: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:197: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/build.xml:35: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:35: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:91: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/sdk.tests/customTargets.xml:18: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/sdk.tests/allElements.xml:16: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/genericTargets.xml:111: Processing inclusion from feature org.eclipse.sdk.tests: Bundle org.eclipse.jdt.apt.tests_3.3.400.v20120403-0538 failed to resolve.:
        Missing required plug-in org.eclipse.jdt.core.tests.binaries_1.0.0.
Comment 34 Paul Webster CLA 2012-04-05 14:38:08 EDT
(In reply to comment #33)
> I see this map entry in jdtcore.map
> 
> plugin@org.eclipse.jdt.core.tests.binaries=GIT,tag=v20120112-1458,repo=git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.binaries.git,path=org.eclipse.jdt.core.tests.binaries
> 
> But, do not see tag v20120112-1458 in the git repo. 
> Could it be this merely needs to be updated to "most recent" tag? 
> (v201204...something) 

in eclipse4/build/supportDir/gitCache/eclipse.jdt.core.binaries I see the tag v20120112-1458 from the last time Olivier changed the binaries, and it's in the public repo as well.

I'll look at the build output.

PW
Comment 35 Paul Webster CLA 2012-04-05 14:47:31 EDT
(In reply to comment #33)
> 
> I'm guess this is a part of the build that is creating "test packages" ... but,
> honestly, do not know what its trying to do. 
> 
> = = = = 
> 
> generateScript:
> [eclipse.buildScript] Some inter-plug-in dependencies have not been satisfied.
> [eclipse.buildScript] Bundle org.eclipse.jdt.apt.tests:
> [eclipse.buildScript]   Missing required plug-in
> org.eclipse.jdt.core.tests.binaries_1.0.0.
> [eclipse.buildScript] Bundle org.eclipse.jdt.core.tests.performance:
> [eclipse.buildScript]   Missing required plug-in
> org.eclipse.jdt.core.tests.binaries_1.0.0.
> 

This is building from eclipse.platform.releng/features/org.eclipse.sdk.tests

We didn't take what was in master because we'd made changes for R4_HEAD.

But that's the feature that should include the binaries plugin, without that PDE won't try and build it.

Let met try and merge it.

PW
Comment 36 Paul Webster CLA 2012-04-05 14:57:23 EDT
I've merged in the changes into R4_HEAD.  It now includes more tests:


org.eclipse.equinox.ds.tests
org.eclipse.equinox.region.tests
org.eclipse.jdt.core.tests.binaries
org.eclipse.equinox.bidi.tests



PW
Comment 37 David Williams CLA 2012-04-05 15:07:20 EDT
(In reply to comment #36)
> I've merged in the changes into R4_HEAD.  It now includes more tests:
> 
> 
> org.eclipse.equinox.ds.tests
> org.eclipse.equinox.region.tests
> org.eclipse.jdt.core.tests.binaries
> org.eclipse.equinox.bidi.tests
> 

Thanks Paul, I would never have figured that out, but can now see a little more how it works. ... I just hope we don't end up with EVERYTHING in R4_HEAD :)
Comment 38 David Williams CLA 2012-04-05 16:04:29 EDT
Am I seeing this right ... exact same error when ran again? 


generateScript:
[eclipse.buildScript] Some inter-plug-in dependencies have not been satisfied.
[eclipse.buildScript] Bundle org.eclipse.jdt.apt.tests:
[eclipse.buildScript]   Missing required plug-in org.eclipse.jdt.core.tests.binaries_1.0.0.
[eclipse.buildScript] Bundle org.eclipse.jdt.core.tests.performance:
[eclipse.buildScript]   Missing required plug-in org.eclipse.jdt.core.tests.binaries_1.0.0.

BUILD FAILED
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:59: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:197: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/build.xml:35: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:35: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:91: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/sdk.tests/customTargets.xml:18: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/sdk.tests/allElements.xml:16: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/genericTargets.xml:111: Processing inclusion from feature org.eclipse.sdk.tests: Bundle org.eclipse.jdt.apt.tests_3.3.400.v20120403-0538 failed to resolve.:
        Missing required plug-in org.eclipse.jdt.core.tests.binaries_1.0.0.
Comment 39 Paul Webster CLA 2012-04-05 16:13:33 EDT
(In reply to comment #38)
> Am I seeing this right ... exact same error when ran again? 
> 

It doesn't look like auto-tagging ran, as it's not picking up my changes from R4_HEAD compared to the second commit:


bash$ git log -2 --decorate 
commit e844e766a0ede31cc8f2072a521681adbfaca375 (HEAD, origin/R4_HEAD, R4_HEAD)
Author: Paul Webster <pwebster@ca.ibm.com>
Date:   Thu Apr 5 14:55:22 2012 -0400

    Bug 375807 - Need to get auto tagging working on 4.2 primary build
    
    Added test plugins for master that weren't included in 4.2 before.

commit 20195c47cff14f09af83a9aeac1c3b95aa4d8da6 (tag: v20120405-1401, tag: I20120405-1114)
Author: Paul Webster <pwebster@ca.ibm.com>
Date:   Thu Apr 5 10:01:28 2012 -0400

PW
Comment 40 David Williams CLA 2012-04-05 16:19:31 EDT
(In reply to comment #39)
> (In reply to comment #38)
> > Am I seeing this right ... exact same error when ran again? 
> > 
> 
> It doesn't look like auto-tagging ran, as it's not picking up my changes from
> R4_HEAD compared to the second commit:
> 
> 

Ok, thanks. I'll check that part of the log/script .. probably still buggy. 

Thanks,
Comment 41 David Williams CLA 2012-04-05 19:54:36 EDT
Created attachment 213688 [details]
end of log from latest failure

The fun never stops ... 

Its now getting to one of the final steps, and getting error message related to a p2Directory "security exception"  for all the .source jars from ecf. Since "source" files, I'm suspecting this is a pack200 problem. Not sure on which end ... but, this is the kind of error I know all too well how to investigate.
Comment 42 Ian Bull CLA 2012-04-05 20:12:24 EDT
(In reply to comment #41)
> Its now getting to one of the final steps, and getting error message related to
> a p2Directory "security exception"  for all the .source jars from ecf. Since
> "source" files, I'm suspecting this is a pack200 problem. Not sure on which end
> ... but, this is the kind of error I know all too well how to investigate.

Kim and I updated ECF the week before EclipseCon (and I don't think we ever got a build out). Maybe that's related.  However, the test build you had early this week appeared to have the newer ECF.

Another thing that may cause Security Exceptions are line endings between platforms. I know we checked in some files (back the CVS days) on one platform (windows) and we got security exceptions on linux.  

Hope that helps (I'm sure it didn't) ;-).
Comment 43 David Williams CLA 2012-04-05 21:44:52 EDT
(In reply to comment #42)
> (In reply to comment #41)

> ... However, the test build you had early this
> week appeared to have the newer ECF.

Well, we weren't signing then. We are signing now (Yay!, some successes :) 

My first thought ECF had changed the way they pack200 their jars ... but ... the script I had inherited did oddly have "java15home" pointing to a 1.6 JDK which would also lead to pack200 problem in .source jars. I'm a little surprised "we" would be conditioning and signing ECF's jars to begin with ... but for now, I'll assume that's what was agreed to and the way its supposed to work. 

Using a java 1.5 JDK to do our own conditioning is the right thing to do anyway, so worth just trying that first and see if that fixes the ECF issue too. We'll know in a few hours :)
Comment 44 David Williams CLA 2012-04-06 09:20:04 EDT
(In reply to comment #43)
> (In reply to comment #42)
> > (In reply to comment #41)
> 

> Using a java 1.5 JDK to do our own conditioning is the right thing to do
> anyway, so worth just trying that first and see if that fixes the ECF issue
> too. We'll know in a few hours :)

After several "trial and error" failures (each taking several hours to fail), I've decided I'm actually going to have to study the issue! (that is, develop some small tests to figure out what's going on). So, in the meantime, I'll disable "signing" for now, just to get a complete test build, with auto tagging enabled. I'll document signing (and "conditioning" and "packing") issues in bug 376032.
Comment 45 David Williams CLA 2012-04-06 14:02:12 EDT
As far as I can tell, "autotagging" is working as expected. 

The "messages" (emailed to me only, at the moment) say "nothings changed" since ... nothing would have since the last test build. 

Anyone around to "commit" a "real" thing or two? If you do, let me know and I'll post "mail message" here to make sure it matches what you expected. 



To see the latest "good" test build, see 
http://build.eclipse.org/eclipse/eclipse4/build/supportDir/I20120406-0935/

I did a "diff" with an "install" from "last known I-build" if anyone can tell from that if it looks right. 
http://build.eclipse.org/eclipse/eclipse4/diffrefout.txt
Comment 46 David Williams CLA 2012-04-07 00:55:51 EDT
Now I am hitting an odd problem ... 


fetchElement:
    [mkdir] Created dir: /shared/eclipse/eclipse4/build/supportDir/src/features
    [mkdir] Created dir: /shared/eclipse/eclipse4/build/supportDir/src/plugins

main:
      [cvs] cvs export: CVSROOT must be an absolute pathname (not `tag=v20120405-1401')
      [cvs] cvs export: when using local access method.
      [cvs] cvs [export aborted]: Bad CVSROOT: `tag=v20120405-1401'.

BUILD FAILED
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:51: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:214: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:78: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/sdk.examples/customTargets.xml:9: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/genericTargets.xml:56: Could not retrieve feature.xml or build.properties for feature org.eclipse.sdk.examples.

Not sure if this was from some previous "bad run" of autotagging" or ... it almost seems like some piece of code is executing that should not be executing?
Comment 47 David Williams CLA 2012-04-07 02:19:00 EDT
(In reply to comment #46)
> Now I am hitting an odd problem ... 

>       [cvs] cvs export: CVSROOT must be an absolute pathname (not
> `tag=v20120405-1401')
>       [cvs] cvs export: when using local access method.
>       [cvs] cvs [export aborted]: Bad CVSROOT: `tag=v20120405-1401'.
> 

This appeared to be from a corrupt ~/.cvspass file, so I just deleted its contents.
Comment 48 David Williams CLA 2012-04-07 13:23:45 EDT
(In reply to comment #47)
> (In reply to comment #46)
> > Now I am hitting an odd problem ... 
> 
> >       [cvs] cvs export: CVSROOT must be an absolute pathname (not
> > `tag=v20120405-1401')
> >       [cvs] cvs export: when using local access method.
> >       [cvs] cvs [export aborted]: Bad CVSROOT: `tag=v20120405-1401'.
> > 
> 
> This appeared to be from a corrupt ~/.cvspass file, so I just deleted its
> contents.

This problem keeps "coming back". I think we (or, I) have changed just enough that the "new way" (autotagging) is conflicting with something in "the old way" (or, vice versa). I'm seeing stuff in the customTargets.xml that is still (trying) to do stuff with map files, that I think it should no longer be doing. For example, see 

		<!--compare the map files project-->
		<antcall target="compareMapFiles" />
		<!--tag the map files project-->
		<antcall target="tagMapFiles" />

So, firstly, that might have actually "messed up" some tags (from one run to the next). And, secondly, I think easiest way forward might be to "rip all that out" of customTargets.xml. 

Conceptually, I suppose, it might _make_ sense for customTargets.xml to "refetch" map files the auto-tagger has auto-tagged, but, at most, that'd be it. I'm not even sure that's necessary right now, but handy for the point when we want to "reproduce a build"? 

I'm sure I'm missing something conceptually, but ... unless someone says some thing soon [I know, its a holiday weekend, so may be on my own here :) ]  I may (continue) "ripping out" anything cvs or map related from customTargets.xml, get it working from the maps the auto-tagger has already retrieved ... 

or, at most, simply retrieve what its told to retrieve by "auto-tagger".

If anyone knows better, let me know. But, as far as I can tell, the code is "trying to live in two worlds" (or, three, or four worlds) and a) is confusing, and b) unless someone knows that they are doing :) one world can easily stomp on the other.
Comment 49 David Williams CLA 2012-04-07 17:05:53 EDT
So, the "fetch file" being created says, in part


                <cvspass cvsRoot="tag=v20120405-1401" password="repo=git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git"                />


The fun never ends :)
Comment 50 David Williams CLA 2012-04-08 03:19:24 EDT
(In reply to comment #49)
> So, the "fetch file" being created says, in part
> 
> 
>                 <cvspass cvsRoot="tag=v20120405-1401"
> password="repo=git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git"
>                />
> 
> 
> The fun never ends :)

I think was caused by trying to run the builder under Java 1.5. If Java 1.6 i required, that's fine ... but ... something else must be wrong for it it get so far and end so badly, so I opened bug 376291 as well as bug 376292.

Either that, or it very sensitive to map tags that start with v instead of I, which seems unlikely.
Comment 51 David Williams CLA 2012-04-08 03:20:30 EDT
Next (new) failure. (But this one does sounds like something I might have caused). 


BUILD FAILED
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:51: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/buildAll.xml:220: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/build.xml:91: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/master/customTargets.xml:18: The following error occurred while executing this line:
/shared/eclipse/eclipse4/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/master/allElements.xml:16: The following error occurred while executing this line:
/opt/public/eclipse/eclipse4/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/genericTargets.xml:111: Processing inclusion from feature org.eclipse.equinox.p2.core.feature: Unable to find plug-in: org.eclipse.equinox.security.win32.x86_64_0.0.0. Please check the error log for more details.
Comment 52 David Williams CLA 2012-04-08 03:26:47 EDT
(In reply to comment #51)
> Next (new) failure. (But this one does sounds like something I might have
> caused). 
> 
Well, maybe not, there seems to be two version of this bundle in map files, and is described in bug 338925. 

I'll just try commenting out the one that says "temporary fix".
Comment 53 David Williams CLA 2012-04-09 21:58:37 EDT
This is working as expected, I think ... one open question is if there have been no changes, then we should also not tag the maps directory (right?). 

And, normally, we'd want to not build, either ... but ... we'd want an "override" to keep building, even if no changes, for when we are testing the build, etc. 

As I noted in bug 366279 comment 3, the following "commit" returns "1" ("no changes, nothing to commit"). I suggest we use that as a sign there's been no changes, no need to tag and (unless "forceBuildOnNoChange") we'd not build and send a nice message to mailing list saying build was canceled since no changes found. 

I'll head down that path, but let me know if anyone has any better ideas.
Comment 54 David Williams CLA 2012-04-10 01:27:42 EDT
(In reply to comment #53)

> 
> As I noted in bug 366279 comment 3, the following "commit" returns "1" ("no
> changes, nothing to commit"). I suggest we use that as a sign there's been no
> changes, no need to tag and (unless "forceBuildOnNoChange") we'd not build and
> send a nice message to mailing list saying build was canceled since no changes
> found. 

Well, not so sure. I've seen "1" twice, and "159" once ... not sure what the difference could be, but may not be all that reliable. The docs usually say "non zero is returned on errors" and stackoverflow posts "it is usually '1'" ... but ... guess it is not well specified, that I can find so far. So I changed logic that if its "non zero" to assume "nothing to commit" (and hence, nothing to tag, etc.) ... we'll see how it works out, I guess and tweak as we learn more.
Comment 55 David Williams CLA 2012-04-10 03:07:24 EDT
I think this is working better than I thought (more consistently) but I just need to get straight on bash variables, string, numbers, tests, exits, and returns. 

So, for now, I've removed all the "error checking" so we can get out a complete build on Tuesday.
Comment 56 Paul Webster CLA 2012-04-10 08:26:28 EDT
(In reply to comment #53)
> This is working as expected, I think ... one open question is if there have
> been no changes, then we should also not tag the maps directory (right?). 
> 
> And, normally, we'd want to not build, either ... but ... we'd want an
> "override" to keep building, even if no changes, for when we are testing the
> build, etc. 

If there's really been no change, by default we used to abort the build.  That would solve the problem over simply pushing tags (just stop).

And for running test builds you would want to run with -noTag so that there were no tags anyway.

PW
Comment 57 David Williams CLA 2012-04-11 03:48:56 EDT
I'd like to count this particular bug as "fixed" because it works well enough to produce an I build. 

Some of the "fine tunning" and improvements in mail messages will be traced in bug 376460 and others.
Comment 58 David Williams CLA 2012-04-16 23:45:47 EDT
In doing a "practice I build" in preparation for one on Tuesday, I noticed that the "report.txt" file produced by auto-tagging seemed to have some "stray" directories in the list of projects/repos (see below). Think this is cause of concern? Something I'm doing wrong? Just a quirk you've seen before? 

= = = = =

The build contains the following changes:
+ Bug 98062 - PDE text models don&apos;t honor line delimiter preferences (FIXED)
+ Bug 304066 - [flex] Race conditions in processing of delta and content updates. (FIXED)
+ Bug 344181 - [Compatibility] Not all emacs keybindings override default keybindings (REOPENED)
+ Bug 355763 - Provide tag which prohibits a PartStack to collapse when empty (FIXED)
+ Bug 356209 - [Compatibility] HandlerUtil.getActiveShell gives me always the parent (Workbench) Shell (ASSIGNED)
+ Bug 361389 - API Tools quick fix changes all line delimiters in .api_filters (FIXED)
+ Bug 367920 - Perspective view layout not controllable on a per-perspective basis. (NEW)
+ Bug 374096 - Cheese in the manifest editor (FIXED)
+ Bug 374153 - [patch][breakpoints] Show accelerator for toggle breakpoint modifiers in ruler popup menu. (FIXED)
+ Bug 374483 - [Compatibility] NPE while Debug is trying to switch perspectives with a dead perspective around (FIXED)
+ Bug 375744 - Command-Q does not ask to confirm exit (FIXED)
+ Bug 376155 - Remove dummy toolbar for fullscreen support in Lion (FIXED)
+ Bug 376354 - [regression] Cannot set breakpoint on final inner type (FIXED)
+ Bug 376443 - [expressions] Adding multiple expressions to view can leave the &quot;Add New Expression&quot; at top. (FIXED)
+ Bug 376818 - Accidental revert of previous commit (FIXED)

The following projects have changed:
build.properties
.classpath
META-INF
model
org.eclipse.e4.ui.model.workbench
org.eclipse.e4.ui.workbench
org.eclipse.e4.ui.workbench.addons.swt
org.eclipse.e4.ui.workbench.renderers.swt
org.eclipse.e4.ui.workbench.renderers.swt.cocoa
org.eclipse.pde.api.tools
org.eclipse.pde.core
org.eclipse.pde.runtime
org.eclipse.pde.ui
org.eclipse.ui.workbench
plugin.properties
plugin.xml
.settings
src
testprograms
tests
ui
Comment 59 Dani Megert CLA 2012-04-17 05:43:01 EDT
> Think this is cause of concern?
Yes, I think so too. It should only list projects, not files/folders.

> Just a quirk you've seen before? 
I can't remember having seen this.
Comment 60 Paul Webster CLA 2012-04-17 07:29:46 EDT
(In reply to comment #58)
> In doing a "practice I build" in preparation for one on Tuesday, I noticed that
> the "report.txt" file produced by auto-tagging seemed to have some "stray"
> directories in the list of projects/repos (see below). Think this is cause of
> concern? Something I'm doing wrong? Just a quirk you've seen before?

We haven't really seen this before because the 3.8 I builds weren't sending out the report  email regularly.

> testprograms
> tests
> ui

Looks like it's this subdirectory:

eclipse.jdt.debug/org.eclipse.jdt.debug.tests/testprograms

It's this line in the script 
git diff --name-only ${LAST_TAG} ${BUILD_TAG} | cut -f2 -d/ | sort -u >>/tmp/proj_changed_$$.txt

it works for repos like eclipse.platform.ui and eclipse.pde.ui but not for the JDT repos.

PW
Comment 61 David Williams CLA 2012-04-17 09:49:01 EDT
(In reply to comment #60)
> (In reply to comment #58)

> 
> We haven't really seen this before because the 3.8 I builds weren't sending out
> the report  email regularly.
> 
> > testprograms
> > tests
> > ui
> 
> Looks like it's this subdirectory:
> 
> eclipse.jdt.debug/org.eclipse.jdt.debug.tests/testprograms
> 
> It's this line in the script 
> git diff --name-only ${LAST_TAG} ${BUILD_TAG} | cut -f2 -d/ | sort -u
> >>/tmp/proj_changed_$$.txt
> 
> it works for repos like eclipse.platform.ui and eclipse.pde.ui but not for the
> JDT repos.
> 
> PW

So ... would cause cause the "merge failures" like I saw this morning?
Or, should I just open another bug report to improve this part of the auto-tagging script? 

We'll know in about 10 minutes if this is the cause of the merge failures :/
Comment 62 Paul Webster CLA 2012-04-17 09:59:38 EDT
(In reply to comment #61)
> 
> So ... would cause cause the "merge failures" like I saw this morning?
> Or, should I just open another bug report to improve this part of the
> auto-tagging script? 
> 
> We'll know in about 10 minutes if this is the cause of the merge failures :/

No, this isn't the merge failures, that's just an artifact of how we do the push.  I've commented in bug 366279

PW
Comment 63 David Williams CLA 2012-04-17 11:27:17 EDT
(In reply to comment #60)
> (In reply to comment #58)

> 
> Looks like it's this subdirectory:
> 
> eclipse.jdt.debug/org.eclipse.jdt.debug.tests/testprograms
> 
> It's this line in the script 
> git diff --name-only ${LAST_TAG} ${BUILD_TAG} | cut -f2 -d/ | sort -u
> >>/tmp/proj_changed_$$.txt
> 
> it works for repos like eclipse.platform.ui and eclipse.pde.ui but not for the
> JDT repos.
> 

To cross reference, I've opened bug 376988 to disuss "cleanup" of the reports.

At least, I'm assuming we can fix in our scripts, and jdt doesn't have to change their repos :/
Comment 64 David Williams CLA 2012-04-17 14:07:13 EDT
I am not sure there is a problem here, but thought I would document what I've seen in case others can tell. 

I'm wondering if this part of our script is working as expected, or doing double work? 

cat clones.txt| xargs /bin/bash git-map.sh $gitCache $buildTag \
        $relengRepo > maps.txt


I ask because by chance I was checking "running processes" during a test N build, essentially doing this 

ps -ef | grep eclipse4N

and happened to see two processes that appear to be doing the identical thing?

[And note, I know during an normal N build, it would not even be executing this part of the code, but ... as part of my "tests" I have it run through the first part of autotagging ... to test/debug]. 

I'll paste in all the processes I saw running, but it is the ones with 

/bin/bash git-map.sh 

that I'm curious about. They seem identical to me. 

  PID USER         ELAPSED  CPU  SIZE   RSS    VSZ COMMAND

 8395 e4Build        00:00  0.0   448   576  12748 /bin/bash git-map.sh /shared/eclipse/eclipse4N/build/supportDir/gitCache N20120417-1307 /shared/eclipse/eclipse4N/build/supportDir/gitCache/eclipse.platform.releng.maps git://git.eclipse.org/gitroot/equinox/rt.equinox.bundles.git git://git.eclipse.org/gitroot/equinox/rt.equinox.framework.git git://git.eclipse.org/gitroot/equinox/rt.equinox.incubator.git git://git.eclipse.org/gitroot/equinox/rt.equinox.p2.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.binaries.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.debug.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.ui.git git://git.eclipse.org/gitroot/pde/eclipse.pde git://git.eclipse.org/gitroot/pde/eclipse.pde.build.git git://git.eclipse.org/gitroot/pde/eclipse.pde.ui.git git://git.eclipse.org/gitroot/platform/eclipse.platform.debug.git git://git.eclipse.org/gitroot/platform/eclipse.platform.resources.git git://git.eclipse.org/gitroot/platform/eclipse.platform.git git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git git://git.eclipse.org/gitroot/platform/eclipse.platform.runtime.git git://git.eclipse.org/gitroot/platform/eclipse.platform.team.git git://git.eclipse.org/gitroot/platform/eclipse.platform.text.git git://git.eclipse.org/gitroot/platform/eclipse.platform.ua.git git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git

10907 e4Build        05:35  0.0   448  1732  12748 bash /shared/eclipse/eclipse4N/masterBuild.sh -buildType N -eclipseStream 4.2 -buildRoot /shared/eclipse/eclipse4N -mapVersionTag R4_HEAD

11026 e4Build        05:27  0.0   316  1664  12616 /bin/bash /shared/eclipse/eclipse4N/build/supportDir/org.eclipse.releng.eclipsebuilder/scripts/git-release.sh -mapVersionTag R4_HEAD -relengMapsProject org.eclipse.releng -relengRepoName eclipse.platform.releng.maps -buildType N -gitCache /shared/eclipse/eclipse4N/build/supportDir/gitCache -buildRoot /shared/eclipse/eclipse4N -gitEmail "e4Build" -gitName "e4Builder-R4" -timestamp 201204171307 -oldBuildTag 
N20120416-2000 -buildTag N20120417-1307 -submissionReportFilePath /shared/eclipse/eclipse4N/siteDir/eclipse/downloads/drops4/N20120417-1307/report.txt -tag false

14264 e4Build        04:23  0.0   532   692   5600 xargs /bin/bash git-map.sh /shared/eclipse/eclipse4N/build/supportDir/gitCache N20120417-1307 /shared/eclipse/eclipse4N/build/supportDir/gitCache/eclipse.platform.releng.maps

14265 e4Build        04:23  0.1   448  1676  12748 /bin/bash git-map.sh /shared/eclipse/eclipse4N/build/supportDir/gitCache N20120417-1307 /shared/eclipse/eclipse4N/build/supportDir/gitCache/eclipse.platform.releng.maps git://git.eclipse.org/gitroot/equinox/rt.equinox.bundles.git git://git.eclipse.org/gitroot/equinox/rt.equinox.framework.git git://git.eclipse.org/gitroot/equinox/rt.equinox.incubator.git git://git.eclipse.org/gitroot/equinox/rt.equinox.p2.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.binaries.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.debug.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.ui.git git://git.eclipse.org/gitroot/pde/eclipse.pde git://git.eclipse.org/gitroot/pde/eclipse.pde.build.git git://git.eclipse.org/gitroot/pde/eclipse.pde.ui.git git://git.eclipse.org/gitroot/platform/eclipse.platform.debug.git git://git.eclipse.org/gitroot/platform/eclipse.platform.resources.git git://git.eclipse.org/gitroot/platform/eclipse.platform.git git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git git://git.eclipse.org/gitroot/platform/eclipse.platform.runtime.git git://git.eclipse.org/gitroot/platform/eclipse.platform.team.git git://git.eclipse.org/gitroot/platform/eclipse.platform.text.git git://git.eclipse.org/gitroot/platform/eclipse.platform.ua.git git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git
Comment 65 Paul Webster CLA 2012-04-18 08:01:02 EDT
(In reply to comment #64)
> I am not sure there is a problem here, but thought I would document what I've
> seen in case others can tell. 
> 

It might be that the first one listed is a leftover process from another run?  If you had the PPIDs you could tell which process spawned what.

It also might be that xargs isn't appropriate way to launch that, since in theory it can run the command multiple times (although I don't believe that would cause a problem in this case).  Maybe we should change it just to be safe:

/bin/bash git-map.sh $gitCache $buildTag \
        $relengRepo $( cat clones.txt ) > maps.txt

PW
Comment 66 David Williams CLA 2012-04-19 15:23:21 EDT
For my education ... 

I'm trying a realistic test build of 3.8 I build, and put I20120320-1400 in the "Ibuilds.properties" files (the last 3.8 I build), and confirmed that there was a version of the maps project tagged as I20120320-1400. But, still the report came back as empty. 

At first, I thought maybe a sign of trouble, not now I'm wondering, is that because the projects code was not tagged with I20120320-1400 (since autotagging was enabled then?) 

If so, guess we just start from now? (and will be accurate in future?)
Comment 67 Paul Webster CLA 2012-04-19 16:36:09 EDT
(In reply to comment #66)
> At first, I thought maybe a sign of trouble, not now I'm wondering, is that
> because the projects code was not tagged with I20120320-1400 (since autotagging
> was enabled then?) 
> 
> If so, guess we just start from now? (and will be accurate in future?)

right.  The old 3.8 auto-tagging didn't tag the project repos, only the releng repos.  You can either tag each of the project repos before you generate the report, or just leave it and it will be correct going forward.

PW
Comment 68 David Williams CLA 2012-04-23 03:22:59 EDT
(In reply to comment #65)
> (In reply to comment #64)

> ...  Maybe we should change it just to be
> safe:
> 
> /bin/bash git-map.sh $gitCache $buildTag \
>         $relengRepo $( cat clones.txt ) > maps.txt
> 

I have now changed this code as you recommend above. It didn't seem like a case for xargs, you were/are expecting one invocation with a list you looped through in git-map.sh, where as, as I understand it, with xargs, you really do separate, multiple invocations of something ... and to be honest ... I have never ever been able to get xargs to work the way I think it should ... I think I do not understand it. I like getting rid of code I dont' understand :) [I did try, by the way, some experiments and I can not imagine how either form could have cause the process to be invoked twice ... so, maybe was just left over from previous run?  Or, maybe I had (have?) a bug where some larger section of code is called
twice. At any rate, I like this form better, so have committed/pushed.