Community
Participate
Working Groups
I've noticed quirks before that makes be wonder if the N-builds are really working as expected. And, now with but 377442 comment 5 think it deserves a bug to track cases ... and/or investigate systematically?
The code I changed, in buildAll.xml, as we made 4.2 the primary, from scratch build used to look like this (when 3.8 was primary): <condition property="fetchTag" value="CVS=HEAD,GIT=master;git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git=R3_development"> <equals arg1="${buildType}" arg2="N" /> </condition> For "from scratch" 4.2 builds, I changed to be as follows <!-- It will be a miracle of programming, if this works :) FYI, we shouldn't have anything in CVS? but ... will leave for now --> <condition property="fetchTag" value="CVS=HEAD,GIT=master;git://git.eclipse.org/gitroot/platform/eclipse.platform.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.maps.git=R4_HEAD"> <equals arg1="${buildType}" arg2="N" /> </condition> I originally wrote and tracked this in bug 376683. And, seemed to work, though I never really understood how. (never traced or dug deep enough to find "fetchTag"). I'm assuming some code somewhere does a search based on "specificity" and uses that. One possible complication, is that since then, I (we) have moved the "get map" task earlier than it used to be in order to get "stream specific" information early enough to be of most use. Maybe that caused the original fix to break somehow? (such as some deep code in base builder was looking to see if maps already existed, and if not, then checked for N and then checked for fetchTag?) I have never seen "fetchTag" itself used for anything in eclipsebuilder ... but search for it now, I see a similar condition in build.xml file, in eclipsebuilder, which I did not fix originally, but did probably a week ago (should use only first one in buildAll.xml, I think). <condition property="fetchTag" value="CVS=HEAD,GIT=master;git://git.eclipse.org/gitroot/platform/eclipse.platform.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.maps.git=R4_HEAD"> <equals arg1="${buildType}" arg2="N" /> </condition> Or ... any "typos" in above? I got this list form repositories.txt file. I'm assuming platform.ui is master, right? It is according to repositories.txt. Or, maybe what ever is processing "fetchTag" isn't as smart as I first thought, and literally looks at only first three entries in list? Much to investigate ...
Ah, I see fetchTag is used in one of our custom (pde base builder?) ant tasks: FetchBaseTask.java So, a) that'd be the thing to study to see how it works, and b) could probably check the "fetch_* files on build machine to see what's being fetched for an N build vs. I build.
Note it doesn't look to me like we are fetching from the wrong branch at any time. If the 4.2 I-build pulled org.eclipse.platform or org.eclipse.sdk from master instead of R4_HEAD, the build would be totally broken since these bundles store the e4 model and CSS files which don't exist in master (3.x). However it does seem like something odd is happening where N-builds are not always getting the latest content for these bundles.
Created attachment 214564 [details] all fetch_* files from 0424 build Just so we don't lose them. Here's all the "fetch" files from last nights N build. a 5 second glance they all look right to me. I also checked the actual "source" directories, and it does have one of the icons said to be missing: .../src/plugins/org.eclipse.platform $ ll eclipse256.png -rw-rw-r--+ 1 e4Build users 57K 2012-04-24 20:13 eclipse256.png Ah, but does not seem to have the right "build.properties" file. I'll attach that next.
Created attachment 214565 [details] "old" build.properties in N build src area
Created attachment 214568 [details] "new" build.properties build.properties from the src area for latest I build: build:/opt/public/eclipse/eclipse4I/build/supportDir/src/plugins/org.eclipse.platform/build.properties
FYI, I do delete the whole src directory, /opt/public/eclipse/eclipse4I/build/supportDir/src/ before each build ... but ... admit I haven't checked closely to see if that's working (even then, seems like that couldn't interfere with a simple "replacement" of a file?) Not sure where to look next ... it will be interesting to see if it gets updated in tonight's N build. Maybe there's some "time" difference somewhere (utc vs eastern)? :/
Created attachment 214577 [details] org.eclipse.platform/build.properties Well, seems consistently wrong in N builds. I've attached the version from 04/25 build, and appears to be same as one from 04/24, except that its "qualifier" property has been updated. qualifier=v20120425-2008
I think I've narrowed down the general area of the problem ... though, may take some git expertise to help narrow down the solution. Searching through the log, I found this 3068 GitCheckoutTagInLocalRepo: 3069 [echo] [GIT] /opt/public/eclipse/eclipse4N/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/scmCache/git___git_eclipse_org_gitroot_platform_eclipse_platform_git >> git checkout --force R4_HEAD 3070 [exec] Your branch is behind 'origin/R4_HEAD' by 1 commit, and can be fast-forwarded. 3071 [exec] Already on 'R4_HEAD' Which reminded me: 1) I don't know why "scmCache" is put so deep under "basebuilder" :/ 2) the message Your branch is behind 'origin/R4_HEAD' by 1 commit, and can be fast-forwarded. is probably telling us we are missing your one, most recent commit, but not sure why the fetch statement/command does not specify "get the latest, fast forward or not" 3) this is an area I changed a week or so ago ... for a while I could not get the build to build twice in a row, so I completely deleted everything included basebuilder and eclipsebuilder directories. So, I tried "easing off" that and no longer removed basebuilder. Oddly enough, if I go to the eclipse4I counterpart, and do a "git status", at first everything looks "better" (not "behind") anything [22:34:06] david_williams@build:/opt/public/eclipse/eclipse4I/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/scmCache/git___git_eclipse_org_gitroot_platform_eclipse_platform_git/platform/org.eclipse.platform $ git status # Not currently on any branch. nothing to commit (working directory clean) But, if I look at the build.properties file there, it also looks "out of date" (does not include the eclipse256 in its exported files) even though it is dated at time of today's I build: 2012-04-25 12:24 build.properties So ... several layers of complication. I suspect the "quick fix" is to go back to deleting everything ... unless someone knows enough git and PDE to easily "see" the problem. Here, I think is the related part of the noon I-build respin: 3324 GitUpdateLocalRepo: 3325 [echo] [GIT] /opt/public/eclipse/eclipse4I/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/scmCache/git___git_eclipse_org_gitroot_platform_eclipse_platform_git >> git fetch 3326 [exec] From git://git.eclipse.org/gitroot/platform/eclipse.platform 3327 [exec] * [new tag] I20120425-0800 -> I20120425-0800 3328 [exec] * [new tag] I20120425-1200 -> I20120425-1200 3329 3330 GitCheckSkipClone: 3331 3332 GitCloneRepoToLocalRepo: 3333 3334 GitCheckoutTagInLocalRepo: 3335 [echo] [GIT] /opt/public/eclipse/eclipse4I/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/scmCache/git___git_eclipse_org_gitroot_platform_eclipse_platform_git >> git checkout --force v20120424-1332 3336 [exec] Previous HEAD position was e853498... Merge in ant and update from origin/master 3337 [exec] HEAD is now at 2b916de... Bug 377442 - Eclipse icon missing in 4.2.0 Build id: N20120422-2000
Here's some of the other bugs I've opened and documented some of the oddities about "cached data". Bug 375780 - Surprising error when ran twice in a row indicating previous not cleaned up Bug 375794 - refine "cleanup" when a build starts Bug 355430 - change primary builder to 4.2 (comment 59, comment 63) Bug 375788 - scmCache seems in an odd location
I think until this is fixed we should not publish N-builds. See also bug 377735 comment 1. Not sure whether this is also caused by the same source, but the N-builds are lacking some images.
*** Bug 377735 has been marked as a duplicate of this bug. ***
(In reply to comment #11) > I think until this is fixed we should not publish N-builds. I think that's best also. We only have two more planned before we turn them off for M7 week. I will plan to still let them run, to help diagnose this problem, but not promote them to "downloads" until we are confident in them. Committers (well, anyone) can still "get" to the build by drilling down in in the build machine directories under http://build.eclipse.org/eclipse/ For example, for "finished builds", see http://build.eclipse.org/eclipse/eclipse4N/siteDir/eclipse/downloads/drops4/ http://build.eclipse.org/eclipse/eclipse4N/siteDir/eclipse/downloads/drops4/N20120425-2000/ For the "working area" of the latest build, it is under the 'build' branch in the hierarchy tree, such as, see http://build.eclipse.org/eclipse/eclipse4N/build/supportDir/src/ or source code cache as fetched for the build is under "scmCache", such as http://build.eclipse.org/eclipse/eclipse4N/build/supportDir/src/scmCache/ The "full log" of every message captured during build is at http://build.eclipse.org/eclipse/eclipse4N/fullmasterBuildOutput.txt (and careful, it's huge, like 14M)
I think there are a couple of issues and a couple of approaches to investigate the issue. The first, is that scmCache might be getting in a bad state, occasionally, and might not be being cleaned between N build. To investigate this, before tonight's (Thursday 4/26 8 PM build) I'd like to "manually" clean out the related "working areas" of N build to make sure its clean. Then we'll see if the same problem occurs, if so, then long term cure (I think) would be to improve gitFetch factory to either "force" completely fresh content, or if that's not possible (or desirable) to have it "fail" with appropriate error. On the other hand, if it some how "going wrong" even when everything is completely clean, then that might indicate some other type of error. So, this evening, I'll completely clean working areas and we'll see if still occurs. My guess is it will not. If not, then if someone has something they could commit to N build only for Friday's, we'll see if it ever gets new content, after caching git data, or if its something that happens routinely. My guess is routinely. Still indicating the fetch factory doesn't fetch with quite the right options? Suggestions or git-expertise welcome.
(In reply to comment #14) > I think there are a couple of issues and a couple of approaches to investigate > the issue. The first, is that scmCache might be getting in a bad state, > occasionally, and might not be being cleaned between N build. From my experience with scmCache and git fetch factory it might be less an issue of cleaning, but of actually doing a fetch/pull into the right branch? Maybe only "integration" is updated, but if nightlies get their sources from the local "master" branch, they won't see new stuff in origin/master? There was discussion whether the git fetch factory should automatically prepend "origin/" for checkout commands or whether this should be done explicitly in map files. Bug 347151 has some more, as also Bug 365944.
(In reply to comment #15) > (In reply to comment #14) Thank, Stephan. Little that I understand it, it does sound like the same issue as discussed in > Bug 347151 But, I will clarify (if not obvious) for our "nightlies" we use a whole different directory than I build, so the scmCache should "be the same" from night to night. master for some, R4_HEAD for others. But, that's not to say that during "test builds" or merely my converting to build.eclipse.org in a short period of time, that I haven't "messed up" the repository from night to night. But, if I take bug 347151, at face value, instead of a fetchTag of value="CVS=HEAD,GIT=master;git://git.eclipse.org/gitroot/platform/eclipse.platform.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git=R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.maps.git=R4_HEAD"> It should be value="CVS=HEAD,GIT=origin/master;git://git.eclipse.org/gitroot/platform/eclipse.platform.git=origin/R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git=origin/R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.git=origin/R4_HEAD;git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.maps.git=origin/R4_HEAD"> Am I interpreting bug 347151 correctly, that would be the case. And, I have not yet "cleaned" the old nightly scmCache, so with the smallest encouragement :) I could make the fetchTag value change for tonight's N build to see if that fixes things? Might be better than blindly cleaning?
(In reply to comment #16) > And, I have not yet "cleaned" the old nightly scmCache, so with the smallest > encouragement :) I could make the fetchTag value change for tonight's N build > to see if that fixes things? Might be better than blindly cleaning? Yes, in theory your suggestion should fix it in the existing nightly repos without having to delete and re-clone them PW
(In reply to comment #17) > (In reply to comment #16) > > Yes, in theory your suggestion should fix it in the existing nightly repos > without having to delete and re-clone them "In theory" is all the encouragement I need. :) I did save a copy of the fetch files and even scmCache so I can do a quick compare directly on build machine. If this works, it would be a "good practice" I assume? And essentially as fast? Any other places we should use "origin/master" instead of "master"? Any impact to other specs, such as should tags be "origin/${buildID}"? [mostly just asking for education here].
Three steps forward, two steps back ... the "back" steps are in my understanding, I think the build is working better and correctly with the "origin/xxxx" spec. If any one wants to look at the N build on the build machine, then can navigate through http://build.eclipse.org/eclipse/eclipse4N/siteDir/eclipse/downloads/drops4/N20120426-2000/ full log is at http://build.eclipse.org/eclipse/eclipse4N/siteDir/eclipse/downloads/drops4/N20120426-2000/fullmasterBuildOutput.txt The "good news" is the part of the build log I mentioned in comment 9 that mentions Your branch is behind 'origin/R4_HEAD' by 1 commit, and can be fast-forwarded. now says "You are in 'detached HEAD' state": GitCheckoutTagInLocalRepo: [echo] [GIT] /opt/public/eclipse/eclipse4N/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/scmCache/git___git_eclipse_org_gitroot_platform_eclipse_platform_git >> git checkout --force origin/R4_HEAD [exec] Note: checking out 'origin/R4_HEAD'. [exec] [exec] You are in 'detached HEAD' state. So, I think that means the "origin/R4_HEAD" specification is doing what "we" wanted it to.
The part that I'm (still) confused about (and is probably/maybe a sign something is not quite right) is that the "copy" I made of scmCache and the fetch files were gone! The "proving" that things _were_ being cleaned up still from build to build. After re-reading my notes here I realized there are _two_ locations for scmCache, and the one I "copied" (and its original) were cleaned up, but not those buried so deep under "basebuilder". The second location is at /opt/public/eclipse/eclipse4N/build/supportDir/src/scmCache And appears to be a subset of all the repos we use: $ ls -1 git___git_eclipse_org_gitroot_equinox_rt_equinox_bundles_git git___git_eclipse_org_gitroot_equinox_rt_equinox_framework_git git___git_eclipse_org_gitroot_equinox_rt_equinox_p2_git git___git_eclipse_org_gitroot_jdt_eclipse_jdt_git git___git_eclipse_org_gitroot_pde_eclipse_pde git___git_eclipse_org_gitroot_pde_eclipse_pde_build_git git___git_eclipse_org_gitroot_pde_eclipse_pde_ui_git git___git_eclipse_org_gitroot_platform_eclipse_platform_releng_git git___git_eclipse_org_gitroot_platform_eclipse_platform_runtime_git git___git_eclipse_org_gitroot_platform_eclipse_platform_team_git git___git_eclipse_org_gitroot_platform_eclipse_platform_ui_git Whereas the one "buried" under basebuilder is at /opt/public/eclipse/eclipse4N/build/supportDir/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.8.0.v20120119-1950/scripts/scmCache and considerably more $ ls -1 git___git_eclipse_org_gitroot_equinox_rt_equinox_bundles_git git___git_eclipse_org_gitroot_equinox_rt_equinox_framework_git git___git_eclipse_org_gitroot_equinox_rt_equinox_p2_git git___git_eclipse_org_gitroot_jdt_eclipse_jdt_core_binaries_git git___git_eclipse_org_gitroot_jdt_eclipse_jdt_core_git git___git_eclipse_org_gitroot_jdt_eclipse_jdt_debug_git git___git_eclipse_org_gitroot_jdt_eclipse_jdt_git git___git_eclipse_org_gitroot_jdt_eclipse_jdt_ui_git git___git_eclipse_org_gitroot_pde_eclipse_pde git___git_eclipse_org_gitroot_pde_eclipse_pde_build_git git___git_eclipse_org_gitroot_pde_eclipse_pde_ui_git git___git_eclipse_org_gitroot_platform_eclipse_platform_common_git git___git_eclipse_org_gitroot_platform_eclipse_platform_debug_git git___git_eclipse_org_gitroot_platform_eclipse_platform_git git___git_eclipse_org_gitroot_platform_eclipse_platform_releng_git git___git_eclipse_org_gitroot_platform_eclipse_platform_resources_git git___git_eclipse_org_gitroot_platform_eclipse_platform_runtime_git git___git_eclipse_org_gitroot_platform_eclipse_platform_swt_binaries_git git___git_eclipse_org_gitroot_platform_eclipse_platform_swt_git git___git_eclipse_org_gitroot_platform_eclipse_platform_team_git git___git_eclipse_org_gitroot_platform_eclipse_platform_text_git git___git_eclipse_org_gitroot_platform_eclipse_platform_ua_git git___git_eclipse_org_gitroot_platform_eclipse_platform_ui_git
FYI, the second "step back" in my understanding has to do with Bug 377844 - Many "compile errors" in build logs from obsolete SWT code I'm assuming unrelated to this general issue, but just noticed tonight for the first time.
(In reply to comment #18) > Any other places we should use "origin/master" instead of "master"? Any impact > to other specs, such as should tags be "origin/${buildID}"? [mostly just asking > for education here]. It's fine for us to use the origin/branchName spec in our nightlies. It would probably even be fine for us to use them in our auto-tagging repos that would have no commit changes. So for auto-tagging, if the repo will just have some tags pushed, that would be fine. If, like eclipse.platform.releng.maps we were also committing something, then it has to be the local branch, managed by pulls and pushes. Some places use a convention for build tags "build/vWhatever" or "build/IWhatever" to group them, although we don't follow that convention. PW
FWIW, I would like to change the location of the scmCache. I think easy and relatively safe to do. Please see bug 375788 comment 6 for details. Just thought I'd document it here, also (though, technically not related to this problem) ... just since the topic has gotten so much discussion here. I still don't plan to "publish" tonight's N build since no one said "new ones looks good to me" ... so will be easy to revert if required.
I think the latest N-build looks OK, but need to see a real "new" N-build for final confirmation.
Let's count this as "fixed" and then once we have "real" new N builds we can mark as verified (or, reopened).
