Community
Participate
Working Groups
Now that we have a "Releng HIPP", running under a user Id that has access to "downloads area" (for Eclipse and Equinox) we can simplify the "promote builds" process. Historically, when running under 'e4Build' id (a "shared" id) it was not allowed to upload things to "downloads" since it was not a "committer". Therefore our solution was to have a "queue" of jobs to do the promotion, and committers to Equinox and another committer to Eclipse had cronjobs which would check those queues and "do the upload" if data was found to work on. We have "transfered" that same logic while moving our cron jobs to Releng HIPP. But, once we get comfortable that Releng HIPP fills our needs, we can avoid that "queue" step. Since the Releng HIPP instance, which runs under a user id like genie.releng which *is* a "member" of the proper groups to allow uploads to Eclipse and Equinox download areas we can modify the logic so that instead of creating a file to "put in the queue" we can just go ahead and do the promotion directly at the end of the build.
New Gerrit change created: https://git.eclipse.org/r/77141
First, don't forget, there are two of them -- one for Eclipse and one for Equinox. The solution you have in the Gerrit patch might work, but I do not think it is a very good solution since it leaves the same logic of "creating an extra script file" and then executing that script file later. I think it would be far better to not create the "extra script file" to begin with, and just execute what the script file would have executed. This is easiest to see with Equinox since all the script file did was rsync. This is in "promote-build.sh". See the following code: # Here is content of promotion script (note, use same ptimestamp created above): echo "#!/usr/bin/env bash" > ${promoteScriptLocationEquinox}/${scriptName} echo "# promotion script created at $ptimestamp" > ${promoteScriptLocationEquinox}/${scriptName} echo "rsync --times --omit-dir-times --recursive \"${eqFromDir}\" \"${eqToDir}\"" >> ${promoteScriptLocationEquinox}/${scriptName} # we restrict "others" rights for a bit more security or safety from accidents chmod -v ug=rwx,o-rwx ${promoteScriptLocationEquinox}/${scriptName} Notice the "extra script file" we create consists of two comments and then rsync. (Followed by a "chmod" for some tiny security improvement. So, all of that could be replace with simply rsync --times --omit-dir-times --recursive "${eqFromDir}" "${eqToDir}" And then the Hudson job that "checks for the file" disabled. I think. = = = = = Eclipse is similar, but there are other scripts that are called since there is more processing done. = = = = = The only hard part might be in determining if the exact "location" that the promote code is executed needs to be adjusted. I don't think so but have not looked that closely.
(In reply to David Williams from comment #2) > First, don't forget, there are two of them -- one for Eclipse and one for > Equinox. > > The solution you have in the Gerrit patch might work, but I do not think it > is a very good solution since it leaves the same logic of "creating an extra > script file" and then executing that script file later. > > I think it would be far better to not create the "extra script file" to > begin with, and just execute what the script file would have executed. > > This is easiest to see with Equinox since all the script file did was rsync. > > This is in "promote-build.sh". See the following code: > > # Here is content of promotion script (note, use same ptimestamp created > above): > echo "#!/usr/bin/env bash" > ${promoteScriptLocationEquinox}/${scriptName} > echo "# promotion script created at $ptimestamp" > > ${promoteScriptLocationEquinox}/${scriptName} > echo "rsync --times --omit-dir-times --recursive \"${eqFromDir}\" > \"${eqToDir}\"" >> ${promoteScriptLocationEquinox}/${scriptName} > > # we restrict "others" rights for a bit more security or safety from > accidents > chmod -v ug=rwx,o-rwx ${promoteScriptLocationEquinox}/${scriptName} > > Notice the "extra script file" we create consists of two comments and then > rsync. (Followed by a "chmod" for some tiny security improvement. > > So, all of that could be replace with simply > > rsync --times --omit-dir-times --recursive "${eqFromDir}" "${eqToDir}" > > And then the Hudson job that "checks for the file" disabled. > > I think. > > = = = = = > > Eclipse is similar, but there are other scripts that are called since there > is more processing done. > > = = = = = > > The only hard part might be in determining if the exact "location" that the > promote code is executed needs to be adjusted. I don't think so but have not > looked that closely. I agree with you analysis. Even though my solution creates one more intermediate script, I thought it would be easier to push the promotion immediately. The final solution is to not have this script generated at all. I am currently working on that. Thanks for the analysis David
(In reply to David Williams from comment #2) > First, don't forget, there are two of them -- one for Eclipse and one for > Equinox. > > The solution you have in the Gerrit patch might work, but I do not think it > is a very good solution since it leaves the same logic of "creating an extra > script file" and then executing that script file later. > > I think it would be far better to not create the "extra script file" to > begin with, and just execute what the script file would have executed. > > This is easiest to see with Equinox since all the script file did was rsync. > > This is in "promote-build.sh". See the following code: > > # Here is content of promotion script (note, use same ptimestamp created > above): > echo "#!/usr/bin/env bash" > ${promoteScriptLocationEquinox}/${scriptName} > echo "# promotion script created at $ptimestamp" > > ${promoteScriptLocationEquinox}/${scriptName} > echo "rsync --times --omit-dir-times --recursive \"${eqFromDir}\" > \"${eqToDir}\"" >> ${promoteScriptLocationEquinox}/${scriptName} > > # we restrict "others" rights for a bit more security or safety from > accidents > chmod -v ug=rwx,o-rwx ${promoteScriptLocationEquinox}/${scriptName} > > Notice the "extra script file" we create consists of two comments and then > rsync. (Followed by a "chmod" for some tiny security improvement. > > So, all of that could be replace with simply > > rsync --times --omit-dir-times --recursive "${eqFromDir}" "${eqToDir}" > > And then the Hudson job that "checks for the file" disabled. > > I think. > > = = = = = > > Eclipse is similar, but there are other scripts that are called since there > is more processing done. > > = = = = = > > The only hard part might be in determining if the exact "location" that the > promote code is executed needs to be adjusted. I don't think so but have not > looked that closely. Hi David, Here are the tasks done by the promote job. 1. Promote update site. 2. promote the sdk builds 3. Start tests 4. Send the mail to releng-dev with the build completion There are two ways to avoid the hudson job 1. Calling the generated script as soon as it is created 2. call the syncjob directly If we do the rsync only we will miss on update site and starting of tests. Also I prefer separation of the promote script as it provides modularity. Uploaded a new patch this has modification in promote script. Intermediate script does not get generated
Gerrit change https://git.eclipse.org/r/77141 was merged to [master]. Commit: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=3dbae3ff448ecdb72e8be57afa84dfeb1e24f56f
I have merge changes into "head" but will wait to mark "fixed" until the Hudson Releng cron jobs are "disabled". We will have to wait until the I-build is done before doing that. So tonight's N-build will be the first time this code is "exercised". Due to timezone differences, I will disable the Hudson Releng HIPP jobs this afternoon some time .... unless I see you do it first :)
The I-build was promoted, to Eclipse and Equinox, so I have now disabled the Releng HIPP jobs of https://hudson.eclipse.org/releng/view/Equinox/job/rt.equinox.releng.promote/ and https://hudson.eclipse.org/releng/view/Releng/job/eclipse.releng.checkAndPromoteFinishedBuilds/ We can remove them completely eventually (say, in a week or two), once satisfied current scripts work for all the builds (maintenance, patch builds, etc.)
(In reply to David Williams from comment #7) > The I-build was promoted, to Eclipse and Equinox, so I have now disabled the > Releng HIPP jobs of > > https://hudson.eclipse.org/releng/view/Equinox/job/rt.equinox.releng.promote/ > > and > > https://hudson.eclipse.org/releng/view/Releng/job/eclipse.releng. > checkAndPromoteFinishedBuilds/ > > We can remove them completely eventually (say, in a week or two), once > satisfied current scripts work for all the builds (maintenance, patch > builds, etc.) I also removed the directories /shared/eclipse/promotion and /shared/eclipse/equinox/promotion which is where the "extra files" were put until the cronjob handled them. They should no longer be needed, so removing them now will provide yet another level of verification that everything is working as intended.
Reopening for a minor issue. I had removed /shared/eclipse/promotion/queue but something, somewhere, recreated it. It was empty, so pretty sure nothing still tries to use it but we should clean up whatever code creates it.
I think I see at least one place where /shared/eclipse/promotion/queue was being created and have removed it: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=afe69849d9f32a0a64d465dd8d7afc8274bc1516 = = = = = = = = = = = Note: our current procedures for promoting milestones and releases I *think* no longer need that location, but did not check all of those scripts to see if any remaining remnants of old code left. = = = = = = = = = = = Several scripts still refer to /shared/eclipse/promotion/queue such as the "cleanup" jobs. That "dead code" should be removed eventually, but it is sort of designed to handle directories that do not exist. Several also still mention the /shared/eclipse/equinox/promotion/queue but does not seem to be created so I did not examine those too closely. I suggest you open an enhancement bug to clean up all of that sort of code.
