Community
Participate
Working Groups
This is a follow up to bug 486366 comment 9. The intent is this is for "Platform releng", with a unique "genie.*" ID specific to platform.releng, and only the eclipse.platform.releng have access. (And, myself to have "admin" access :) And the "genie.*" id should be member of "eclipse.platform.releng". Thanks,
I would to take a step back and think a little bit more how the Eclipse Platform is built and how HIPPs could help. Currently, we have a single "platform" HIPP which is mostly used to build gerrit reviews for the following projects: platform, pde, jdt, equinox, e4. I call this set of project platform-at-large. There are some sub-projects exceptions, e.g., p2 has its own HIPP. I understand that having the build of the platform-at-large on a single HIPP or to handle the releng of all these projects from a single entry point makes it easier to manage (e.g., the releng.aggregator git repo). However, I really think we should work toward a more decoupled build and promotion. If we impose constraints on ourself to be able to build all projects independently, it will make the life of our contributors easier. So I'm really reluctant to create a "eclipse.platform.releng" and make it cross the boundaries of the "Eclipse Platform" project (i.e. give it access to JDT, PDE, ...). It forces us to create a lot of "special rules" for the platform. I reckon that having consistency between all projects (platform and others) will help us to provide better services, more security and make our life easier if we ever need to migrate to another solution. (we should also talk about the "callisto-*" group later ;)). Thus, instead of creating of platform-releng HIPP, I would propose to start again with the issue you had in bug 486366, and create a HIPP for equinox only, which has only access to equinox related directories? (Please, don't read this as me not trying to be flexible, I'm just looking for better solutions.)
Better solutions are great but am still not sure what it is in this case. If we had a "equinox" HIPP which for now would be to promote builds from build machine to DL server, the THAT user id would need access to "eclipse.platform.releng" as well as equinox-dev. So, that's why I was thinking a third id that was a member of equinox.dev and eclipse.platform.releng would be the most secure and safest. FYI, this eclipse.platform.releng id would not need access to PDE or JDT. At least, not for the purposes for which I am requesting it. If/when we did production builds on a HIPP instance, then it might, in order to 'tag' those repositories ... all the more reason to have a special "releng" instance to me. So it would play the role of our current 'e4Build' ID. But, fine with me to continue to discuss alternatives.
(In reply to David Williams from comment #2) > Better solutions are great but am still not sure what it is in this case. > > If we had a "equinox" HIPP which for now would be to promote builds from > build machine to DL server, the THAT user id would need access to > "eclipse.platform.releng" as well as equinox-dev. "for now" is why I would like to think about where we're going with the build of the platform. If the goal is to move to a Hudson Instance *Per Project*, then we should create one HIPP per Eclipse Project (or sub-projects like for p2). I acknowledge that during the transition time, the HIPP owner will have special permissions. > So, that's why I was thinking a third id that was a member of equinox.dev > and eclipse.platform.releng would be the most secure and safest. Agreed. I am not thinking about the the most secure and safest way "for now", but how it could be in the mid-term. > FYI, this eclipse.platform.releng id would not need access to PDE or JDT. At > least, not for the purposes for which I am requesting it. If/when we did > production builds on a HIPP instance, then it might, in order to 'tag' those > repositories ... all the more reason to have a special "releng" instance to > me. So it would play the role of our current 'e4Build' ID. Don't you think that this e4Build (and others cross-projects "releng" IDs) are a bit outdated? IMHO, each project should have its own build/promotion cycle and Eclipse platform projects which have dependencies to the former should consume their build artifacts as any other consumer. E.g., the platform build should consume Equinox the same way as any other project who wants to build an Equinox based project. > But, fine with me to continue to discuss alternatives. and I'm willing to help with implementing the alternatives if you find them viable.
(In reply to Mikael Barbero from comment #3) > Don't you think that this e4Build (and others cross-projects "releng" IDs) > are a bit outdated? IMHO, each project should have its own build/promotion > cycle and Eclipse platform projects which have dependencies to the former > should consume their build artifacts as any other consumer. E.g., the > platform build should consume Equinox the same way as any other project who > wants to build an Equinox based project. No, not outdated yet. That is, we could not do builds the way you are suggesting -- without a lot of work and refactoring. We still have too many "cross-project" dependencies. I think that sort of build is still a minimum of a year away. Bug 398613 tracks such issues. If not obvious, the reason many of the Gerrit jobs work is that when they have a "cross-project" dependency they use the pre-built dependency from the previous build. We can't do production build like that, obviously. So I think your suggestion is more a long term consideration, not mid-term. Given that, I can not think of any other solutions, other than 'status quo' and then someone besides John should get a shell account to "take over" whatever cronjobs he is running.
Ok, then let's go with a platform.releng HIPP. I've queued the creation of the HIPP. I will keep you posted when it's ready.
I'm late to the conversation, but in order to make the Equinox build self-contained then we would have to move some components of Equinox back to platform. For example, the Equinox security bits that depend on Eclipse UI and the p2 bits that have Eclipse UI. Personally I would not mind if all of p2 went back to the Eclipse project, but I digress :) Then we would have to move the test harness from the eclipse project to Equinox so that we can remove the cyclic dependencies between Equinox and Eclipse. That would greatly simplify the Equinox build, but all that needs to be prioritized and agreed upon by Equinox leadership as well as the Eclipse PMC. I think it could be agreed upon, but the work would not happen in a short time-frame.
Sorry for the delay. Your HIPP is now available at https://hudson.eclipse.org/releng/. eclipse.platform.releng committer can log into this HIPP with their email address. genie.releng is part of eclipse.platform.releng group so it can write wherever committers of that group can. I've also made you an admin on this HIPP. Please reopen if you need more assistance.
Thank you. The only other thing needed were changes corresponding to those made in bug 486366. That is the 'genie.releng' userid needs to be added to the 'rt.equinox' group. And to "undo" the change in bug 486366. That is 'genie.platform' userid needs to be removed from the rt.equinox group (it is too broad). = = = = = I might need another "sweep" of relevant directories to make sure the group permissions are correct and that those directories have a sticky bit set for 'group', but have not looked yet (plus I am not sure what I can fix myself). So if ok with you I will open a new bug for that if I find out it is needed. I'll probably start some "conversions" over the next few days -- others may take weeks (or months?). = = = = = Thanks again
(In reply to David Williams from comment #8) > Thank you. > > The only other thing needed were changes corresponding to those made in bug > 486366. > > That is the 'genie.releng' userid needs to be added to the 'rt.equinox' > group. Done. > > And to "undo" the change in bug 486366. That is 'genie.platform' userid > needs to be removed from the rt.equinox group (it is too broad). Done. > > = = = = = > > I might need another "sweep" of relevant directories to make sure the group > permissions are correct and that those directories have a sticky bit set for > 'group', but have not looked yet (plus I am not sure what I can fix myself). > > So if ok with you I will open a new bug for that if I find out it is needed. > I'll probably start some "conversions" over the next few days -- others may > take weeks (or months?). Sure, no problem. I'm always glad to help. > Thanks again You're welcome. I'm always glad to help.
Marking as fixed. [Another possible complication is that I have sent John an email to "remove" (or modify) his cron job but have not heard back yet. I will get started on that one "promote" job and see how it goes.] Thanks again,
(In reply to David Williams from comment #10) > > [Another possible complication is that I have sent John an email to "remove" > (or modify) his cron job but have not heard back yet. I did hear from John, and he is pretty sure the equinox promote was the only thing in is cron table, so I think safe to remove it. You might save a copy, crontab -l before removing it crontab -r.
(In reply to David Williams from comment #11) > (In reply to David Williams from comment #10) > > > > [Another possible complication is that I have sent John an email to "remove" > > (or modify) his cron job but have not heard back yet. > > I did hear from John, and he is pretty sure the equinox promote was the only > thing in is cron table, so I think safe to remove it. > > You might save a copy, crontab -l before removing it > crontab -r. Oh, and should have mentioned ... John was still working on the "paperwork" or he would have done it himself.
I've cleared John's crontab (see bug 486366 comment 17).
I can not seem to access "download server" from this HIPP instance. A simple script with "find /home/data/httpd/download.eclipse.org/equinox/drops ..." replies with find: ‘/home/data/httpd/download.eclipse.org/equinox/drops’: No such file or directory See the 'fixDLsite' job. I've tried both "direct" access, and executing a script in /shared/eclipse/equinox/ with same results.
Similar result when I try to "rsync" a drop there: rsync: mkdir "/home/data/httpd/download.eclipse.org/equinox/drops" failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(656) [Receiver=3.1.0] I assume rsync is trying to "mkdir" because it "thinks" it is not there. This was working when using the "genie.platform" userid. (That is, this part of it was working -- I am not suggesting to revert :).
/home/data/httpd/download.eclipse.org was not properly mounted on the machine of genie.releng HIPP. This should be ok now. Matt, I think you should have a look at your mount sanity check for SLES 12. I suspect this issue comes from the reboot from last week.
(In reply to Mikael Barbero from comment #16) > /home/data/httpd/download.eclipse.org was not properly mounted on the > machine of genie.releng HIPP. This should be ok now. > > Matt, I think you should have a look at your mount sanity check for SLES 12. > I suspect this issue comes from the reboot from last week. Thanks! That worked great. On to the next problem. :) I could do several "promotes" and several "removes" of old builds, via the HIPP, but there are some directories I could not remove, such as M20160127-1000 M20160128-1225 in /home/data/httpd/download.eclipse.org/equinox/drops Nor could I even "add" a file to them (via 'touch' -- I was going to try to "hide" them, with a trick we have using "buildHidden" file in the directory) but in all cases, to remove them, or 'touch' a file under them, I got permission denied. It seems that those -- and several others -- do not have "group write" bit set. I am not sure why. Is this something I need to fix, such "before I 'promote' them (using rsync) to make sure group write is set? Or, just something ?acls? or "sticky bit"? Actually, looking closer, it seems the one case that was promoted by "genie.releng" is correct. Perhaps this is just a matter of "fixing" all the earlier ones, with 'root' id? such as chmod -c -R g+w /home/data/httpd/download.eclipse.org/equinox (I have intentionally given the "parent" of 'drops' in the example, since there are other directories under 'equinox' that I will probably need to "mess with" someday?) Thanks for advise, or fixing. And thanks again for the quick 'mount' -- I was worried there for a bit I had misunderstood something! :)
It may issue with umask of users previously creating those folder... or something else. It sounds fait enough to just fix it. I've run your chmod command. Their is no file without the g+w permission in /home/data/httpd/download.eclipse.org/equinox: build:/home/data/httpd/download.eclipse.org/equinox # find . ! -perm -g+w | wc -l 0 Waiting for the next problem ;)
I may have replied too quickly... /s/It may issue/It may be an issue/ /s/folder/folders/ /s/their/there/
(In reply to Mikael Barbero from comment #18) > Waiting for the next problem ;) Well, no need to wait, you can move on to other things now. It is working perfectly. Mr. "HIPP release engineer" is happy. :)
