[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [dash-dev] Streamlining the Athena promote process (was Re: [gmf-dev] Head up: last night's build failure was a real failure in org.eclipse.gmf.xpand.migration)
|
As an outsider, anything that would prevent me from hacking cron jobs
in crontab to publish builds would be appreciated.
On Sun, Jan 31, 2010 at 9:26 AM, David Carver <d_a_carver@xxxxxxxxx> wrote:
> Denis is there a place where the security requirements are documented for
> moving files between systems at eclipse? The reason I'm asking is that it
> would help me come up with an integrated solution (i.e. possibly updating
> the SCP plugin for specific user logins). Let me see what type of license
> the SCP plugin at Hudson is under. I can start with that, as a basis for a
> possible DASH project for Hudson specific plugins and see what happens from
> there. I'm fine for the Cron job idea, as long as it's documented.
>
> Dave
>
>
> Nick Boldt wrote:
>>
>> Dude, "creating a cron job" is like 5 mins of work. Copy, paste, issue
>> command `crontab -e`, set polling time/interval, and vavoom, done. Hardly a
>> high barrier. Getting yourself a bash account on build.eclipse.org
>> <http://build.eclipse.org> takes longer (and that only requires filing a bug
>> and waiting). :)
>>
>> As to a Hudson Plugin project in Dash, sure... cuz it's that or at SF or
>> Google Code or wherever else code is hosted. If you want to jump thru the
>> hoops to make it happen, by all means do.
>>
>> As to slaves, one workaround there is to enable the 'publishing plugin'
>> which allows the output of a build to be pushed into another Hudson instance
>> (then poll that instance for updates); alternatively, if you tie your job to
>> a single slave, it's just as effective as being tied to master: poll the
>> slave/master on which your build runs.
>>
>> At some point, when we have the option to run across multiple
>> slaves/platforms, the only requirements will therefore be that the crontab
>> script poll all the appropriate places and that the slaves all support
>> ssh/scp/rsync in order to be pollable. Or failing that, ftp/http, I suppose.
>>
>> N
>>
>> On Fri, Jan 29, 2010 at 10:51 AM, David Carver <d_a_carver@xxxxxxxxx
>> <mailto:d_a_carver@xxxxxxxxx>> wrote:
>>
>> Well, the SCP plugin is Open Source, so I'm sure if some
>> enterprising committer wanted to modify it to allow more
>> configuration options one could do so. These types of concerns
>> are why I think Dash may need to have a Hudson Plugin subproject.
>> This way you can address the eclispe specific environment concerns.
>>
>> Nick, how does your Batch Task/Cron idea work in a distributed
>> system when Slave machines? What about slave machines that
>> aren't Linux based? How does it work with Windows based
>> machines? Some things to keep in mind.
>>
>> I just think having to make users create cron jobs is just another
>> barrier that isn't necessarily necessary. I'd still prefer
>> something that was more integrated instead of doing batch magic
>> which can be still as insecure as SCP rights. Just takes one
>> wrong batch job executing at the wrong access level to mess things up.
>>
>> Dave
>>
>>
>> On 01/29/2010 06:24 AM, Denis Roy wrote:
>>>
>>> +1
>>>
>>> Cron jobs that poll are not nearly as elegant, but they do allow
>>> for better security.
>>>
>>> D.
>>>
>>> On 01/28/2010 10:59 PM, Nick Boldt wrote:
>>>>
>>>> Problem with SCP plugin is that it uses global scp access rights
>>>> to move files; it doesn't seem to be able to key in to the LDAP
>>>> database so that I can push files to
>>>> nickb@xxxxxxxxxxxxxxx:~/downloads/
>>>> <mailto:nickb@xxxxxxxxxxxxxxx:%7E/downloads/>, for example.
>>>>
>>>> My workaround is to have Hudson write to its own workspace (eg.,
>>>> create a .lock file that says "job foo's build #34 is ready to
>>>> promote") and then have nickb's crontab look for that file every
>>>> minute or two; if found, it can perform the publish and either
>>>> delete the .lock file, or record somehow that it's already
>>>> pushed that build so as to not do so again... unless the Batch
>>>> Task is kicked again to refresh the file.
>>>>
>>>> So, like with the signing genie, we'd have a process that can
>>>> pass information between users@dev.eclipse
>>>> <mailto:users@dev.eclipse> and the shared user hudsonBuild in
>>>> order to allow uease of use AND security.
>>>>
>>>> N
>>>>
>>>> On 01/28/2010 05:02 PM, David Carver wrote:
>>>>>
>>>>> If we are concerned about giving Hudson write access to the
>>>>> downloads
>>>>> directory, then we might want to consider the Batch Tasks
>>>>> Plugin that
>>>>> Nick has suggested. You could run an ftp/scp command from there
>>>>> and ftp
>>>>> the files over to the download server. At least this way it is
>>>>> running
>>>>> under specific committer ids, which would then limit it to
>>>>> specific
>>>>> directories instead of the whole thing.
>>>>>
>>>>> There is also the SCP plugin for Hudson which allows for a secure
>>>>> connection and copying of files.
>>>>>
>>>>> http://wiki.hudson-ci.org/display/HUDSON/SCP+plugin
>>>>>
>>>>> Dave
>>>>>
>>>>> On 01/28/2010 12:41 PM, Kim Moir wrote:
>>>>>>
>>>>>> It would be nice to be able to tag cvs projects as part of a
>>>>>> hudson
>>>>>> build. Today, when we run our integration builds, we tag our
>>>>>> builder
>>>>>> and our map file project so that the build is reproducible.
>>>>>> Once we
>>>>>> have test hardware at the foundation and can run our builds on
>>>>>> the
>>>>>> hudson install on build.eclipse.org
>>>>>> <http://build.eclipse.org>, the only way to do this is to
>>>>>> have cronjob entries that correspond the time our build
>>>>>> starts, which
>>>>>> isn't 100% foolproof. That being said, I totally understand
>>>>>> Denis'
>>>>>> concern with the Hudson user having write access to the larger
>>>>>> download and source filesystem. Not good from a security
>>>>>> perspective.
>>>>>>
>>>>>> Kim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *David Carver <d_a_carver@xxxxxxxxx>
>>>>>> <mailto:d_a_carver@xxxxxxxxx>*
>>>>>> Sent by: dash-dev-bounces@xxxxxxxxxxx
>>>>>> <mailto:dash-dev-bounces@xxxxxxxxxxx>
>>>>>>
>>>>>> 01/28/2010 11:54 AM
>>>>>> Please respond to
>>>>>> Tools for Committer Community <dash-dev@xxxxxxxxxxx>
>>>>>> <mailto:dash-dev@xxxxxxxxxxx>
>>>>>>
>>>>>>
>>>>>> To
>>>>>> Tools for Committer Community <dash-dev@xxxxxxxxxxx>
>>>>>> <mailto:dash-dev@xxxxxxxxxxx>
>>>>>> cc
>>>>>> Subject
>>>>>> Re: [dash-dev] Streamlining the Athena promote process
>>>>>> (was Re:
>>>>>> [gmf-dev] Head up: last night's build failure was a real
>>>>>> failure in
>>>>>> org.eclipse.gmf.xpand.migration)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Promotion should only require write access to to the various
>>>>>> download
>>>>>> area folders for the projects.
>>>>>>
>>>>>> It wouldn't necessarily have to have tagging priviledges to
>>>>>> CVS/SVN/Git as it stands right now all projects use MAP files
>>>>>> for this.
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>> On 01/28/2010 08:24 AM, Denis Roy wrote:
>>>>>> For Hudson to do the promotion itself, I assume the
>>>>>> hudsonBuild user
>>>>>> would need write access to most, if not all of the downloads
>>>>>> area. If
>>>>>> the promotion process involves tagging or committing to CVS, then
>>>>>> hudsonBuild would need commit access to CVS.
>>>>>>
>>>>>> Ick.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 01/28/2010 11:11 AM, David Carver wrote:
>>>>>> Nick, Hudson already has plugins that would allow the
>>>>>> promotion. The
>>>>>> problem is that the user that Hudson runs under does not have
>>>>>> rights
>>>>>> to the download server to be able to push the necessary bits.
>>>>>> We have
>>>>>> the necessary plugins installed in Hudson, just don't have the
>>>>>> access
>>>>>> rights.
>>>>>>
>>>>>> Instead of jumping through work arounds and hoops we really
>>>>>> need to
>>>>>> address the problem at hand. Hudson should have necessary access
>>>>>> rights to be able to do the promotion itself.
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>> On 01/27/2010 11:43 PM, Nick Boldt wrote:
>>>>>> Theoretically, it should be posible to create a hudson job
>>>>>> that would
>>>>>> be used simply to promote the latest clean builds by flipping
>>>>>> a bit to
>>>>>> notify a listening process (namely a crontab script) that it's
>>>>>> time to
>>>>>> promote.
>>>>>>
>>>>>> But does having a button on a webpage really make publishing
>>>>>> easier
>>>>>> than ssh'ing to a server and running a script? The same code
>>>>>> that the
>>>>>> crontab runs can be run on demand, and you could wrap that
>>>>>> line of
>>>>>> code w/ a wrapper script such that all you'd need to do is ssh to
>>>>>> build.eclipse and run "~/p" to push your bits. Only marginally
>>>>>> easier
>>>>>> than logging in to build.eclipse.org
>>>>>> <http://build.eclipse.org> to push a button.
>>>>>>
>>>>>> Would an Eclipse plugin would be perhaps a better idea? Perhaps
>>>>>> something graphically modeled (hint: GMF? Zest?), with not just a
>>>>>> button but a view of your latest builds in Hudson and their
>>>>>> test results?
>>>>>>
>>>>>> Or, perhaps a Hudson plugin is better, such that in addition
>>>>>> to the
>>>>>> build button we already have, we could also have a promote
>>>>>> button?
>>>>>>
>>>>>> N
>>>>>>
>>>>>> On 01/25/2010 09:54 AM, Anthony Hunter wrote:
>>>>>> Hi Nick,
>>>>>>
>>>>>> With GMF and the common modeling build, have a web page with two
>>>>>> buttons, build and promote
>>>>>> With the Athena build, have a web page with a build button,
>>>>>> but no
>>>>>> promote yet. You last communicated promote must be manually
>>>>>> ran from the
>>>>>> command line or cron.
>>>>>>
>>>>>> >From my point of view, the lack of a quick easy promote have
>>>>>> Athena a
>>>>>> hard sell.
>>>>>>
>>>>>> To be honest however, I have not had any time to look further
>>>>>> that
>>>>>> Athena for GEF.
>>>>>>
>>>>>> Once GEF and EMF have fully moved, the rest of the modeling
>>>>>> stack (and
>>>>>> GMF) should have no excuse to move.
>>>>>>
>>>>>> And the goal should be / is for all modeling projects to share
>>>>>> the same
>>>>>> build technology.
>>>>>>
>>>>>> Cheers...
>>>>>> Anthony
>>>>>> -- Anthony Hunter _mailto:anthonyh@xxxxxx.com_
>>>>>> Software Development Manager
>>>>>> IBM Rational Software: Aurora / Modeling Tools
>>>>>> Phone: 613-270-4613
>>>>>>
>>>>>>
>>>>>> Inactive hide details for Nick Boldt ---2010/01/25 12:54:34
>>>>>> AM---If only
>>>>>> there was a better way... like building against updateNick Boldt
>>>>>> ---2010/01/25 12:54:34 AM---If only there was a better way...
>>>>>> like
>>>>>> building against update sites
>>>>>>
>>>>>>
>>>>>> From:
>>>>>> Nick Boldt _<nickboldt@xxxxxxxxx>
>>>>>> <mailto:nickboldt@xxxxxxxxx>_ <mailto:nickboldt@xxxxxxxxx>
>>>>>>
>>>>>> To:
>>>>>> "GMF Project developer discussions." _<gmf-dev@xxxxxxxxxxx>
>>>>>> <mailto:gmf-dev@xxxxxxxxxxx>_
>>>>>> <mailto:gmf-dev@xxxxxxxxxxx>
>>>>>>
>>>>>> Date:
>>>>>> 2010/01/25 12:54 AM
>>>>>>
>>>>>> Subject:
>>>>>> Re: [gmf-dev] Head up: last night's build failure was a real
>>>>>> failure in
>>>>>> org.eclipse.gmf.xpand.migration
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> If only there was a better way... like building against update
>>>>>> sites
>>>>>> instead of (or in addition to) SDK zips...
>>>>>>
>>>>>> oh, wait... there is!
>>>>>> _
>>>>>>
>>>>>> __http://wiki.eclipse.org/Common_Build_Infrastructure/Defining_Binary_Dependencies_
>>>>>>
>>>>>>
>>>>>>
>>>>>> With Galileo SR2 nearly complete, does GMF plan to move to
>>>>>> Athena/Hudson during the Helios cycle? If not, what blocks
>>>>>> you? I'd
>>>>>> like a list of blocking requirements so I can better
>>>>>> prioritize Athena
>>>>>> TODOs and get it into a form that will meet more (if not all)
>>>>>> of your
>>>>>> needs.
>>>>>>
>>>>>> BTW, the tagandrelease system works just as well for Athena
>>>>>> builds as
>>>>>> for Modeling builds; the only difference is that you have full
>>>>>> control
>>>>>> over it (ie., no CVS commit issues) and would run it on
>>>>>> build.eclipse.org <http://build.eclipse.org> instead of
>>>>>> modeling.eclipse.org <http://modeling.eclipse.org>.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>> On Wed, Jan 20, 2010 at 9:55 AM, Anthony Hunter
>>>>>> _<anthonyh@xxxxxxxxxx> <mailto:anthonyh@xxxxxxxxxx>_
>>>>>> <mailto:anthonyh@xxxxxxxxxx> wrote:
>>>>>> > Hi Team,
>>>>>> >
>>>>>> > Releng update. I am trying to restore
>>>>>> org.eclipse.releng.tools.tagandrelease
>>>>>> > for GMF. It runs from cron on modeling.eclipse.org
>>>>>> <http://modeling.eclipse.org>. It checks CVS
>>>>>> for new
>>>>>> > changes, tags and releases the new code in the gmf map files
>>>>>> and then
>>>>>> runs a
>>>>>> > build. This would be a completely hands off run a nightly
>>>>>> Integration
>>>>>> build
>>>>>> > when a change is committed.
>>>>>> >
>>>>>> > Everything is now working fine, but the dependencies
>>>>>> calculator does
>>>>>> not
>>>>>> > work when firing a new GMF build.
>>>>>> >
>>>>>> > This is he explanation for the constant failing GMF builds,
>>>>>> you need
>>>>>> to pick
>>>>>> > the correct dependencies as well as using the linux-gtk-x86_64
>>>>>> Eclipse SDK.
>>>>>> >
>>>>>> > This is
>>>>>> _https://bugs.eclipse.org/bugs/show_bug.cgi?id=299802_ that
>>>>>> we have
>>>>>> > yet to fix.
>>>>>> >
>>>>>> > Last night I ran a integration build manually selecting what
>>>>>> I think
>>>>>> was the
>>>>>> > latest dependencies:
>>>>>> > _
>>>>>>
>>>>>> __http://modeling.eclipse.org/modeling/gmf/gmf/downloads/drops/2.3.0/I201001192155/buildlog.txt_
>>>>>>
>>>>>>
>>>>>> >
>>>>>> > It has failed in org.eclipse.gmf.xpand.migration
>>>>>> >
>>>>>> > Can the tooling confirm?
>>>>>> >
>>>>>> > The dependencies were:
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/eclipse/downloads/drops/I20100119-0800/eclipse-SDK-I20100119-0800-linux-gtk-x86_64.tar.gz_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/modeling/emf/emf/downloads/drops/2.6.0/I201001101746/emf-xsd-SDK-I201001101746.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/modeling/mdt/uml2/downloads/drops/3.1.0/S200912141514/mdt-uml2-SDK-3.1.0M4.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/tools/orbit/downloads/drops/R20090825191606/orbitBundles-R20090825191606.map_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://modeling.eclipse.org/modeling/emf/query/downloads/drops/1.4.0/S200912161005/emf-query-SDK-1.4.0M4.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://modeling.eclipse.org/modeling/emf/transaction/downloads/drops/1.4.0/S200912161108/emf-transaction-SDK-1.4.0M4.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://modeling.eclipse.org/modeling/emf/validation/downloads/drops/1.4.0/S200912161043/emf-validation-SDK-1.4.0M4.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/tools/gef/downloads/drops/3.6.0/I201001151504/GEF-SDK-I201001151504.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/modeling/m2m/qvtoml/downloads/drops/3.0.0/S200912160721/m2m-qvtoml-SDK-3.0.0M4.zip_
>>>>>>
>>>>>>
>>>>>> > -URL
>>>>>> > _
>>>>>>
>>>>>> __http://download.eclipse.org/modeling/mdt/ocl/downloads/drops/3.0.0/I201001070745/mdt-ocl-SDK-I201001070745.zip_
>>>>>>
>>>>>>
>>>>>> >
>>>>>> > Cheers...
>>>>>> > Anthony
>>>>>> > --
>>>>>> > Anthony Hunter _mailto:anthonyh@xxxxxx.com_
>>>>>> > Software Development Manager
>>>>>> > IBM Rational Software: Aurora / Modeling Tools
>>>>>> > Phone: 613-270-4613
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > gmf-dev mailing list
>>>>>> > _gmf-dev@eclipse.org_ <mailto:gmf-dev@xxxxxxxxxxx>
>>>>>> > _https://dev.eclipse.org/mailman/listinfo/gmf-dev_
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Nick Boldt :: JBoss by Red Hat
>>>>>> Productization Lead :: JBoss Tools & Dev Studio
>>>>>> Release Engineer :: Dash Athena _
>>>>>> __http://nick.divbyzero.com_ <http://nick.divbyzero.com/>
>>>>>> _______________________________________________
>>>>>> gmf-dev mailing list _
>>>>>> __gmf-dev@eclipse.org_ <mailto:gmf-dev@xxxxxxxxxxx> _
>>>>>> __https://dev.eclipse.org/mailman/listinfo/gmf-dev_
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> dash-dev mailing list
>>>>>> _dash-dev@eclipse.org_ <mailto:dash-dev@xxxxxxxxxxx>
>>>>>> _https://dev.eclipse.org/mailman/listinfo/dash-dev_
>>>>>>
>>>>>> _______________________________________________
>>>>>> dash-dev mailing list
>>>>>> dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
>>>>>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> dash-dev mailing list
>>>>>> dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
>>>>>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> dash-dev mailing list
>>>>> dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
>>>>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>>>
>>>
>>> -- EclipseCon 2010*Denis Roy*
>>> Manager, IT Infrastructure
>>> Eclipse Foundation, Inc. -- http://www.eclipse.org/
>>> Office: 613.224.9461 x224 (Eastern time)
>>> denis.roy@xxxxxxxxxxx <mailto:denis.roy@xxxxxxxxxxx>
>>>
>>>
>>> _______________________________________________
>>> dash-dev mailing list
>>> dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
>>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>>
>>
>>
>> --
>> Nick Boldt :: JBoss by Red Hat
>> Productization Lead :: JBoss Tools & Dev Studio
>> Release Engineer :: Dash Athena
>> http://nick.divbyzero.com
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> dash-dev mailing list
>> dash-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/dash-dev
>>
>
> _______________________________________________
> dash-dev mailing list
> dash-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/dash-dev
>
--
Cheers,
Chris Aniszczyk
http://aniszczyk.org
