Re: [mylyn-dev] continuous testing

[Steffen Pingel <steffenp@xxxxxx>]

We have always taken a very agile development aproach and have been open to 
explore changes to our process. It's useful that you point out that there may 
have been mistakes in the commit messages. This will most likely only apply 
to commits made by myself. I'll watch out more carefully for that in the 
future. 

I don't believe it's reasonable to disregard the use of the new test result 
repository altogether because we are missing a few contexts. Also I find that 
often it is not necessary to attach a new context when fixing a test case as 
the relevant information is already in an existing context.

Steffen


On Thursday 20 December 2007, Eugene Kuleshov wrote:
> Steffen,
>
>   To clarify, it is not just about optimal. There been CVS commits
> (supposedly driven by these external tasks) that break Mylyn dev
> guidelines [1] because they didn't have corresponding Bugzilla task
> and/or associated task context. Maybe if current system can't work with
> Bugzilla, perhaps there should be some manual step to bring these tasks
> back to Bugzilla.
>
>   Support for multiple task repositories for given resource in Mylyn
> maybe is too exotic solution to this issue and from my knowledge of
> repository linking support in Mylyn, it would cause an intrusive API
> change. But if decided this stuff should be implemented before hand, so
> no task context info and linking from the change sets will be broken.
>
>   regards,
>   Eugene
>
> [1] http://wiki.eclipse.org/Mylyn_Contributor_Reference#Contributors
>
> Steffen Pingel wrote:
> > Eugene, thank you for your feedback. There are certainly numerous
> > improvements that can be made to the current aproach for continuous
> > testing. The setup is still work in progress. I aimed at getting the the
> > maximum value by spending the minimal amout of my spare time. Now that it
> > is  up and running we can get some mileage and evaluate if this will
> > provide enough value to make it part of our development process.
> >
> > I agree with your concern that it is not optimal to track the test
> > results separately from the Eclipse Bugzilla. At this point the tool I
> > created for reporting the test results is by no means sufficiently tested
> > or optimized to publish to a production repository though. It would
> > require significantly more work to harden the tool and permission from
> > the Eclipse webmasters etc. to do that.
> >
> > Contexts can be easily attached to the tickets that store the test
> > results and shared that way. Commit messages should point back to the
> > original Bugzilla report that is related to the cause of the test
> > failure. This workflow might impose additional overhead for committers
> > but it will also help us learn more about using multiple task
> > repositories for a single source base and optimize the tooling around
> > that use case if necessary.
> >
> > Please don't be discouraged by the fact that the orignial bug 213100 is
> > marked as resolved. I will read your comments and respond appropriately.
> > If the discussion brings up ideas that are feasible and there is time to
> > pursue them we can create additional bug reports.
> >
> > Steffen
> >
> > On Wednesday 19 December 2007, Eugene Kuleshov wrote:
> >> Steffen,
> >>
> >>   After looking at this for a few days I think there are some problems
> >> with this approach that need to be addressed somehow. I've added few
> >> comments to the suggested bug report, though it is odd, because report
> >> is already closed as resolved.
> >>
> >>   Anyways, the biggest issue is that it introduced second source of
> >> issues that is participating in the code changes (in other words change
> >> sets and CVS commits, history, annotations, etc), which is not listed as
> >> the project linked issue tracker. So we are losing contextual
> >> information about these changes, including the task contexts.
> >>
> >>   I am not sure what would be the right way to resolve this, but maybe
> >> continuous testing tool should use project Bugzilla for all reporting.
> >>
> >>   regards,
> >>   Eugene
> >>
> >> Steffen Pingel wrote:
> >>> It's in CVS in sandbox/org.eclipse.mylyn.test.report. I'll add some
> >>> documentation about how to run it in the next few days.
> >>>
> >>> Steffen
> >>>
> >>> On Sunday 16 December 2007, Eugene Kuleshov wrote:
> >>>>   Great stuff Steffen! I wonder if code for your headless application
> >>>> available? It would be interesting to know how all the pieces are
> >>>> glued together.
> >>>>
> >>>>   Thanks
> >>>>
> >>>>   Eugene
> >>>>
> >>>> Steffen Pingel wrote:
> >>>>> I am in the process of setting up a continuous build process that
> >>>>> will run the Mylyn AllTests suite and AllPerformanceTest suite
> >>>>> nightly. I have implemented a headless application based on Mylyn
> >>>>> that publishes test results to a Trac repository:
> >>>>>
> >>>>>  http://mylyn.eclipse.org/bitten/
> >>>>>
> >>>>> Every open ticket corresponds to a failing test in the latest nightly
> >>>>> build. The last comment on a ticket describes the cause of the test
> >>>>> failure including a stack trace. Once a test succeeds the ticket is
> >>>>> closed. It is reopened if a test fails during a future build.
> >>>>>
> >>>>> Monitoring tests is easy by creating a query against the repository
> >>>>> that includes all tickets (the repository allows anonymous access).
> >>>>> Please note that some of the tests failed due to running in a
> >>>>> deployed environment.
> >>>>>
> >>>>> The goal is to make this a seamless part of the Mylyn development
> >>>>> workflow to help us avoid having failed tests go unnoticed. Please
> >>>>> comment on this bug for feedback and suggestions:
> >>>>>
> >>>>>  213100: automate publishing of failed tests
> >>>>>  https://bugs.eclipse.org/bugs/show_bug.cgi?id=213100
> >>>>>
> >>>>> Steffen
> >>>>
> >>>> _______________________________________________
> >>>> mylyn-dev mailing list
> >>>> mylyn-dev@xxxxxxxxxxx
> >>>> https://dev.eclipse.org/mailman/listinfo/mylyn-dev
> >>
> >> _______________________________________________
> >> mylyn-dev mailing list
> >> mylyn-dev@xxxxxxxxxxx
> >> https://dev.eclipse.org/mailman/listinfo/mylyn-dev
>
> _______________________________________________
> mylyn-dev mailing list
> mylyn-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/mylyn-dev



-- 
Steffen Pingel - steffenp@xxxxxx - http://steffenpingel.de