Community
Participate
Working Groups
Due to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7023180, tests aren't run in the order specified in source. I'm only partly interested in that problem being addressed -- I get it. But, as a developer who is relying on coding in a certain order, and expecting to review the results of those tests in the same order, I'd like to view the results in an order I expect. Since you can parse the AST, showing the original ordering of the tests would be preferred. I realize this can't happen in cases where source is not accessible. The problem is that the tool is now no longer serving me useful results; it's serving me confusing results. I need Eclipse to be my assistant here and provide me better information. Please consider providing something helpful here in the near future. Thanks, Robert
(In reply to comment #0) > Due to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7023180, tests > aren't run in the order specified in source. I'm only partly interested in > that problem being addressed -- I get it. But, as a developer who is relying > on coding in a certain order, and expecting to review the results of those > tests in the same order, I'd like to view the results in an order I expect. If the developer relies on the order, then he most likely also relies on the order when executing those tests, hence simply changing the view won't help you much. It was a design decision of JUnit not to guarantee/specify a certain order because tests should not depend on each other in general. Having said that, there are scenarios where a developer explicitly relies on a given order. For that you can use org.eclipse.test.OrderedTestSuite (see bug 369818 for more details). *** This bug has been marked as a duplicate of bug 369818 ***
> Please consider providing something helpful here in the near future. Please talk to Oracle about their decision to screw callers of Class#getDeclaredMethods() (instead of strengthening the method contract to return the methods in declaration order). A second possible solution would be a workaround in JUnit: TestSuite and TestClass could e.g. use Class#getResourceAsStream(String) and get the declared order by parsing the class file.
This is not a duplicate of 369818. There is the use case where the user enjoys having the tests run in random order to make sure the results are independent, but they would like the results listed in the Eclipse UI in the order of the source file. If you add a new test to the end of the file, and you expect it to fail, it will be jarring to see a test in the middle of the list fail instead of at the end of the list. Likewise, if you have a bunch of tests with a common prefix corresponding to many tests on a single method, like testFoo_Bar, testFoo_Jar, testFoo_Yum... and you put them next to each other in the source file, you'd like to see them next to each other in the results -- maybe they all failed for some reason (like Foo is completely broken now due to some dependency) and it's nice to see the failures next to each other.
(In reply to comment #3) > Likewise, if you have a bunch of tests with a common > prefix corresponding to many tests on a single method, like testFoo_Bar, > testFoo_Jar, testFoo_Yum... and you put them next to each other in the > source file, you'd like to see them next to each other in the results -- > maybe they all failed for some reason (like Foo is completely broken now due > to some dependency) and it's nice to see the failures next to each other. Would it make more sense to always display the tests in alphabetical order then?
I'd say alphabetical is much better than random, but obviously the order they're in the file would be best. I think a separate bug to make it alphabetical until someone has time to do the hard way would be good.
Thanks to everybody for their prompt replies. > If the developer relies on the order, then he most likely also relies > on the order when executing those tests, hence simply changing the > view won't help you much. Dani, I see what you mean, but I can say with certainty that when applying that to myself, I care about seeing results in the order which I lay out the code. Execution order doesn't matter to me - if it did I would be writing deterministic tests; which is bad. So I am willing to accept not running tests in a given order. I just care about the order they're reported in. But even more so, at Google this is understood throughout the organization. So I can assure you that I represent a large number of engineers who care about that. > It was a design decision of JUnit not to guarantee/specify a certain > order because tests should not depend on each other in general. Having > said that, there are scenarios where a developer explicitly relies on > a given order. For that you can use org.eclipse.test.OrderedTestSuite > (see bug 369818 for more details). I agree that would help with forcing an order, but it's not something we need. Google has plenty of test cases that already define ordering in a certain way, even test suite builders that implicitly randomize their order. I need to visually correlate failures and successes based on how I lay out my code. In the same way that I can view a certain order using, in the same way that the Outline View and Quick Outline View rely on order for comprehension. Can we at least consider this separate from what you marked as a duplicate? (Note: reopening the bug; I've been told that's within the bounds of social acceptance in the Eclipse Bugzilla world. :)) (I should also point out that OrderedTestSuite won't work well when performing a rename refactoring [if I read the sample code correctly.]) > Please talk to Oracle about their decision to screw callers of > Class#getDeclaredMethods() (instead of strengthening the method contract > return the methods in declaration order). Markus, I hear you. Google has people who actively talk with Oracle about Java. Plus I saw your comment many months back. I don't know much about the feasibility of fixing this in a future JDK, perhaps I'm more pessimistic than others about this, and so just want to see results in a way that makes sense. For a use case where that happens, see below. > Would it make more sense to always display the tests in alphabetical order then? I don't think so. Here's an example. I tend to have a first test in my list called testSanity(), which may just test a simplest use case (call a constructor, call the no-arg method, make sure it doesn't throw an exception.) I want to see that on the top of my results; it's a clue that something more serious than a specific case has failed. I don't want to have to rename it to, for instance, testAaaa. So, I'm not asking for the tests to be run in a separate order; I'm asking for the UI to help me, and provide a UI that presents the results in a way I would expect. By the way, in my opinion an ideal bug fix would be one where a button toggles between execution order and code order.
(In reply to comment #6) > Thanks to everybody for their prompt replies. > > > If the developer relies on the order, then he most likely also relies > > on the order when executing those tests, hence simply changing the > > view won't help you much. > > Dani, I see what you mean, but I can say with certainty that when applying > that to myself, I care about seeing results in the order which I lay out the > code. Execution order doesn't matter to me - if it did I would be writing > deterministic tests; which is bad. So I am willing to accept not running > tests in a given order. I just care about the order they're reported in. But > even more so, at Google this is understood throughout the organization. So I > can assure you that I represent a large number of engineers who care about > that. > > > It was a design decision of JUnit not to guarantee/specify a certain > > order because tests should not depend on each other in general. Having > > said that, there are scenarios where a developer explicitly relies on > > a given order. For that you can use org.eclipse.test.OrderedTestSuite > > (see bug 369818 for more details). > > I agree that would help with forcing an order, but it's not something we > need. Google has plenty of test cases that already define ordering in a > certain way, even test suite builders that implicitly randomize their order. > I need to visually correlate failures and successes based on how I lay out > my code. In the same way that I can view a certain order using, in the same > way that the Outline View and Quick Outline View rely on order for > comprehension. Can we at least consider this separate from what you marked > as a duplicate? (Note: reopening the bug; I've been told that's within the > bounds of social acceptance in the Eclipse Bugzilla world. :)) > > (I should also point out that OrderedTestSuite won't work well when > performing a rename refactoring [if I read the sample code correctly.]) > > > Please talk to Oracle about their decision to screw callers of > > Class#getDeclaredMethods() (instead of strengthening the method contract > > return the methods in declaration order). > > Markus, I hear you. Google has people who actively talk with Oracle about > Java. Plus I saw your comment many months back. I don't know much about the > feasibility of fixing this in a future JDK, perhaps I'm more pessimistic > than others about this, and so just want to see results in a way that makes > sense. For a use case where that happens, see below. > > > Would it make more sense to always display the tests in alphabetical order then? > > I don't think so. Here's an example. I tend to have a first test in my list > called testSanity(), which may just test a simplest use case (call a > constructor, call the no-arg method, make sure it doesn't throw an > exception.) I want to see that on the top of my results; it's a clue that > something more serious than a specific case has failed. I don't want to have > to rename it to, for instance, testAaaa. > > So, I'm not asking for the tests to be run in a separate order; I'm asking > for the UI to help me, and provide a UI that presents the results in a way I > would expect. By the way, in my opinion an ideal bug fix would be one where > a button toggles between execution order and code order. Maybe two options then: 1) test execution order, 2) outline view order. Outline view has a choice of viewing either alphabetically or in code layout order. As a user I would want the JUnit view to be consistent with the outline view if I care about ordering.
Probably. If the outline view matches the code order, I'll be behind that entirely. In a case where I'm running a test class where I don't have the .java file (which unfortunately happens from time to time at Google) I'd still want the testSanity case to work, if possible.
> I'm asking for the UI to help me, and provide a UI that presents the results > in a way I would expect. Fair enough. We can keep this bug open, but we will definitely not work on this. If you want this, please provide a high quality patch. If you do so, you need to make sure that the UI doesn't look noisy when it marks tests as passed or failed while executing the tests. And I would not want it depend on the Outline view.
Note that in general, JUnit test cases don't have to correspond to test methods. That's just what TestSuite and the JUnit4 runner do. The implementation has make sure it doesn't break when test classes can't be found.
There's a relevant question how to order JUnit tests in Stack Overflow, here's the link http://stackoverflow.com/questions/512778/ordering-unit-tests-in-eclipses-junit-view/27891022#27891022
The correct Stack Overflow link is http://stackoverflow.com/questions/512778/ordering-unit-tests-in-eclipses-junit-view/27891022 . I accidentally included superfluous anchor in the previous comment.
In general all tests should agree that the order of the tests should not impact the result. However, if it does by accident, it is currently a big pain to find the reason by debugging in Eclipse. I have a project where the tests fail in maven but succeed in Eclipse. I can figure out the order that was used in maven and would like to reproduce the bug in Eclipse debugger so I can fix the tests. If I select multiple tests in Eclipse projects/project explorer/package explorer, It would be awesome if I could run them as JUnits in the order I selected them. Otherwise if I have to run via package/folder, it would be nice to have an option afterwards to rerun the set of tests in a different custom order somehow.
May 2 cents on this. I don't really care the order in which tests are executed (I think JUnit is already doing the best for me), what I really miss is the JUnit view to let me sort the executed tests in alphabetical order (or some other reasonable criteria) for VIEW PURPOSES ONLY!! Right now, my tests are all scrambled, if I have to find one of them to see if it has passed or failed, I must scan them from top to bottom... and since I have hundreds tests and I'm not a computer, it's quite a hard work. Yes, I can jump from one failure to another and assume that if I don't find my test it means it has passed, but that is quite an annoying workaround... Also, I've not read the full discussion, but it seems like to me the situation got worst recently, because I can't remember this to be so hard some months/years ago, probably it's due to a change in JDK or JUnit? Tests are currently not sorted in declaration order either.
(In reply to Mauro Molinari from comment #14) > Also, I've not read the full discussion, but it seems like to me the > situation got worst recently, because I can't remember this to be so hard > some months/years ago, probably it's due to a change in JDK or JUnit? Tests > are currently not sorted in declaration order either. This is due to a change/optimization in JDK 7.
Concur with Comment 14. I don't care which order the tests are executed, I just want to view all the tests I have written in the same section. So if the solution is a right click to order by method name, that would be sufficient. By having the tests ordered by method name, I can easily show others in a peer review that my tests pass.
(In reply to brandon autrey from comment #16) > Concur with Comment 14. I don't care which order the tests are executed, I > just want to view all the tests I have written in the same section. So if > the solution is a right click to order by method name, that would be > sufficient. By having the tests ordered by method name, I can easily show > others in a peer review that my tests pass. See bug 219466.
