Bug 24804 - Organize imports wipes comments between statements [code manipulation]
Summary: Organize imports wipes comments between statements [code manipulation]
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 2.0.1   Edit
Hardware: All All
: P3 major with 3 votes (vote)
Target Milestone: 3.8 M2   Edit
Assignee: Olivier Thomann CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 3772 32604 34823 48461 61695 102939 242042 263640 332879 (view as bug list)
Depends on:
Blocks: 376734
  Show dependency tree
 
Reported: 2002-10-15 15:10 EDT by Ritchie Schacher CLA
Modified: 2012-04-13 11:28 EDT (History)
16 users (show)

See Also:


Attachments
First draft (10.73 KB, patch)
2011-08-19 13:42 EDT, Olivier Thomann CLA
no flags Details | Diff
Patch v2 (10.90 KB, patch)
2011-08-19 13:51 EDT, Olivier Thomann CLA
no flags Details | Diff
Patch v3 (15.67 KB, patch)
2011-08-19 16:28 EDT, Olivier Thomann CLA
no flags Details | Diff
Patch v4 (15.67 KB, patch)
2011-08-31 14:26 EDT, Olivier Thomann CLA
no flags Details | Diff
Patch v5 (22.69 KB, patch)
2011-08-31 15:45 EDT, Olivier Thomann CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ritchie Schacher CLA 2002-10-15 15:10:00 EDT
Any comments between import statements, eg, copyrights, are lost when doing 
organize imports.  They should be preserved.
Comment 1 Martin Aeschlimann CLA 2002-10-16 04:36:51 EDT
The problem is, how? Are these comments belonging to an import, to a group?
I'm afraid there can't be done much.
Comment 2 Dirk Baeumer CLA 2003-02-05 16:14:36 EST
There are plans to support comments in ASTs. Postponing until we have more 
support here.
Comment 3 Dirk Baeumer CLA 2003-12-11 11:18:13 EST
*** Bug 48461 has been marked as a duplicate of this bug. ***
Comment 4 Dirk Baeumer CLA 2004-05-11 08:42:15 EDT
*** Bug 61695 has been marked as a duplicate of this bug. ***
Comment 5 Dani Megert CLA 2008-07-31 02:55:36 EDT
*** Bug 242042 has been marked as a duplicate of this bug. ***
Comment 6 Dani Megert CLA 2009-02-05 02:32:20 EST
*** Bug 102939 has been marked as a duplicate of this bug. ***
Comment 7 Dani Megert CLA 2009-02-05 02:32:25 EST
*** Bug 34823 has been marked as a duplicate of this bug. ***
Comment 8 Dani Megert CLA 2009-02-05 02:33:29 EST
*** Bug 3772 has been marked as a duplicate of this bug. ***
Comment 9 Dani Megert CLA 2009-02-05 02:34:06 EST
*** Bug 32604 has been marked as a duplicate of this bug. ***
Comment 10 Dani Megert CLA 2009-02-05 02:41:17 EST
ImportRewrite is now owned by JDT Core.
Comment 11 Dani Megert CLA 2009-02-05 02:41:27 EST
*** Bug 263640 has been marked as a duplicate of this bug. ***
Comment 12 Bouchet Stéphane CLA 2009-02-05 03:49:17 EST
The problem is that import reorganize moves the imports, add some and remove some according to the preferences (especially the reordering). 

My problem is that we use comments to specify a block that should not be reorganized : 
import java.util.List;
//Start of user code
import java.util.ArrayList;
import mycompany.MyObject;
//End of user code

i understand that is difficult to do, and there is lot of questions ( how to do with just one comment ? remove unused imports ? where to add new ones ?? )


cheers,

Stéphane
Comment 13 Dani Megert CLA 2010-12-20 04:18:12 EST
*** Bug 332879 has been marked as a duplicate of this bug. ***
Comment 14 Micha Riser CLA 2011-08-16 10:51:24 EDT
This is an important issue and should finally be addressed. There are pragmatic solutions to the problems mentioned. 

Say, divide the import statements into ''import groups'' which are separated by comments, then
 * Do not move imports out of their import group, only order the imports in each import group relatively to each other.
 * If imports are merged place the merged imports where the first of the original imports has been.
 * If new imports are added, add them below the best matching exisitng import. (Or more pragmatically at the end of the list.)
 * If a comment is on the same line behind an import, do not merge that import with others.

This may not be perfect but is certainly better than just throwing away comments anywhere near the imports!
Comment 15 John Arthorne CLA 2011-08-16 11:09:12 EDT
(In reply to comment #14)
> This is an important issue and should finally be addressed. There are pragmatic
> solutions to the problems mentioned. 
> 
> Say, divide the import statements into ''import groups'' which are separated by
> comments, then
>  * Do not move imports out of their import group, only order the imports in
> each import group relatively to each other.
>  * If imports are merged place the merged imports where the first of the
> original imports has been.
>  * If new imports are added, add them below the best matching exisitng import.
> (Or more pragmatically at the end of the list.)
>  * If a comment is on the same line behind an import, do not merge that import
> with others.
> 
> This may not be perfect but is certainly better than just throwing away
> comments anywhere near the imports!

There is some limited support for "import groups" on the Organize Imports preference page. It seems to me conceptually user-organized imports such as you describe is incompatible with the concept of Organize Imports. Personally I would be happy if the comments were preserved in any form (such as all comments moved above first import, or each comment stays with the import immediately below it). To me the major bug is the pure loss of user data today, such as comment #0 where the copyright was accidentally placed between imports and got deleted. I find it doubtful that an automatic import tool can preserve the correct intended position of all comments in all cases.
Comment 16 Micha Riser CLA 2011-08-16 11:25:28 EDT
John Arthorne, what options on the Organize Import do you mean?

There should really be a way to not delete the comments when rewriting the imports. Simple example:

// --- begin code --
package bla;

import java.util.ArrayList;

// added by user xy:
import java.util.Collection;

public class Foo {

	ArrayList<Integer> foo;
	
}
// --- end code --

Pressing organize imports will remove the comment "// added by user xy:" as well as the import below. Why not just leave the comment as is?
Comment 17 John Arthorne CLA 2011-08-16 12:23:56 EDT
(In reply to comment #16)
> John Arthorne, what options on the Organize Import do you mean?

The Organize Imports preference page allows some limited amount of grouping - for example com.* imports before java.* imports, etc.

> Pressing organize imports will remove the comment "// added by user xy:" as
> well as the import below. Why not just leave the comment as is?

I completely agree we should not delete the comments. I'm just saying it won't be possible for an automated tool to always preserve the correct comment position. To modify your example:

// --- begin code --
package bla;

import java.util.ArrayList;

// added by user xy:
import java.util.Collection;

import java.util.List;

public class Foo {

    List<Integer> foo = new ArrayList<Integer>();

}
// --- end code --

Using a simple rule to preserve comments we might end up with:

// --- begin code --
package bla;

import java.util.ArrayList;
// added by user xy:
import java.util.List;

public class Foo {

    List<Integer> foo = new ArrayList<Integer>();

}
// --- end code --

Now it looks like the java.util.List was added by user xy, which is not what the comment author intended. Guessing the right place for all comments in the general case is not possible. I am just arguing that fixing the data loss problem is more important than trying to come up with a complicated algorithm to preserve intended comment positions.
Comment 18 Dani Megert CLA 2011-08-19 04:31:34 EDT
At least the data loss needs to be fixed for 3.8.
Comment 19 Olivier Thomann CLA 2011-08-19 07:56:56 EDT
I'll take a look.
Comment 20 Olivier Thomann CLA 2011-08-19 13:42:09 EDT
Created attachment 201821 [details]
First draft

I haven't heavily tested it, but this is one step in the right direction.
Since the organize import operation is called to not preserve existing imports it is difficult to position the comments relative to the previous imports even if they are the same.
At least with this patch, the comments are not lost anymore. They are all moved to the end of the imports after the rewrite.
Comment 21 Olivier Thomann CLA 2011-08-19 13:45:46 EDT
I have a small issue with the comment before the import (between the package declaration and the first import). With the patch, it ends up being duplicated.

I am working on it.
Comment 22 Olivier Thomann CLA 2011-08-19 13:51:43 EDT
Created attachment 201822 [details]
Patch v2

Fix the issue with first comment.
Comment 23 Olivier Thomann CLA 2011-08-19 15:10:37 EDT
Now I need to add support for comments inside the imports that are not part of any import extended range. Will be fixed with the next patch.
Comment 24 Olivier Thomann CLA 2011-08-19 16:28:31 EDT
Created attachment 201837 [details]
Patch v3

This seems to preserve all existing comments. I'll test it more in the case the existing imports are preserved inside the ImportRewriteAnalyzer.
Comment 25 Olivier Thomann CLA 2011-08-31 14:26:45 EDT
Created attachment 202555 [details]
Patch v4

Same with minor typos fixed.
I won't work on relocating comments around existing imports. The current patch makes sure that no comments end up being lost.
I'll release it for now. This should be good enough to close this bug.
Comment 26 Olivier Thomann CLA 2011-08-31 15:43:37 EDT
Markus,
3 tests are failing in the org.eclipse.jdt.ui.tests.core.ImportOrganizeTest tests.

I'll provide a patch for those.
Comment 27 Olivier Thomann CLA 2011-08-31 15:45:35 EDT
Created attachment 202562 [details]
Patch v5

This patch removes the whitespaces in front of comments when preserving them and it contains the patch for jdt.ui test failures (tests with comments inside imports that were expected to be lost).
Markus, let me know when you want to release the fix in the tests. We need to synchronize the release of this patch.
Comment 28 Paul Benedict CLA 2011-09-01 10:44:32 EDT
If comments between imports are important to the developer, the developer should be able to turn off reorganization for those files. In the spirit of Bug 311617, can something like //@organizeimports:off be implemented? If that comment exists before the imports, skip the file.
Comment 29 Dani Megert CLA 2011-09-02 02:22:04 EDT
(In reply to comment #28)
> If comments between imports are important to the developer, the developer
> should be able to turn off reorganization for those files. In the spirit of Bug
> 311617, can something like //@organizeimports:off be implemented? If that
> comment exists before the imports, skip the file.

There are no plans to do this, but if someone signs up to implement this, then we would look at a good quality patch.
Comment 30 Markus Keller CLA 2011-09-02 12:26:51 EDT
I haven't reviewed the whole patch, but I played around with it and the behavior looked good (just accumulates all comments inside the imports section at the end, in original order).

Released the UI test to HEAD.
Comment 31 Olivier Thomann CLA 2011-09-02 12:31:28 EDT
Released for 3.8M2.
Comment 32 Satyam Kandula CLA 2011-09-13 05:27:49 EDT
Verified for 3.8M2 using build I20110912-0800