Bug 312249 - Refactorings do not handle non-ASCII characters correctly
Summary: Refactorings do not handle non-ASCII characters correctly
Status: RESOLVED FIXED
Alias: None
Product: PTP
Classification: Tools
Component: Photran.Refactoring Engine (show other bugs)
Version: 5.0   Edit
Hardware: PC Linux
: P1 major (vote)
Target Milestone: 6.0   Edit
Assignee: Jeffrey Overbey CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-05-10 09:09 EDT by Michel DEVEL CLA
Modified: 2011-05-14 06:43 EDT (History)
1 user (show)

See Also:

Attachments
.log file of the workspace with traces of cases when refactoring fails (219.34 KB, text/x-log)
2010-05-10 09:09 EDT, Michel DEVEL CLA 		no flags Details
Unicode support patch (89.02 KB, patch)
2010-05-11 22:37 EDT, Jeffrey Overbey CLA 		g.watson: iplog+
Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Michel DEVEL CLA 2010-05-10 09:09:09 EDT 
Created attachment 167699 [details]
.log file of the workspace with traces of cases when refactoring fails

Consider the following little program:

program nonASCII_bug

! comment with a non ASCII character: é
do i=1,2
   print *,i
enddo

end

If the non-ASCII character "é" at the end of the comment line is removed, the 'introduce implicit none' refactoring works. if the "é" is there, it does not work.
I even think that it would be the case for any refactoring (I tested it also for the "Change Keyword case..." refactoring.

the workspace/.metadata/.log is included
Comment 1 Jeffrey Overbey CLA 2010-05-11 21:12:31 EDT 
Just reproduced this.

The example works as expected when the file is MacRoman encoded (default on OS X).

It gives "This refactoring does not change any source code" when the file is UTF-16 encoded.

When the file is UTF-8 encoded, it gives the error in the log file (org.eclipse.text.edits.MalformedTreeException: End position lies outside document range)
Comment 2 Jeffrey Overbey CLA 2010-05-11 22:37:40 EDT 
Created attachment 168076 [details]
Unicode support patch

The changes to support this were pervasive but not particularly complicated.  Attaching a patch for reference.
Comment 3 Jeffrey Overbey CLA 2010-05-11 22:39:12 EDT 
Patch committed to CVS.