Bug 540911 - Terminal cannot display Chinese characters correctly
Summary: Terminal cannot display Chinese characters correctly
Status: NEW
Alias: None
Product: CDT
Classification: Tools
Component: terminal (show other bugs)
Version: Next   Edit
Hardware: PC All
: P3 normal with 1 vote (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact: Jonah Graham CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-08 05:09 EST by han zhi CLA
Modified: 2021-07-09 21:35 EDT (History)
4 users (show)

See Also:


Attachments
only part of Chinese characters can be displayed (6.67 KB, image/png)
2018-11-08 05:09 EST, han zhi CLA
no flags Details
Some Chinese characters cannot be displayed (7.37 KB, image/png)
2018-11-08 05:11 EST, han zhi CLA
no flags Details
show Chinese Character twice (26.14 KB, image/png)
2020-05-04 13:58 EDT, Gao Hao CLA
no flags Details
animated gif of me reproducing the failure (248.31 KB, image/gif)
2020-05-05 18:04 EDT, Jonah Graham CLA
no flags Details
screenshot where I hack the terminal code as a test (8.10 KB, image/png)
2020-05-05 20:05 EDT, Jonah Graham CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description han zhi CLA 2018-11-08 05:09:07 EST
Created attachment 276506 [details]
only part of Chinese characters can be displayed

Terminal can't display Chinese characters correctly, only part of Chinese characters can be displayed
Comment 1 han zhi CLA 2018-11-08 05:11:36 EST
Created attachment 276508 [details]
Some Chinese characters cannot be displayed
Comment 2 han zhi CLA 2018-11-08 05:14:04 EST
The character set used is UTF-8
Comment 3 Jonah Graham CLA 2020-05-01 10:08:45 EDT
The Terminal component of the Eclipse Ecosystem has a new home. The Terminal is now part of the Eclipse CDT project[1].

This change means a new Git repo[2], P2 site[3] and Bugzilla component. The terminal will continue to be delivered as part of the quarterly Simultaneous Release of Eclipse as well.

The marketplace entry[4] had not been updated in a few years. It will once again install the latest release of the terminal on the latest release of the whole IDE (currently 2020-03).

If this bug is no longer relevant, please feel free to comment or close the bug. If you can confirm if this issues still occurs in the latest release please do let me know in a comment.

[1] https://wiki.eclipse.org/CDT/User/NewIn911
[2] https://git.eclipse.org/c/cdt/org.eclipse.cdt.git (in the terminal directory)
[3] current release is 9.11 - P2 site https://download.eclipse.org/tools/cdt/releases/9.11/
[4] https://marketplace.eclipse.org/content/tm-terminal

(This comment was added to all open terminal bugs along with changing the Product/component pair to CDT/terminal.)
Comment 4 Gao Hao CLA 2020-05-04 13:58:05 EDT
Created attachment 282694 [details]
show Chinese  Character twice
Comment 5 Gao Hao CLA 2020-05-04 14:03:53 EDT
There are two main problems:
1) show Chinese Characters twice
2) black background for all characters, which is ugly
Comment 6 Gao Hao CLA 2020-05-04 14:09:24 EDT
One workaround for the problem 1) is to Use Legacy Console Mode (https://docs.microsoft.com/zh-cn/windows/console/legacymode#using-legacy-console-mode), restart the system and the Eclipse IDE, but I think this workaround is not good.
Comment 7 Jonah Graham CLA 2020-05-05 17:16:01 EDT
(In reply to Gao Hao from comment #6)

Thank you for confirming the problem is still present.

I am hoping the winpty upgrade in Bug 562776 will resolve the issue of misrendered characters - however looking through the code their may remain an additional problem because the terminal code may miscalculate character sizes, so I may need to compare how and what the terminal rendering code does compared to the standard text editors in Eclipse.

As for colors, I am trying to progress the color handling. Please follow Bug 549697 which is about fixing background colors and all colors - in addition Bug 540737 adds full 8 and 24-bit color.
Comment 8 Jonah Graham CLA 2020-05-05 18:04:20 EDT
Created attachment 282714 [details]
animated gif of me reproducing the failure
Comment 9 Jonah Graham CLA 2020-05-05 18:34:52 EDT
This is not a Windows problem. I can reproduce my animated gif issue on  Linux too. It is something about calculating the size of fonts. 

I think it is something about StyleMap.updateFont(String) and measureChar.
Comment 10 Jonah Graham CLA 2020-05-05 19:25:43 EDT
I see similar (but much less significant) problems in the editor too, for example using this shows the W in line 2 and 3 don't line up as expected.

// echo  汉字汉字汉字汉字汉字汉字汉字
// 123456汉汉汉汉汉汉汉aaaa汉字汉字汉W
// WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
// llllllllllllllllllllllllllllll
// 123456789012345678901234567890
//          WWWWWWWWWWWWWWlWlWlWW
Comment 11 Jonah Graham CLA 2020-05-05 20:05:27 EDT
Created attachment 282716 [details]
screenshot where I hack the terminal code as a test

There may be minor issues in the editor - but the terminal problem is much more serious. The only workaround I see is using a font that is indeed fixed width for Chinese characters. I am not experienced in this area, so I may misunderstand something here.

The problem in the terminal code is that it essentially assumes all characters are the same width. So what is happening is when a Chinese character is drawn it is wider than the widest A-Z character, and then the next character is drawn on top.

My editing the code to report all characters as twice the width of a W produces non-overlapped Chinese characters, but then leaves large gaps.

So the problem here is the current code design assumes the whole terminal is a grid - that is what needs to be addressed to proceed on this. 


For reference I made the above screenshot by applying this diff.

diff --git a/terminal/plugins/org.eclipse.tm.terminal.control/src/org/eclipse/tm/internal/terminal/textcanvas/StyleMap.java b/terminal/plugins/org.eclipse.tm.terminal.control/src/org/eclipse/tm/internal/terminal/textcanvas/StyleMap.java
index 605c50e..4f5de61 100644
--- a/terminal/plugins/org.eclipse.tm.terminal.control/src/org/eclipse/tm/internal/terminal/textcanvas/StyleMap.java
+++ b/terminal/plugins/org.eclipse.tm.terminal.control/src/org/eclipse/tm/internal/terminal/textcanvas/StyleMap.java
@@ -198,7 +198,7 @@
 	}
 
 	public int getFontWidth() {
-		return fCharSize.x;
+		return fCharSize.x * 2;
 	}
 
 	public int getFontHeight() {
@@ -296,7 +296,7 @@
 	}
 
 	public boolean isFontProportional() {
-		return fProportional;
+		return true;// fProportional;
 	}
 
 	/**
@@ -307,6 +307,6 @@
 	public int getCharOffset(char c) {
 		if (c >= fOffsets.length)
 			return 0;
-		return fOffsets[c];
+		return fOffsets[c] + getFontWidth() / 4;
 	}
 }
Comment 12 Jonah Graham CLA 2020-05-05 20:07:08 EDT
Please disregard comment 10 - it is not valid.
Comment 13 Jonah Graham CLA 2021-05-24 20:47:11 EDT
The problem here is not related to WinPTY vs ConPTY (although the upgrade changes the behaviour, not better or worse, just different amounts of overlapping)
Comment 14 Eclipse Genie CLA 2021-07-09 21:35:04 EDT
New Gerrit change created: https://git.eclipse.org/r/c/cdt/org.eclipse.cdt/+/182953