Bug 494261 - JVM crash when associating external editor with file extension
Summary: JVM crash when associating external editor with file extension
Status: CLOSED DUPLICATE of bug 157010
Alias: None
Product: Platform
Classification: Eclipse Project
Component: SWT (show other bugs)
Version: 4.5.2   Edit
Hardware: PC Windows 7
: P3 critical (vote)
Target Milestone: ---   Edit
Assignee: Platform-SWT-Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-05-23 03:09 EDT by Eike Stepper CLA
Modified: 2016-05-24 02:21 EDT (History)
3 users (show)

See Also:


Attachments
Crash log (46.70 KB, application/octet-stream)
2016-05-23 03:09 EDT, Eike Stepper CLA
no flags Details
Neon RC1 crash log (73.97 KB, application/octet-stream)
2016-05-23 03:10 EDT, Eike Stepper CLA
no flags Details
Mars.2 crash log (startup) (51.70 KB, application/octet-stream)
2016-05-23 03:13 EDT, Eike Stepper CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Eike Stepper CLA 2016-05-23 03:09:38 EDT
Created attachment 261928 [details]
Crash log

Steps to reproduce:

1. Download, unzip and execute https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/mars/2/eclipse-java-mars-2-win32-x86_64.zip

2. Open the preference page "File Associations" and select any file type.

3. Press the lower "Add..." button to open the editor selection dialog.

4. Select the "External programs" radio button and pick any program.

5. Sometimes the JVM crashes immediately, sometimes when closing the editor selection dialog with OK, sometimes a little bit later, when closing the preference page with OK.
Comment 2 Eike Stepper CLA 2016-05-23 03:13:14 EDT
Created attachment 261930 [details]
Mars.2 crash log (startup)

One of my Mars.2 IDEs now crashes during startup. Seems I must delete it.
Comment 3 Niraj Modi CLA 2016-05-23 03:18:26 EDT
This bug seems duplicate of bug 157010.

*** This bug has been marked as a duplicate of bug 157010 ***
Comment 4 Andrey Loskutov CLA 2016-05-23 03:36:14 EDT
Eike, what would happen if you comment out code which loads images for external programs, for example in org.eclipse.ui.internal.misc.ExternalProgramImageDescriptor.getImageData? 
I suspect that crashes could be caused by loading broken or unusual images on your system.
Comment 5 Eike Stepper CLA 2016-05-23 04:08:01 EDT
I have no idea how to comment out code in a downloaded executable.
Comment 6 Andrey Loskutov CLA 2016-05-23 06:47:27 EDT
(In reply to Eike Stepper from comment #5)
> I have no idea how to comment out code in a downloaded executable.

I meant of course that you run Eclipse from debugger, with related UI code commented out. Since this issue is most likely not reproducible and very specific to your environment, it would be great if you could try to debug it and to find possible root cause. If you never had UI projects in your workspace, you can import them (with source) from yor current Eclipse or even let Eclipse import plugin wizard import them from Git.
Comment 7 Eike Stepper CLA 2016-05-23 08:54:28 EDT
(In reply to Andrey Loskutov from comment #6)
> Since this issue is most likely not reproducible and very
> specific to your environment, 

So, is this just a suspicion, or has a Platform committer already tried it out in the past ten years?
Comment 8 Andrey Loskutov CLA 2016-05-23 10:12:27 EDT
(In reply to Eike Stepper from comment #7)
> (In reply to Andrey Loskutov from comment #6)
> > Since this issue is most likely not reproducible and very
> > specific to your environment, 
> 
> So, is this just a suspicion, or has a Platform committer already tried it
> out in the past ten years?

I've recently did some work on that dialog and had a pleasure to debug/test exact the described use case with external editors a lot, with not a single crash. Hovewer I was on Linux /Windows 10, and have definitely different programs installed ;)
Comment 9 Eike Stepper CLA 2016-05-24 02:21:53 EDT
Here some more insights:

I've asked Ed to try it on his laptop and two things were different. In the list of external editors only the default text icon was displayed for all external programs. And we were not able to produce a VM crash there.

Back on my laptop, we were able to reproduce the crash in a debug launch. In org.eclipse.ui.internal.misc.ExternalProgramImageDescriptor.getImageData() we logged the program name to the console in the hope to see what program (icon) causes the problem. But the results were different each time we launched. A crash happened each time, but at totally different programs. That is in line with the observation that the crash even happens at different times, e.g., sometimes when displaying the list of external programs, sometimes when pressing OK for a selected external program, and sometimes only when pressing OK on the underlying preference dialog.

Then we started to comment out some pieces of code in ExternalProgramImageDescriptor.getImageData(). Surprisingly, even when we replaced the entire body with "return null;" the crash still occured.

Then we replaced the entire body of org.eclipse.swt.program.Program.getImageData() with "return null;" -- the crash still occured.

That all implies that the problematic code is some native code that is called before the program icons are loaded. So we replaced the org.eclipse.swt.program.Program.getPrograms() body with this:

	Program program = new Program();
	program.command="\"C:\\Program Files\\7-Zip\\7zFM.exe\" \"%1\"";
	program.extension=null;
	program.iconName="C:\\Program Files\\7-Zip\\7z.dll,9";
	program.name="001 Archive";
	
	Program [] programs = {program};
	return programs;

And now we could no longer reproduce a crash!

To us that suggests that the problem is somewhere in the OS.RegEnumKeyEx(...) call in Program.getPrograms(). This problem probably doesn't manifest itself immediately, but rather seems to corrupt the Java heap in some random way, so that subsequent heap access leads to VM crashes.