Bug 387504 - Bugs in program argument parsing (compared to command line)
Summary: Bugs in program argument parsing (compared to command line)
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Debug (show other bugs)
Version: 4.2   Edit
Hardware: All All
: P3 normal (vote)
Target Milestone: 4.3 M3   Edit
Assignee: Markus Keller CLA
QA Contact:
URL:
Whiteboard:
Keywords: contributed
: 387026 (view as bug list)
Depends on:
Blocks:
 
Reported: 2012-08-17 14:47 EDT by Markus Keller CLA
Modified: 2019-10-15 06:07 EDT (History)
2 users (show)

See Also:
Michael_Rennie: review+


Attachments
Print.java (1.30 KB, text/plain)
2012-08-17 14:47 EDT, Markus Keller CLA
no flags Details
Fix for platform.debug (23.28 KB, patch)
2012-08-31 16:13 EDT, Markus Keller CLA
no flags Details | Diff
Fix for jdt.debug (1.29 KB, patch)
2012-08-31 16:14 EDT, Markus Keller CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Markus Keller CLA 2012-08-17 14:47:01 EDT
Created attachment 220018 [details]
Print.java

It should be possible to copy program parameters from the Eclipse launch configuration dialog or from the process properties dialog to the command line and get the same results as when launching from Eclipse.

While trying to fix bug 387026, I found a couple of bugs in the program argument parsing code (compared to command line).

The main omission is that command lines concatenate "quoted args" and non-whitespace characters that precede or follow the quoted string.
DebugPlugin#parseArguments(String),
ProcessPropertyPage#setCommandLine(Text, String), and StandardVMRunner#renderCommandLine(String[]) don't do that.

Further problems are the bugs in JDK's ProcessImpl on Windows:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6511002
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6518827
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6468220

... and Microsoft's "ingenious" quoting/escaping rules:
http://msdn.microsoft.com/en-us/library/a1y7w461.aspx

I first wanted to fix the problems I found, but then they became so numerous and platform-dependent that I had to time-box my efforts. I mainly opened this bug to document the problems in case someone wants to fix them.

If this bug is going to be fixed, then we should fix all problems at once. Otherwise, existing launch configurations will be broken, and even if users manage to recover, fixing the remaining issues will break them again.

The attached Print.java is a nice test case, but it misses some of the intricacies of Microsoft's "Parsing C Command-Line Arguments" (since I wrote it before I found that document).
Comment 1 Markus Keller CLA 2012-08-31 16:13:54 EDT
Created attachment 220626 [details]
Fix for platform.debug

I realized that bug 387026 cannot be fixed cleanly without fixing this bug. These two patches implement the complete solution that allows lossless round-trips (parse > render > parse). The tests verify that the underlying platforms agree with our code.

This patch also includes the fix for bug 387026.
Comment 2 Markus Keller CLA 2012-08-31 16:14:14 EDT
Created attachment 220627 [details]
Fix for jdt.debug
Comment 3 Markus Keller CLA 2012-08-31 16:15:32 EDT
*** Bug 387026 has been marked as a duplicate of this bug. ***
Comment 5 Markus Keller CLA 2012-09-05 12:40:43 EDT
> Looks good and even has a tonne of unit tests!

Yeah, this was a case where "test first" development really paid out
(because the API and the expected results were clear from the beginning).
Comment 6 Curtis Windatt CLA 2012-09-20 16:22:41 EDT
Reopening as this change appears to be causing 22 test failures on the build machines.  They pass when run locally.

The failures all look like the following:

testEmpty	Error	N/A

java.lang.NullPointerException
at org.eclipse.core.internal.runtime.Activator.getURLConverter(Activator.java:321)
at org.eclipse.core.runtime.FileLocator.toFileURL(FileLocator.java:205)
at org.eclipe.debug.tests.launching.ArgumentParsingTests.runCommandLine(ArgumentParsingTests.java:78)
at org.eclipe.debug.tests.launching.ArgumentParsingTests.execute(ArgumentParsingTests.java:61)
at org.eclipe.debug.tests.launching.ArgumentParsingTests.execute(ArgumentParsingTests.java:52)
at org.eclipe.debug.tests.launching.ArgumentParsingTests.testEmpty(ArgumentParsingTests.java:118)
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:650)
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:305)
at org.eclipse.test.UITestApplication$2.run(UITestApplication.java:197)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
...
Comment 7 Markus Keller CLA 2012-09-21 06:20:54 EDT
(In reply to comment #6)
Sorry about the NPE. I wrongly assumed that the test bundle would be unJARed when automated tests are run. I'll fix this for M3.
Comment 9 Markus Keller CLA 2012-09-26 11:15:56 EDT
ArgumentTests.testProgramArgEmptyString found a bug on *nix platforms.
Fixed with http://git.eclipse.org/c/platform/eclipse.platform.debug.git/commit/?id=7f144247727de01bfce8af95c05a898f6ec16d55
Comment 10 Markus Keller CLA 2012-09-28 10:46:47 EDT
(In reply to comment #9)
I should also have fixed that in the Windows parser. With that fix, I can also revert org.eclipse.jdt.debug.tests.core.ArgumentTests#testProgramArgOneQuote() to correctly parse a single " as one empty argument. (On the command line, this is only supported on Windows; in sh, it's a "continuation").

Fixed with http://git.eclipse.org/c/platform/eclipse.platform.debug.git/commit/?id=b644a77916417dbe0eafa2ea3b3842931e283abc and http://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=377a54c2a7fd426ef9cf643cc2766fe39162b69d
Comment 11 Markus Keller CLA 2012-10-10 09:15:37 EDT
Text#addSegmentListener(..) is missing an implementation on the Mac, see bug 383185. Changed the process properties page to use StyledText instead:

http://git.eclipse.org/c/platform/eclipse.platform.debug.git/commit/?id=a68f972007c011994af90bf8a7a9f1b81616f306
Comment 12 Markus Keller CLA 2012-10-10 09:16:41 EDT
> Text#addSegmentListener(..) is missing an implementation on the Mac

Bug 388578, I meant.
Comment 13 Markus Keller CLA 2012-10-24 21:51:48 EDT
Found another problem on Windows. My External Tools launch configuration to show the selected resource in the Windows Explorer doesn't work any more if the resource_loc contains spaces:
C:\WINDOWS\explorer.exe /E,/select=${resource_loc}

Analysis: I fell into the same trap as the designers of the ProcessBuilder and 
Runtime#exec(String[] cmdarray, ..) APIs. These APIs cannot be implemented properly on Windows, since a Windows process is not created with a list of arguments, but with a single commandline argument to the native CreateProcess method.

There's no "correct" way to split a Windows command line into arguments, since argument parsing is implemented separately by each application. Likewise, there's no "correct" way to quote/escape and merge an array of arguments into an equivalent command line. Some applications use the rules cited by http://msdn.microsoft.com/en-us/library/a1y7w461.aspx , but others don't.

E.g.:
explorer.exe /E,/select="C:\path\with spaces\embedded"   works
explorer.exe /E,/select=C:\path\with spaces\embedded     works
explorer.exe "/E,/select=C:\path\with spaces\embedded"   fails!

For External Tools launch configurations, the workaround from
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6511002 should not be applied. The workaround should be removed from DebugPlugin#exec(String[], ..) again. It should only be used in the AbstractVMRunner, where we know how the java.exe application will parse arguments.