Bug 571469 - Commands invoked with large number of argument/long arguments fails to execute
Summary: Commands invoked with large number of argument/long arguments fails to execute
Status: NEW
Alias: None
Product: CDT
Classification: Tools
Component: cdt-build-managed (show other bugs)
Version: Next   Edit
Hardware: All All
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact: Jonah Graham CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-24 10:24 EST by Torbjörn Svensson CLA
Modified: 2021-02-25 05:48 EST (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Torbjörn Svensson CLA 2021-02-24 10:24:09 EST
Generating long command lines in the makefile generator might result in failed builds.
A common case is when there is a large amount of object files that are locate in a large directory tree.

On Windows, it appears that the limit is roughly 8k characters, but other OS might have a higher limit.

One way to work around this limitation is to provide some, or all, command line arguments to GCC using the @file notation, but this notation is limited to GCC tools.
As the CDT toolchain integration is not constrained with only supporting GCC based toolchains, the makefile generators cannot assume that all tools support the @file notation.

I can see a few different solution to the problem:
- Automatically detect that the command line matches ".*(gcc|g\+\+|ar|ld)(.exe)?$" and automatically use the @file notation?
- Add another boolean option to the tool on the toolchain integration that simply identifies that the tool supports @file notation.
- Any other ideas?

The error shown thrown from make is:
make (e=206): The filename or extension is too long.
Comment 1 Christian Walther CLA 2021-02-24 11:56:34 EST
Our patches for this problem are posted on bug 72965, although they are probably not up to date with what we are currently shipping. (I could take a look as I’m in the process of updating to CDT 10.2 anyway.) We don't attempt to handle anything else than a GCC-compatible linker, using @file response files, though.

As far as I can see at a glance, we enable it using a method `boolean GnuMakefileGenerator.useResponseFiles()` that returns false but subclasses can override to return true, but your idea of a tool option is probably better than requiring everyone to subclass GnuMakefileGenerator.
Comment 2 Torbjörn Svensson CLA 2021-02-24 13:25:28 EST
(In reply to Christian Walther from comment #1)
> Our patches for this problem are posted on bug 72965, although they are
> probably not up to date with what we are currently shipping. (I could take a
> look as I’m in the process of updating to CDT 10.2 anyway.) We don't attempt
> to handle anything else than a GCC-compatible linker, using @file response
> files, though.
> 
> As far as I can see at a glance, we enable it using a method `boolean
> GnuMakefileGenerator.useResponseFiles()` that returns false but subclasses
> can override to return true, but your idea of a tool option is probably
> better than requiring everyone to subclass GnuMakefileGenerator.

Looking at those patches, they do about the same as I do in https://git.eclipse.org/r/c/cdt/org.eclipse.cdt/+/176552/2/build/org.eclipse.cdt.managedbuilder.core/src/org/eclipse/cdt/managedbuilder/makegen/gnu/GnuMakefileGenerator.java, just a little bit differently.
The solution for this topic needs to be generic enough to also support other tools, like the compiler with a lot of include paths that are long...

I was a little inspired by bug 72965 comment 47 and the use of file[1]. Maybe this can be used in a canned recipe[2] that would do the check on make level rather than java level to allow full benefit of the makefile.{defs,init,targets} hooks.
The need for each of the tools to, either identify that they support @file notation, or do some auto-detection based on their executable name is still needed though.

By the way, I think part of what you have in your patch is already implemented in HEAD now.

[1] https://www.gnu.org/software/make/manual/html_node/File-Function.html
[2] https://www.gnu.org/software/make/manual/html_node/Canned-Recipes.html
Comment 3 John Dallaway CLA 2021-02-25 04:09:47 EST
Another possibility would be to use the xargs tool, but we would need to consider how the user would be expected to acquire this tool on win32.

  xargs < objects.list gcc -o myfile.elf

Some versions of xargs support reading arguments from file directly:

  xargs -a objects.list gcc -o myfile.elf
Comment 4 jan baeyens CLA 2021-02-25 05:48:07 EST
Sloeber has this problem as well and Sloeber works around it by using sh.exe

Some background
Sloeber is working "out of the box" Therefore Sloeber checks at start-up on the existence of a toolchain and for windows Sloebers installs make (download and extract zip). On windows Sloeber calls make with a fully qualified name.

What livius (gnu mcu) learned me is that if you have an sh.exe in the same folder as make.exe the "windows command line limitation" is lifted.

off course there are some differences between the 2 (especially in quoting of quoutes) which caused I didn't fully switch yet. 
It is however a commonly used workaround.