Community
Participate
Working Groups
This bug tracks discussion on enhancements to CDT's build model to improve support for other managed build systems, such as CMake, Qt's qmake, autotools configure, and other vendor build systems. The hope is to provides something that everyone can plug into and take advantage of CDT's build system for things like scanner discovery, and other automations. I'll be attaching a document shortly describing what I want to do.
Created attachment 175675 [details] Proposal v0.1 This is really rough, but it gives an idea of where I'm heading. Please comment.
BTW, I also need to incorporate scanner discovery into these changes. I was going to make that a separate bugzilla, but it's probably better to talk about it in conjunction with this proposal.
> * Active configuration for indexing (scanner config) It would be nice to have multiple databases (bug 183781) to enable that. "As we work through examples, we'll update the build model above to ensure it works well." I created bug 321470 for another builder I'd like to see. Not sure if it's feasible yet :)
There is a lot of good discussion in bug #296935 (reference added above). How do we get that integrated into the proposal here?
> to improve support for other managed build systems, such as CMake, > Qt's qmake, autotools configure, and other vendor build systems. I'm thinking more about what "improve support" means - particulary with respect to the "project file" used by the external builder. The issue is the potential for conflicting information in the external builder project files vs. the Eclipse/CDT project file. An example is the list of files in the project. Eclipse believes it knows what the files in the project are, and much Eclipse/CDT functionality depend upon that inmformation from the CDT "core". The external builder project file (e.g. a Qt .pro file or a Visual Studio MSBuild file) also has a list of what files are in the project. It seesm that the primary question is does the CDT builder: - Read an external builder project file and make sure that the CDT project is updated to relect the project definition from that file. - Write an external builder project file to be used by the builder and possibly be used by an external IDE that supports the external builder project file. - Both. If both, is CDT acting as an external project file "editor" and enabling use of the project file in both CDT and the external IDE? What happens in this case if different CDT configurations use different external builders. You can have 3 or more project files that somehow need to be kept synchronized, or what do you do if they are not synchronized? Most users probably want to use a particular external builder because they have existing project files. In some cases, a one-way import of an external project file may be all that is needed. E.g. if all of the project users were using Qt Creator and now they will all be using CDT. But in many (maybe most) cases, there is a reason for keeping the external project file up to date - e.g. to be used in another IDE or for building on another system.
(In reply to comment #5) Yes, this comes back to what I've talked about with a "File Managed" build. I'll provide the hooks for build systems to get the list of files for a given configuration. I'll need to work on the details on how that happens, either at build time or resource delta time, or even file creation time in the new source file wizard. Depending on that, we can decide the conflict resolution mechanism. The use case I'm building this for is for teams where the command-line is king, which is still the world we live in. The CDT is an optional add-on, not everyone uses it. So it can't be the master of the build information. And in most cases, not even the file list which can get pretty complicated in multi-target builds.
BTW, I'll be using the TCF reference agent as my test subject. I'll be using CMake for the build. It supports building for a lot of different targets, pretty much every one we support in the open, as well as commercial.
I've created Bug 322458 as a first step, allowing extenders to provide their own build implementation. BTW, yes, this does start leaning toward extending the managed build model to do more fancy things with Makefile-like projects.
(In reply to comment #7) > BTW, I'll be using the TCF reference agent as my test subject. I'll be using > CMake for the build. It supports building for a lot of different targets, > pretty much every one we support in the open, as well as commercial. Hi Doug, I was wondering how building with CMake would be integrated. I am new to CMake, but as I understand it, you need to run CMake first then it generates the Makefiles then you can always use make and it will pickup the changes from CMakeLists.txt without the need to run CMake manually again. So, I guess the only thing missing right now is running the initial CMake at project creation?
I should create a bug for the CMake support. But for now. I guess there are two parts to it. The biggest one is that initial call to CMake and knowning when to do it. Basically that means the IBuilder for CMake would look to see if one or more of the generated files are missing and then call CMake. You also want to run it when the user changes the options for it. The Makefile that CMake generates takes care of the rest. And it would be run from a second IBuilder, i.e. the standard external make builder. BTW, managing multiple IBuilders per configuration is a pretty big part of this task. The other part is managing the list of source files. This is probably a lower priority, but it would be nice to have a per config list of files based on the "exclude from build" property. At the least I would make this optional in case the user really wants to manage their own file list.
(In reply to comment #10) > BTW, managing > multiple IBuilders per configuration is a pretty big part of this task. Of course as I type that, I forget why I wanted to have the CMake IBuilder separate from the external builder. I believe the use case that triggered that was having a different IBuilder for nmake versus gnu make on Windows. CMake for Visual C++ relies on nmake where CMake for MinGW uses gnu make. Or actually it's the other way around. Using the nmake generator on Windows produces Makefiles for Visual C++ where using the Unix makefile generator on Windows produces Makefiles for MinGW. That took a bit to figure out.
But I need to remember why the toolchain couldn't dictate which external builder to call and the environment for that call.
The removal of bug 210248 resp. bug 309769 would be a big enhancement.
Calling cmake without triggering a build would be a very nice feature. It should then be possible to navigate projects using the indexer without the need to build them locally first. CMake can output whole projects structures when called with "-DCMAKE_EXPORT_COMPILE_COMMANDS=On" (See bug 350206). The output, a json file, is generated when cmake is run, the actual build is not neccessary. LangageSettings can be extracted from that json file, the indexer can use them. We usually take most of the build artefacts from our nightly integration instead of building them locally due to project size. Parsing the build output to gather language settings (like scanner info does) works well, but only after a local build. Getting all language settings for the indexer without a build would be a great improvement.
Using this bug going forward for the new build "model". The proposal is documented at https://wiki.eclipse.org/CDT/Build/Doug. I'll copy the discussion section here: Questions from Sergey: 1. Will the new build model support building multiple binary artifacts per project? Since each GTest is a separate binary, it doesn't make sense to create a separate project for every unit test. <Doug> Well, the trick is, there is really no build model. Whatever system you're using to build, CMake, qmake, ..., decide those limitations. Having said that, we need to make sure we deal with multiple binaries in other places like the launch bar. How do we decide which binary to use for a given launch. <Sergey> All *makes support building multiple binaries, don't they? <Doug> Of course they do, well not sure qmake does, but whatever. They are a reality and we need figure out how to deal with them in the launch bar. Maybe for these cases we don't and let the user set up the launch configs themselves. 2. Have you looked at http://clang.llvm.org/docs/JSONCompilationDatabase.html <Doug> Nope but I will. Questions from Marc-Andre: 1. I consider CMake, qmake, autotools, CDT Makefile generator to be "build generators" (build configurators?). Those typically need to be ran once than you execute a build command such as make, ninja, etc. Would it not make sense to have the build generator concept separate from the builders? <Doug> I thought about that. CMake provides a good example of that. The parameters you pass, like "-G Ninja", decides which build command to run next. And the build generators decide what parameters you can pass to the build command. So I think during a given invocation, they are tied together. Having said that, though, I imagine we'll want to provide a class for error parsing the build output like we do with make, or something like that. We should really walk through the use cases in more detail so we can decide. 2. "Right now the assumption is that there is only one build system per project, i.e. Qt, Arduino, CMake, etc.". That's not quite true I think. In the current model, you can have a configuration using the internal builder and another one with the GNU Make builder, and you can choose the toolchain per config too. This can be changed by the user in the project properties, C/C++ Build, Tool Chain Editor. A new project wizard could do step this up for the user if it wanted to. <Doug>Cool, more questions! Keep them coming :). I'm actually talking about my new design in the quote. Everything is driven from the Builder and I don't have a mechanism yet for a Builder to ignore a given Config, e.g stop the Qt builder from assuming it owns a given build configuration. Shouldn't be too hard but need to figure out the best and cleanest way. It's just not a priority yet until we get a single build system per project right, which is by far the most common scenario. 3. "Also, the assumption is that Toolchains are independent of the build system. Hopefully that's true. If not we'll have to introduce another adapter layer to adapt Toolchains to build system specific Toolchains.". CMake has toolchain files for defining toolchains, so I see a need for generating toolchains dynamically. I think there are also other examples out there. I remember reading about Visual Studio 2010 having toolchain files as well. <Doug>Well, you can't generate the scanner info algorithm or error parsers dynamically. You need to know at least what mechanisms are for the given toolchain. You can pick the toolchain dynamically, and you can alter the toolchain configuration, like PATH, dynamically. Everything is programatic and in control of the build configuration. So the CMake build configuration can take a look at the toolchain files and set things up properly. 4. Just a general comment on the page. It's not clear to me what the shortcomings of the current build system are. I think it would be good to identify the shortcomings then specify in what way the new system will solve them. From my perspective, I find it hard to wrap my mind around current system, so I'd like to see its complexity reduced. <Doug>Well the good news is that I'll have a first cut of the build system in place in a week or so and it'll have build configurations for Arduino and Qt projects. It will be better to judge once you see the code. My main object is simplicity and the ability to totally control the build and it's settings while offering a least a little framework to hook things together. And no more dealing with the crazy IProjectDescriptor and friends and the ridiculously complicated managed build model when all you want is to store where qmake is located for a given config, or what board I'm building for with Arduino. Question from Walter: Any thought spent on parallel builds? I.e., e.g. the scanner discovery currently does not really support parallel builds over multiple directories at the same time... <Doug> Build output parsing relies on the build tool to merge the output from the multiple threads in such a way it keeps the integrity of the command lines being spit out. Not sure how you resolve that other than fix the build tool. We are always looking for ways to figure out the includes and macros used by a build that don't rely on build output parsing. Some build tools, like recent CMake's, offer that. I think the builder needs to be more involved in scanner discovery to leverage that. I'll add that to the proposal.
New Gerrit change created: https://git.eclipse.org/r/54924 WARNING: this patchset contains 1678 new lines of code and may require a Contribution Questionnaire (CQ) if the author is not a committer on the project. Please see:https://wiki.eclipse.org/Project_Management_Infrastructure/Creating_A_Contribution_Questionnaire
Gerrit change https://git.eclipse.org/r/54924 was merged to [master]. Commit: http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=64c57825896980a473ebc362f5a32d806c2c85e4
Some thoughts on toolchains thinking of the Qt scenario. With Qt, I can get the XSPEC, things like linux-g++, macx-clang, qnx-armle-v7-qcc for a given qmake. In the past, we've found it convenient to think of a machines spec giving it's OS and ARCH. I'm pretty sure we can figure that out for each XSPEC. If XSPEC == SPEC, then we're compiling for host where we can use things like uname. We can also think of a compiler family for compilers. My initial list would include: gcc, clang, qcc (QNX), vc++ (MS), maybe the Wind River compiler. Each one get's a subclass of CToolChain. There would be one instance of those per toolchain installed on the user's system that we know about. The QtBuildConfiguration points to a given qmake which has a given XSPEC. We should be able to map that to Family, OS, Arch. Then we need toolchains to provide the same mapping so we can pick which one to use for the build. And really what we mostly need out of a toolchain at this point is the environment, especially PATH, so we can run the tools. I would provide an autodiscovery mechanism for toolchains that look at the path, on Windows, the registry, the XCode directory, what have you. You should be able to determine the Family, OS, Arch, by running -v on the toolchain (at least you can for gcc), or using some other heuristics. This would be an extension so toolchain providers can add in their own. Would also provide a mechanism for the user to add toolchains using preferences. Each family could have it's own specialization of the dialog to enter values it needs. Mostly you would need the directory for the PATH variable. For Qt, we'd provide a preference page for users to add Qt installs. They could also use that to map the XSPEC to a toolchain if there is more than one that matches the Family, OS, Arch spec. If the user tries to do a build that doesn't have a toolchain, an error message should show up on the build console. Toolchains are per user, not shared. So are the instances of qmake. The only thing saved in the configuration is the XSPEC. User preferences and auto discovery fill in the rest. Lots of magic. Thoughts? Am I nuts?
(In reply to Doug Schaefer from comment #18) > Thoughts? Am I nuts? You're not nuts! This all makes sense (to me at least). I think it's good that you are thinking ahead of what to put in the configuration and what to keep out, like PATH for toolchains etc. This is one thing that we discussed about CMake support: I think for sure there's needs to be a mapping of compiler families to their actually location. In the case of CMake, a user could create a custom toolchain with a given toolchain file on their file system but set it in the configuration using just an id. The mapping from the id to the location of the toolchain file would be done in preferences, I think similarly to the XSPEC to toolchain mapping you are referring to for Qt. I am hoping that by doing it this way, CDT projects will be easy to share.
Cool. Thanks! CMake is going to be the next big test. We should start thinking of it now. For qmake, my plan as I start to implement it is to store the XSPEC in the build configuration. That is all that is shared. On a user's system, a qmake is registered for each XSPEC they have installed. Using an extension point we map the XSPEC to toolchain Family, OS, and Arch. ToolChains are registered on the user's system and have Family, OS, and Arch. qmakes are mapped to toolchains using that. If there is more than one mapping, the setting in qmake lets you pick. When I say registered, it could be discovered automatically, or the user can manually add. Discovery mechanisms can be added with an extension. An obvious one would be to discover toolchains on the PATH. This follows through to the Launch Bar. The target remote connection gives us the OS and Arch. You can find the Toolchains and qmakes that support the target and auto select them. For CMake, we may need a way to register toolchain files. We may be able to look at them and guess the family, os, arch, or we let the user tell us. I don't think CMake has an XSPEC like system so we may need to store family, os, arch in the build config. Mind you, we may find it easier for qmake to do that too instead of the XSPEC. Anyway, we'll have to discuss this at the summit. I'm probably missing some details. One thing I haven't thought about at all is versioning of toolchains, and in Qt's case, versioning of Qt (there is one qmake per version per xspec).
Or, better yet, can we generate the CMake toolchain file for a given toolchain? That starts to bring in the idea that we need to adapt a toolchain for a given build system for build system specific functionality like that if we can generalize it for all toolchains.
(In reply to Doug Schaefer from comment #21) > Or, better yet, can we generate the CMake toolchain file for a given > toolchain? > > That starts to bring in the idea that we need to adapt a toolchain for a > given build system for build system specific functionality like that if we > can generalize it for all toolchains. It is possible, Yocto plugin does it. But it does not solve all problems: - to generate a toolchain file, CDT needs to know details about the toolchain in advance (compiler binary, CFLAGS, sysroot, etc) that are otherwise defined in the toolchain file if one is provided. - In both cases, these details about the toolchain are available in the json file produced by cmake after the cmake run, before the build (needed to get the scanner info without doing the build). - We would need an interchange format to import / export toolchain settings from one CDT installation to another, which would double information in the toolchain file. + Unfortunately, many (cross-) compilers do not include a cmake toolchain file, so users may have to write them themselfes.
In short: The cmake toolchain file is a nice way for toolchain integrators to spread settings for a toolchain to a large team. CDT can retrieve all it needs from there. But what about the users that neither have a toolchain integrator, nor a toolchain file? Should be configurable, I think.
Cool, thanks, good points. We may need all that information for other build systems anyway and the design doesn't preclude that. And the user will be able to create their own toolchain definitions, just as they can for cmake. But we'll see where things go. BTW, on the scanner info front, a default method we could use is to call make with -B -n to do a full dry-run build to give us all the commands without running them. Not sure why we didn't do that before. Then we could get rid of the scanner discovery builder.
One of the main use cases that the Arduino build system does well is mapping from launch targets, i.e. remote connections listed in the Launch Bar, to build configurations. It's easy there since each board type has one toolchain and the target has the board type. In general, this is the desired workflow, select launch config, mode and target, and the right thing gets built for the launch, as buildForLaunch is supposed to, when the user clicks Build or Launch. Things are less obvious with Qt and probably brings us closer to a general workflow. Here's my current thinking on how it would work. - IRemoteConnections have properties for OS and Arch. We should always be able to determine the values for those. - Toolchains also have properties for OS and Arch. All toolchains target a given combination of those. For GNU compilers, we should be able to derive that from the target name given with the -v option. For example x86_64-apple-darwin14.5.0 means OS = macosx, Arch = x86_64. - Qt installs target a given toolchain family (e.g. gcc versus visual c++), OS, Arch. Given the list of toolchains that match the target, we should be able to gather the list of Qt installs that support those toolchains. Qt installs also have an attribute that allows the user to select a preferred toolchain. - Qt installs have a version and an XSPEC. That is what is saved as build configuration properties. In theory they should be portable to different user's systems who have different Qt and Toolchain installs. This introduces the concept of a toolchain manager to manage the list of toolchains and a UI to show the currently installed (either manually or discovered) and allow users to add more. Comments as always appreciated.
My primary concern is that with the new build system CConfigurationData.setSourceEntries(ICSourceEntry[]) or its equivalent works regardless of whether the project has ManagedCProjectNature.MNG_NATURE_ID nature or not. Currently it works only for projects with the managed build nature.
(In reply to Sergey Prigogin from comment #26) > My primary concern is that with the new build system > CConfigurationData.setSourceEntries(ICSourceEntry[]) or its equivalent works > regardless of whether the project has ManagedCProjectNature.MNG_NATURE_ID > nature or not. Currently it works only for projects with the managed build > nature. Hard to tell what it's used for without any documentation in the ICSourceEntry interface (let alone anywhere else in that code). I assume you're talking about configuration specific lists of sources to include in a build/index? Yes that needs to be a requirement for all build systems. I'm finding a lot of what's new was already there just in a slightly different form. We already had ISourceEntry included in the IPathEntry family. We could have the path entry routines on ICProject take an IBuildConfiguration as a parameter to return config specific settings. I imagine at some point we would want a mapping from ICConfigurations to IBuildConfiguration anyway. Or are you tied heavily to the CConfigurationData interfaces?
Found some old notes and reminded me of qmake -E which does a nice dump of a number of useful variables we can use to construct the compiler command line for scanner info.
(In reply to Doug Schaefer from comment #27) > Hard to tell what it's used for without any documentation in the > ICSourceEntry interface (let alone anywhere else in that code). I assume > you're talking about configuration specific lists of sources to include in a > build/index? Yes that needs to be a requirement for all build systems. ISourceEntry's determine what source files are being indexed. This is the only aspect that I'm concerned with. > Or are you tied heavily to the CConfigurationData interfaces? Not at all. I don't care where the data on the source files' build inclusion is stored, as long as there is a programmatic way to control it.
(In reply to Sergey Prigogin from comment #29) > (In reply to Doug Schaefer from comment #27) > > Hard to tell what it's used for without any documentation in the > > ICSourceEntry interface (let alone anywhere else in that code). I assume > > you're talking about configuration specific lists of sources to include in a > > build/index? Yes that needs to be a requirement for all build systems. > > ISourceEntry's determine what source files are being indexed. This is the > only aspect that I'm concerned with. > > > Or are you tied heavily to the CConfigurationData interfaces? > > Not at all. I don't care where the data on the source files' build inclusion > is stored, as long as there is a programmatic way to control it. Cool. Then, yes, I want to turn back the clock to the original IPathEntry and add the ability to add/remove entries per IBuildConfiguration without introducing a whole layer as was done with ICPathEntry. All these interfaces should be noimplement/noextend so adding things that ICPathEntry added should be doable in IPathEntry. We can use the CDT 9 major release to add those in.
This bug was assigned and targeted at a now released milestone. As that milestone has now passed, the milestone field has been cleared. If this bug has been fixed, please set the milestone to the version it was fixed in and marked the bug as resolved.
