Below
are a few details on my thoughts on how to proceed. I can spend additional
time adding more details to the proposal if there is someone(s) who is
interested in implementing it. I would like to have the time to implement it myself,
but that seems unlikely.
Thanks,
Leo
CDT Core
Build Basic Architecture
The focus of this design is to provide an easy mechanism for
integrating a tool-chain into the core CDT functionality. The intention is to
organize the core tool-chain metadata in one place and provide the “first-step”
for any tool-chain integration. It also defines the relationships between a
tool-chain and other core CDT objects with a particular emphasis on builders.
Some builders require more information than others and the design will support
the ability to define and use builder specific metadata. For a tool-chain
integrator, providing builder specific metadata is an optional task. A
relevant example is the existing Managed Build System (MBS) extension point.
MBS will define and support the metadata required for make file/build file
generation in CDT and the use of the internal builder. No tool-chain is
required to provide that metadata. However if a tool-chain provider wants to
support that builder for his users, he will need to provide the MBS specific
metadata. Other builders can define their own specific metadata that a
tool-chain must provide in order to be used with the builder.
The design uses much existing functionality from CDT. It
also suggests modifying, deprecating, or even removing some existing extension
points. Two primary targets are:
·
Scanner Discovery: This component of CDT does
not interact well with other components of the core. It also does not support
setting up the environment for external commands that can be invoked. The
current functionality enabled by scanner discovery, plus other ways to provide
the same information, will be consolidate in the new core build support. A
recommendation of deprecation or removal is likely.
·
Managed Build System: MBS is a mixture of
tool-chain core metadata and metadata specific to make file/build file
generation and the internal builder. This metadata needs to be split into the
core metadata as addressed by this design and metadata specific to supporting
make file/build file generation and the internal builder. MBS does not play well
with scanner discovery. A recommendation to evolve the MBS schema by removing
the core metadata information, and working with the new core build metadata, is
likely.
The basic elements of the CDT core build system include:
·
Configuration
·
Tool-chain
·
Programming Library / SDK (is the existing
External Settings Provider extension point sufficient?)
·
Builder
·
Error parsers
·
Build console
The tool-chain also interacts with other basic elements in
CDT including:
·
New project creation (AppWizards)
·
Debuggers
·
Platforms
·
Language-specific parsing and editing
functionality
The tool-chain metadata would be defined in a new CDT core
extension point that defines a schema. In these ways it will be similar to the
current MBS schema:
·
As much information as is possible can be specified
declaratively
·
When dynamic behavior is required (e.g. proving
the environment settings for using the tool-chain on a specific user’s system),
object attributes will be provided that expect a tool integrator to implement a
particular interface
In these ways, it will be very different than the current
MBS schema:
·
It will contain “core build” metadata only
·
It will be significantly smaller and less
complex
·
It will be read-only metadata – that is, the
same schema will not be used to store information about the use of the
tool-chain in a particular project configuration
·
There will be no concept of
“inheritance/superclass”
In addition, there should be public APIs for providing the
metadata information through code.
Core Build Metadata
Tool-chain
The tool-chain element will provide the following metadata:
·
Information describing the tool-chain to the
user – name, supported host platforms, supported target platforms, …
·
Information for populating the CDT new project
AppWizard framework. This will enable the same support as the current MBS and
core functionality do.
·
Information for setting up the build environment
of the process created to run a build
·
Information needed by the CDT editing/code
assistance/refactoring functionality – e.g. the paths and macro definitions
currently provided by scanner discovery and other mechanisms. Note that a
builder (e.g. MBS) may also be involved in providing some of this information.
The tool-chain will be the CDT focal point for this information and may need to
use builder specific mechanisms for gathering information from a builder that
the tool-chain supports.
·
The builder(s) that the tool-chain works with.
There will be a “core builder“ “flavor” that can be used with any tool-chain
that provides the core metadata. Support for additional builders (which
require builder specific metadata) can be specified.
·
The debugger(s) that the tool-chain is known to
work with
·
The libraries that the tool-chain is known to
work with
·
The default error parsers for the tools in the
tool-chain
·
The build console(s) that the tool-chain is
known to work with? I don’t know whether build consoles are associated with a
tool-chain or a builder or are simply a user preference setting.
The “known to work with” metadata will not provide an
exclusive list of other elements that the tool-chain will work with. Users may
try other combinations at their own risk.
Builder
The builder element will provide the following metadata:
·
Information describing the tool-chain to the
user – name, supported host platforms, …
·
The default error parser(s) for the builder
·
Information for setting up the build environment
of the process created for the build
·
Information for creating the command line for a
build
·
Whether it requires additional metadata, and if
so, the name of the extension point
From:
cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf
Of Doug Schaefer
Sent: Monday, April 26, 2010 11:32 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] Build Again
You are definitely not in disagreement with the current
architecture. The actually mechanism of providing build information to the
indexer is IScannerInfoProvider. We used to populate that differently if you
were a standard build project, which used the build output parsing (or profile?
I never did understand what a scanner discovery profile was), versus managed
build, which used the build model.
That has been changed with the new project model work a
couple of years ago so now the scanner discovery populates the build model, or
at least the project description part of it, and there is only one scanner info
provider today.
This is part of what I'd like to fix. The extensibility
needs to be revisited. With external managed builders, this information can be
derived without having to parse the build output like parsing other files (e.g.
qmake files). We should allow for that.
On Mon, Apr 26, 2010 at 1:19 PM, <Ed.Swartz@xxxxxxxxx> wrote:
Hi,
Regarding indexer setup and scanner discovery, I'm just tossing
this out there, in case it hasn't been considered yet. I'm not even sure
how it works now, so ignore me if necessary :)
Is scanner discovery an optional component in the ideal
CDT build system of the future? IMHO, it would be best if the
concept of scanner discovery were one option that existed alongside a
completely build-driven indexer configuration mechanism. These could both
implement a concept called "indexer settings discovery" -- one model
scans the build output, another model is driven by the builder itself.
For instance, in Symbian OS, we query the build system
directly (independently of the project) for a majority of the indexer
configuration, and then we parse the build scripts to get the rest of the
information. Thus, in such projects, there is no need for the user to do
anything, or for a build to have taken place at all, since the
indexer configuration can be discovered from well-known
information.
In other models like autotools, there are well-known ways to get a
majority of the data too, by parsing configure.ac and Makefile.am. Though I guess the user
may still want to intervene if he's using per-file configurations or some other
magic that's difficult to discover automatically.
I'm only bringing this up because I see in these recent posts no
mention about non-scanner-discovery models. It would be nice if the
architecture were able to model both of these use cases, or some combination
thereof, with the same model. (If it's already done like this,
then great ;)
Right now, a plugin can configure the indexer settings if it wants,
of course. But having a unified model for this in CDT would also
allow for updating indexer settings in a unified way, e.g., in response to resource
change deltas in a single workspace job. (I.e. if my build script
changes, I can update the indexer settings.) Currently I think several
clients inside and outside CDT may listen to the same resource changes and
respond separately, exposing ordering issues and possibly incurring too much
overhead in triggering reindex jobs.
Thoughts?
-- Ed
From: Doug Schaefer
[mailto:cdtdoug@xxxxxxxxx]
Sent: Monday, April 26, 2010 10:14 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] Build Again
On Mon, Apr 26, 2010 at 12:37 AM, Andrew Gvozdev <angvoz.dev@xxxxxxxxx>
wrote:
Hi Doug,
I think currently standard make project is hurt being an instance of an
overweight MBS project. MBS took over everything related to build including
project model squashing any competitor build system with impenetrable
complexity. If we could make MBS a build system among others it would be a good
thing. MBS is very useful and working well for the most part, but those parts
that are not working properly, such as scanner discovery, near impossible to
fix.
A while ago I wrote a prototype for new scanner discovery (bug
290631). For all the talk I never got much feedback on that. I planned to
continue that work after Helios release. Perhaps I can give you a hand with the
new build system on scanner discovery along these ideas? And if it proves
itself we could back-port it back to MBS.
Sorry I missed this Andrew, I will definitely take a look. I
want to make sure scanner discovery, which is critical for Makefile projects,
is easily extensible to other tool chains, cross gcc compilers in particular
and MSVC probably to ensure we're not too gcc specific. I like the fundamentals
of scanner discovery, i.e. build output parsing and collecting like compiler
commands to ensure we can have scalable per file discovery. But we need make it
easier to extend.
BTW, perhaps you could create a git repository for the new-build plugin rather
than CVS - to try it out before potential move of CDT ones?
That's a good idea. I need to keep track of egit and using for
something real will help. I have an account on github and will set up something
there.
My strategy is to have a build.core and build.ui plug-in
where we can fork what we need so that we don't disturb existing stuff and to
help break the shackles. Scanner discovery should be one of those things. It'll
also have a new New Project wizard to set up the project for the new build
system.
Andrew
On Sat, Apr 24, 2010 at 4:26 PM, Doug Schaefer <cdtdoug@xxxxxxxxx>
wrote:
Thanks guys. I'll definitely use your input.
I will disagree with James statement that "it works
today". I never been able to get scanner discovery to work for a new
toolchain. I know Chris R has had a similar problem. If it did work today I
wouldn't be so frustrated by it. And I think the problem is architectural, not
code quality. Standard make projects used to be very simple. Yes, we needed
them to do more, but they still should be simple.
My focus will be on making it easier to get projects that
use Makefiles and/or external managed build systems like configure, qmake, and
cmake up and running including automatically setting up build environments and
scanner discovery. I certainly won't be replacing CDT's managed build in the short
term. It works well enough, and I only have so many hours to spend on this. And
given the pervasiveness and quality of Make and the external managed build
systems, I'm not even sure it's needed any more.
:D
On Sat, Apr 24, 2010 at 2:02 PM, Treggiari, Leo <leo.treggiari@xxxxxxxxx>
wrote:
Putting together a fully customizable managed build system
was complex and I guess it shows.
See http://www.ddj.com/dept/opensource/197002115
for an exaqmple of its flexibility.
That said, I would suggest that the first step is this. Cleanly separate
the "create a makefile or other build system file" functionality of
the MBS from the basic build functionality. My belief is that all build
systems added to CDT can gain by much of the functionality at the TOP of the
MBS schema - objects like Projects, Configurations, ToolChains. The
ability to specify information such as how to set up the build environment for
a toolchain or provide the information needed by the editing functionality of
CDT is invaluable to a multi-toolchain environment such as CDT - even Visual
Studio finally provides the ability to set up a tool-chain specific environment
in VS 2010. And, then resolve how Scanner discovery interacts with this
basic tool-chain integration functionality - that clearly was never done
properly.
The information from the TOP of the MBS schema could be provided in a new
schema and the MBS schema evolved to the pieces required for generating a build
file or implementing an internal builder. Or maybe some clean split can
be accomplished with the current schema.
Thanks for listening,
Leo
-----Original Message-----
From: cdt-dev-bounces@xxxxxxxxxxx
[mailto:cdt-dev-bounces@xxxxxxxxxxx]
On Behalf Of James Blackburn
Sent: Saturday, April 24, 2010 9:30 AM
To: CDT General developers list.
Subject: Re: [cdt-dev] Build Again
Hi Doug,
You're right: the CDT build-system is tough, the code quality leaves
something to be desired, and the way it's been written makes unit
testing hard, so instead there are brittle end-to-end tests.
That said it works today.
I spent last week fighting to make improvements for a team using
ManagedBuild in anger. One project set has 44 (yes really 44!)
inter-dependent projects. Most are libraries which are brought
together using references to build the top-level app. It works
remarkably well but a there are a few problems: bug 309769 and
friends...
I worry that years have been put into the current build system, and
that it would be impossible with current resourcing to recreate the
functionality. And if we were to re-create it, would we just end up
with the same issues and same problems as before? Taking debug as a
topical example: there are now 3 debug engines, each with pros and
cons and each with committers addressing similar issues. As a user I'm
not really sure why I'd choose one over the other. While it's good
there's choice, in my mind choice creates confusion, and wastes
effort.
The wiki makes some great points. One thing it doesn't mention is how
to go about solving the complexity of the build model. Currently it's
hugely verbose (in terms of lines of code, size of a .cproject, size
of the build system schema) and a huge PIA to maintain. I can't help
but feeling that EMF might make this all easier -- if only I knew
anything about it.
Rather than starting from scratch, would it be better to start from
where we are now, work out where the pain-points are and push for
improvements in these areas? That way we make what we have better
rather than taking the risk of starting from scratch and not having
enough momentum to carry it through.
At the moment I'm pretty invested in the MBS. We have two teams using
it for their projects, and 1 team using Make Targets and standard
make. Unless the project leads give up on it, I can't see myself
getting much time to work on something completely new.
Cheers,
James
On 24 April 2010 16:05, Doug Schaefer <cdtdoug@xxxxxxxxx> wrote:
> Hey gang, FYI,
> As I mentioned on the last call, I've been asked to play a more active
role
> in TCF and help build a community for it and work with you to bring it to
> maturity. That's taking most of my time these days as I get up to speed
and
> figure out what we need to do with it.
> In my spare time, I'm working on bringing the CDT to the Android community
> to help build native libraries for that platform. The focus is on one of
my
> dream projects, building a game engine out of open source parts, in this
> case for Android, but I have eyes on iPhone and MeeGo too. Part of doing
> that is getting the Android gnu toolchain working well with CDT. And
that's
> brought me back into the CDT build nightmare. Once again, I am having a
hell
> of a time getting scanner discovery working for these seemingly simple
> Makefile projects. Just look at a stack trace sometime and you'll see,
> today, the CDT build system is the opposite of simple.
> So I am urging myself to get back into working on a new build system. We
> need to put the architecture back to the way it was, with managed build
> being an add-on, not having everything driven by managed build. Obviously,
> I'm not going to have much time to spend on it, so by definition, I'll
have
> to keep it simple. And I'll have to do it as a separate stack, not as an
> evolution of what we have.
> Over the next few days, I'll jot down some more ideas
> here http://wiki.eclipse.org/CDT/NextGenBuild.
And I'll recreate cdt.build.*
> plug-ins, this time in a separate folder to avoid confusion. And as always
> help is appreciated and hopefully, I'll have new found vested interest in
> bringing it to completion, especially since I want to get back working on
> the Android project.
> Cheers,
> Doug
> _______________________________________________
> cdt-dev mailing list
> cdt-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cdt-dev
>
>
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
