Re: [cdt-dev] weird scanner discovery behaviour

[Chris Recoskie <recoskie@xxxxxxxxxx>]

>  - The language setting entries are used by managed build for
> storing it's paths and preproc symbols.  If you use this method on
> managed projects you'll be changing the set of includes / macros
> that are passed to the compiler, won't you?

Hmmm.... well they are stored in the build model IOption elements as well. I think only the IOption data is actually passed to the compiler. I will make sure I test that scenario. For now I have committed it for XLC. To me, the benefits far outweighed the outstanding issues, as it's not acceptable for us to have the user close and reopen the project to get the settings to show up. I will hold off on doing anything for the GNU tools then and leave that for whatever work is getting done for CDT 7.0

> - There's no way to distinguish between paths you added and paths
> contributed from elsewhere, be it user UI or ManagedBuild. I notice
> in your code you only every add entries.  This means that, if you
> change compiler, or change an include directory on a file, you'll
> end up retaining the older entries

Yep... it sucks but I couldn't find a way around it.

> - Your code uses
> ICConfigurationDescription#getLanguageSettingForFile(resource,
> false).  The false means that it returns a the parent resource
> configuration's language settings if the file specified by path
> doesn't have any of its own. Isn't it the case that, if there aren't
> any per file settings to start with, the settings are only ever set
> at the project level / source folder level?  Should your not be
> creating resource configs where they don't exist?

Ideally yes. However, my brain exploded when I tried to figure out how to create new configs. And it didn't seem that any of the discovered info ever gets put on the resource configs anyway (i.e. before these changes to directly modify the description), so I didn't get too worried about. I am probably going to look at fixing that.

> - This isn't scalable. All the settings are stored in the giant XML-
> file-of-doom. Even medium sized projects here struggle with a
> generous 768m heap. The XML processing just kills performance :( 
> There are tricks to get around this, but it's not pretty.

That's a general issue though with the project description. I haven't done anything to make that worse :-P

> I suspect it's deliberate that you can't set the read-only attribute
> on the setting entries. Read-only was used for scanner contributed
> settings that don't form part of the arguments passed to the managed
> build tools, and that aren't persisted in the .cproject.
>

Hmm... well I think it should be allowed at the API level. The implementation shouldn't be second guessing what the caller requests.

> There are so many APIs for contributing paths: PathEntries,
> ScannerDiscovery, ManagedBuild, externalSettings providers. They all
> plug in in non-obvious ways to the model. With the new project
> model, someone with a project in the unified Makefile/Managed world,
> will have all their custom settings and externalSettings stored in
> the .cproject.  These will contribute to the set of langEntries
> passed to the Toolchain.  Paths contributed via scanner discovery
> contribute to cdt.core but aren't persisted by the model, and aren't
> passed to the toolchain.
>
> I think this is why we really need a unified path store in cdt.core
> which handles all issues of persistence and loading, and allows ISVs
> to add & remove paths under their own unique key. Users can then
> control which path contributors they want enabled.

I agree with all of this. PathEntries are evil and confusing to start with. The way everything plugs into them and into eachother is a byzantine maze of tangled spaghetti strands. What you propose sounds to provide some needed simplicity and flexibility.

===========================
Chris Recoskie
Team Lead, IBM CDT and RDT
IBM Toronto
James Blackburn <jamesblackburn@xxxxxxxxx>

James Blackburn <jamesblackburn@xxxxxxxxx>
Sent by: cdt-dev-bounces@xxxxxxxxxxx

11/26/2009 05:38 AM

Please respond to
"CDT General developers list." <cdt-dev@xxxxxxxxxxx>

To	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
cc
Subject	Re: [cdt-dev] weird scanner discovery behaviour

Hi Chris,

I do something *very* similar for my scanner discovery alternative -- dwarf settings parser.  There are some gotchas though which make this not suitable as a general fix to push into CVS:

 - The language setting entries are used by managed build for storing it's paths and preproc symbols.  If you use this method on managed projects you'll be changing the set of includes / macros that are passed to the compiler, won't you?
- There's no way to distinguish between paths you added and paths contributed from elsewhere, be it user UI or ManagedBuild. I notice in your code you only every add entries.  This means that, if you change compiler, or change an include directory on a file, you'll end up retaining the older entries
- Your code uses ICConfigurationDescription#getLanguageSettingForFile(resource, false).  The false means that it returns a the parent resource configuration's language settings if the file specified by path doesn't have any of its own. Isn't it the case that, if there aren't any per file settings to start with, the settings are only ever set at the project level / source folder level?  Should your not be creating resource configs where they don't exist?
- This isn't scalable. All the settings are stored in the giant XML-file-of-doom. Even medium sized projects here struggle with a generous 768m heap. The XML processing just kills performance :(  There are tricks to get around this, but it's not pretty.

I suspect it's deliberate that you can't set the read-only attribute on the setting entries. Read-only was used for scanner contributed settings that don't form part of the arguments passed to the managed build tools, and that aren't persisted in the .cproject.

There are so many APIs for contributing paths: PathEntries, ScannerDiscovery, ManagedBuild, externalSettings providers. They all plug in in non-obvious ways to the model. With the new project model, someone with a project in the unified Makefile/Managed world, will have all their custom settings and externalSettings stored in the .cproject.  These will contribute to the set of langEntries passed to the Toolchain.  Paths contributed via scanner discovery contribute to cdt.core but aren't persisted by the model, and aren't passed to the toolchain.

I think this is why we really need a unified path store in cdt.core which handles all issues of persistence and loading, and allows ISVs to add & remove paths under their own unique key. Users can then control which path contributors they want enabled.

Cheers,
James

2009/11/25 Chris Recoskie <recoskie@xxxxxxxxxx>

Basically, I do not rely on the scanner discovery infrastructure at all to update the project description, nor do I try to get the infrastructure to automagically try to update the cache by reloading the project description. Rather, in my job that already serializes the scanner info to the .sc file (via the DiscoveredPathManager), I added some code to directly take the newly discovered scanner info, and directly write it to the ICLanguageSettingEntries myself, then set the project description to get the new data to take.

The only weird behaviour I have remaining is that when I create the settings, I specify that the discovered info is built-in and read only, which is what the project level settings were already populated as. This is supposed to ensure that they do not show up unless you specify in the UI to show the built-in info, and you cannot delete or modify them. Despite setting those flags when creating the settings from the user's build output, when the data is read later from the project description, the flags are blank, so the discovered info shows up in the UI as user-defined info, and is deletable/modifiable. I would still like to fix that, but I can live with that for now. It is far better to have data the user can accidentally delete (and will get recreated when they build again), as opposed to no data at all.

Note that one thing that also seems to be missing from DiscoveredPathManager is the ability to handle changes to include files (i.e. explictitly included via command line options) and macro files, as opposed to the include paths and macro definitions, which it does handle. Hence, my code below doesn't handle include/macro files, because this part of discovery system is not tracking that data anyway. Since overall the scanner info system is supposed to handle that type of data, it would be nice at some point when this part of scanner discovery is overhauled to make sure that those features are supported. The build output parser already supports it, but there's no way to do anything with the data given the current infrastructure. It is rather weird that the lowest software layer supports it, as does the top-most layer, but the middle layer doesn't.

Here is a snippet of some of the code. When I get more time in a week or two I will have a look at doing this for the GNU scanner discovery if required. I suspect it has the same issues but I have not tested for sure that that is the case._______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev