[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[cdt-dev] Parallelization of indexer
|
Hi Markus.
Yes, this discussion has been somewhat frustrating and I still do not agree with your comments (and the comments of others about that topic) for several reasons.
1) Everyone involved in this discussion seems to believe, that it is possible to setup one monolithic Eclipse project for a large-scale and real-life C++ project, and that this is the "normal" use-case for CDT, and that parallelization of indexer jobs across Eclipse projects therefore doesn't help much. I already questioned that opinion in b#351659, because I cannot see, how you would setup one monolithic Eclipse project, if the sources require e.g. different sets of #define's or different include pathes (and source files in real-life projects normally do require this). You won't get correct indexer results in that case. But maybe, I missed some point...
2) In fact, my patch does not honour project references, but I explicitly asked, if that would be a real problem, and your reply was, that the only impact is potential replication of some of the symbols in multiple project indices. Therefore, my understanding was that there is no "real" issue, if parallelized indexer does not take project references into account.
3) I cannot see, how parallelization on file level instead of project level can resolve issue 2). If indexer job#1 indexes a file from project A and indexer job#2 indexes a file from project B, then you are back at the same point... Should these jobs honour references between project A and B? I cannot see any difference.
4) Only solution to problem 3) would be to limit parallel indexer jobs (on file level) to the set of files of one and the same project. In that case, there will be at least three other issues... First, when there are many projects with only a few files, parallelization will be poor. Second, at the end of the process of indexing each project, parallization will be poor, because there are more potential free CPU cores than there are files left to index in that single project. Third (and most important) all these parallel jobs on the files of one single project will run into lock contention on the project's index write lock. I already faced a similar issue with my patch and had to change some of the locking code to achieve enough throuput/CPU utilization with my approach.
Therefore, from my point of view...
1) Discussion about paralellization across projects vs. parallelization across source files is independent of the question of honouring project dependencies. Both approaches need either a fix for the performance issues mentioned in b#351659 or a decission not to honour project dependencies while indexing.
2) And yes, of course I could open another bugzilla about that performance issue, but what would that help? I already mentioned that issue, I captured jprof profiling information and attached the profiler data to b#351659, and I asked for someone to look into that data. Nothing happend. Why should that change, if I opened a second bugzilla and attached the same profiler data again?
Kind regards.
Volker
-----Ursprüngliche Nachricht-----
Von: cdt-dev-request@xxxxxxxxxxx
Gesendet: Nov 28, 2011 6:00:06 PM
An: cdt-dev@xxxxxxxxxxx
Betreff: cdt-dev Digest, Vol 81, Issue 32
Send cdt-dev mailing list submissions to
cdt-dev@xxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
https://dev.eclipse.org/mailman/listinfo/cdt-dev
or, via email, send a message with subject or body 'help' to
cdt-dev-request@xxxxxxxxxxx
You can reach the person managing the list at
cdt-dev-owner@xxxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of cdt-dev digest..."
Today's Topics:
1. Re: Parallelization of indexer (Schorn, Markus)
2. Adding keybindings to dynamic menu items (Phil (Philip) Mason)
3. Re: Adding keybindings to dynamic menu items (Phil (Philip) Mason)
----------------------------------------------------------------------
Message: 1
Date: Mon, 28 Nov 2011 06:40:34 +0000
From: "Schorn, Markus" <Markus.Schorn@xxxxxxxxxxxxx>
To: "CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Subject: Re: [cdt-dev] Parallelization of indexer
Message-ID:
<30D36C1BA62C5F4892C482E607D5E77E1FA57921@xxxxxxxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset="us-ascii"
Hi Volker!
I can understand your frustration, however there is an issue with the patch as provided in bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=351659. My view on the matter is the following:
(1) Your patch does not deal with indexing dependent projects. Currently the index of a project is reused by a dependent project. This requires the dependent project to be indexed after its dependencies. Your patch ignores this requirement.
You have identified that indexing with dependencies does introduce a performance issue. This needs further investigation and may lead us to changing the indexer, such that it ignores the project references. However, before we have made such a decision, your patch cannot be applied.
(2) The approach of parallelizing indexing on project level does not help for large projects. I do agree with Sergey, that it would be more rewarding to make parallelization work on file-level.
To move forward on the issue, I encourage you to open a new bug on the performance issue of dependent projects. We need a discussion on that and only if we drop the requirement of reusing the index of a dependent project we can go back to consider your patch.
In parallel it makes sense to think about parallelization of file-level. Because thinking long-term, this is the more promising approach the approach would find more traction.
Markus.
-----Original Message-----
From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Volker Diesel
Sent: Friday, November 25, 2011 23:34
To: CDT Dev
Subject: [cdt-dev] (no subject)
Hello, everybody.
There used to be some discussion about C/C++ indexer parallelization some months ago and (initially) most people agreed, that this would be a great feature.
There is a patch in place, that brings down full C/C++ indexing time from 4hrs to 20mins in our project (see bug#351659).
This patch has now been used in our team (200+ people, 10+Mio lines of C/C++ code) without any issue for several months.
I provided a git patch for CDT master.
I provided a git patch for CDT 8.
I have not received any answer to my latest questions in the above mentioned bugzilla since months.
I wonder, if anyone out there in CDT DEV is still interested in that topic?
I wonder, how such an enhancement will finally find its way to any CDT codeline and what I can else do to bring this feature into official CDT release?
If noone at CDT DEV is any longer interrested in that topic, please let me know. In that case I would simply close that useless bugzilla.
Thanks and kind regards.
Volker
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
------------------------------
Message: 2
Date: Mon, 28 Nov 2011 05:15:55 -0800
From: "Phil (Philip) Mason" <phil.mason@xxxxxxxxxxxx>
To: "CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Subject: [cdt-dev] Adding keybindings to dynamic menu items
Message-ID:
<C28713F579DB094D828AF937C214266A18D8CC4DB8@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset="us-ascii"
Hello all,
I've been trying to add keyboard shortcuts to the number format entries in the menu of the variable view and I'm struggling. The menu is a dynamic collection of Actions and I'm beginning to wonder if the problem is that the menu entries are not created until they are displayed so the any key bindings don't exist until then either.
Does anybody know how to add a key binding to a dynamic menu item created at run time?
Many thanks
Phil
--
Phil Mason
Broadcom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/cdt-dev/attachments/20111128/a5c53997/attachment.htm>
------------------------------
Message: 3
Date: Mon, 28 Nov 2011 08:52:39 -0800
From: "Phil (Philip) Mason" <phil.mason@xxxxxxxxxxxx>
To: "CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Subject: Re: [cdt-dev] Adding keybindings to dynamic menu items
Message-ID:
<C28713F579DB094D828AF937C214266A18D8DBD501@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset="us-ascii"
I think I've worked it out. I can use parameters to pass the different radices through to the handler. Then simply create a key bindings (in org.eclipse.ui.bindings) for the radices I need to have keyboard shortcuts with the correct parameter in each case.
Phil
From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Phil (Philip) Mason
Sent: 28 November 2011 13:16
To: CDT General developers list.
Subject: [cdt-dev] Adding keybindings to dynamic menu items
Hello all,
I've been trying to add keyboard shortcuts to the number format entries in the menu of the variable view and I'm struggling. The menu is a dynamic collection of Actions and I'm beginning to wonder if the problem is that the menu entries are not created until they are displayed so the any key bindings don't exist until then either.
Does anybody know how to add a key binding to a dynamic menu item created at run time?
Many thanks
Phil
--
Phil Mason
Broadcom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/cdt-dev/attachments/20111128/448e5165/attachment.htm>
------------------------------
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev
End of cdt-dev Digest, Vol 81, Issue 32
***************************************
