-----Original Message-----
From: cdt-dev-admin@xxxxxxxxxxx
[mailto:cdt-dev-admin@xxxxxxxxxxx] On Behalf Of Chris Wiebe
Sent: January 13, 2005 4:47 PM
To: cdt-dev@xxxxxxxxxxx
Subject: Re: [cdt-dev] Indexer Requirements for 3.0
Looks good, Bogdan. Will this also include indexing of
external headers (from the project include paths)?
Chris
Bogdan Gheorghe wrote:
Here is a list of indexer requirements for the 3.0 release. The
overall focus for the indexer for 3.0 is on improving the
scalability
and overall usability of the indexer service for all
clients. Feedback
is always appreciated - especially if you think there are missing
requirements. A more detailed design doc is to follow.
Thanks,
Bogdan
----------------------------------------------------------------------
--
Indexer Requirements for 3.0
This document describes the proposed work items for the
indexer for the
CDT 3.0 release.
Author : Bogdan Gheorghe
Revision Date : 11/29/2004 - Version: 0.1.0
: 01/10/2005 - Version: 0.1.1
Change History : 0.1.0 - Document Creation
: 0.1.1 - Revision
Table of Contents
1. Introduction <#intro>
2. Requirements <#reqs>
3. UI Requirements <#proposal>
4. References <#references>
1. Introduction
The Indexer has been around since CDT 1.2 and currently provides
support for Search, Navigation, and Refactoring. Its main
purpose is
to provide rapid access to a complete database of code
elements and to
manage this database in an efficient and non-intrusive
manner. As the
CDT has evolved; so has the indexer - adding more elements to the
index, refining job scheduling, providing feedback
mechanisms for indexes.
Although the indexer is sufficiently developed to provide most
requested information to clients; it has become clear that the next
step in the indexer's evolution will have to address its ability to
handle very large projects efficiently.
Having the indexer work well on large scale projects
requires some new
architecture to reduce the amount of time spent indexing as much as
possible, reuse existing indexes as much as possible and
provide users
with mechanisms to extend the index framework.
This document will address main requirements on the indexer
for CDT 3.0.
1.0 Definitions
Resource
A project, folder or file within the Eclipse workspace
Index profiles
Separate indexes that are created for different
configurations of
include paths/symbols
1.1 Current Architecture
Quick overview of the indexing architecture:
1. The indexer responds to resource events from the
workbench. These
events occur whenever a resource gets created,
modified or deleted.
2. The indexer will create index jobs based on the
resource events.
These jobs might schedule other jobs (such as in the case of
indexing an entire project) but most index jobs
eventually boil
down to an AddCompilationUnitToIndex job.
3. The indexer creates a new parser, passes in the
current include
paths and symbol definitions and parses the file and any other
files included by the file in full parse mode (which generates
cross reference information).
4. The index gets created as the parser returns
information about the
elements in the file - the index is stored in memory and at
certain intervals gets merged with the persisted
index on disk.
Currently the indexes have the following structure:
* Index Version Number
* Summary Block Location
* File Blocks [1 ... N]:
o Each full file block is 8k
o File block entries associate the path of a file
with a unique ID
* Word Blocks [1 ... N]:
o Each full word block is 8k
o Word block entries encode the element in a
special format
and add the referring file number
* Include Blocks [1 ... N]:
o Each full include block is 8k
o Include block entries associate a file with the
initial file
that was parsed to get to the current file
* Summary Block:
o Keeps track of the total number of words, files
and include
entries in the index
o Keeps track of the first file block number, the
first word
block number and the first include block number
o Keeps track of the first file for every File Block, the
first word entry for every Word Block, and the
first include
for every Include Block
1.1 Constraints
Number
Description
C1
Multiple indexer profiles not possible until similar build
configuration notion appears
In order to get an accurate index the indexer depends on
being able to
pass on the relevant includes/symbols to the parser. As
long as there
is no standard representation for build configurations in the core
model for all build types (both standard and managed), it isn't
possible to enable index profiles.
Bug 25682: Indexer Profiles
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=25682>
2. Requirements
List of requirements for the indexer is classified into following
categories:
2.1 Index Management Requirements
Number
Priority
Description
R1
P1
Indexer must provide different types of indexing services
It has become apparent that the one-size-fits all indexing approach
does not meet all of our users needs. Most clients with existing
legacy projects want some form of search/navigation but some don't
want the hassle of having to wait for an entire project to
index fully
before being able to use search/navigation. Thus, in order to
accommodate both sets of user groups (those who are willing to wait
for a full index to complete and those who just want the
absolute bare
minimum index) we need to offer the following indexer options:
1. Full Index: this is the "regular" index mode which
uses the CDT
parser (include paths and symbol definitions need to be setup
properly); everything gets indexed
2. Quick Index with no setup: this will result in a "best effort"
bare bones index that should enable some navigation/search
These indexer options are per project - so it is possible to have
different indexers for different projects.
Bug 69078: C/C++ indexer too slow
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=69078>
R2
P1
Indexes should be shareable between team members
As part of streamlining the indexing process for large projects,
indexes should be able to be shared between users working
on the same project.
All index entries should make use of path variables in
order to allow
indexes to be translated into different workspace locations. (See
Scanner Config Correctness Enhancement FDS for more details
about path
variables)
Bug 79661: All Index Entries should make use of Path Variables
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=79661>
Bug 79518: Path/Variable Manager support service in the core(string
substitution) <https://bugs.eclipse.org/bugs/show_bug.cgi?id=79518>
R3
P1
Indexer should be able to index a project offline
For features that require a complete index it would be
ideal to have
the index be created somewhere separately and imported in
at a later date.
This is especially true for medium to large projects that
need a long
time to index.
Bug 74433: Offline Indexing/Index Hierarchy
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=74433>
R4
P1
Indexer should be able to merge indexes
With all of the new options for indexing, it is conceivable
that any
project might have several sources to look at for a single
index. The
indexer should be able to merge new index information into
an existing
index (through a user action), provided that both index
formats are alike.
Bug 52126: Indexer should maintain per project indices
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=52126>
Bug 74433: Offline Indexing/Index Hierarchy
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=74433>
R5
P2
Indexer should allow user to specify indexer settings
Currently the indexer uses the same settings for all projects. This
might not suit all users. Indexer settings that should be
customizable
include:
* Indexing policy: (default normal)
o normal (always up to date)
o manual (index only when manually requested)
o static (don't update current index)
o after build (don't index until build)
* Index Progress Bar displayed: (Checkbox, default displayed)
* Indexer default setting for new projects:
o Currently the indexer is always on when creating a new
project. This should be changed to allow users
to set the
default index behaviour when creating a new project
Bug 75884: Allow C/C++ Indexing to be set on or off as a default
setting <https://bugs.eclipse.org/bugs/show_bug.cgi?id=75884>
R6
P1
Index Manager must improve job scheduling smarts
The indexer currently has limited smarts when it comes to job
scheduling; it will prevent the same job from being queued up. But
there are other circumstances when being more aware of
what's in the
job queue would solve numerous problems: including the ever
recurring
double index, and source folder changes. Essentially, more
information
needs to be added to individual job requests about the event that
created the job in order to enable the index manager to make smart
choices about how to best schedule the jobs.
The index manager should also run at the minimum priority.
Bug 60084: Indexer should reduce its priority when running in
background <https://bugs.eclipse.org/bugs/show_bug.cgi?id=60084>
R7
P1
Indexer must try to keep as much as possible from all
failed index
attempts
The indexer needs to persist a list of all files that are to be
indexed as part of an index job. As each merge happens, the list is
updated. In the event of a crash, all work that has been merged to
disk shall be considered as sane and the indexer will
restart on the
remaining files on the next startup.
Bug 62366: [Index] Need ability to read a partial index and resume
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=62366>
R8
P1
Indexer needs to deal with new path/symbol addition/deletion
gracefully
The indexer currently reindexes the entire project on the
addition/deletion of each new path/symbol. With the new per-file
scanner settings, we should only index the range of files
that are affected.
(See Scanner Config Correctness Enhancement FDS for more
details about
per file scanner settings).
R9
P1
Indexer needs to handle Source Folder changes gracefully
The indexer currently does a brute force reindex of
whenever anything
changes in the source folder view. This needs to change to a more
scalable solution.
R10
P4
Resources can be added manually to the index
Users can request a new index on any resource in the workspace
(including files that are included from external directories) by
selecting one or more files and choosing the appropriate
option from
the context menu.
Bug 71821: Action on the indexer to parse file/folder
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=71821>
R11
P2
FileType changes should trigger indexes
Changing the file type settings on a project might
introduce new file
types as extensions that have not been indexed as of yet.
The indexer
should react in accordance to the users settings (as defined in R5).
Bug 72396: The indexer is not aware of the File Type
(ResolverModel)
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=72396>
R12
P1
Indexer Extension point
If all the current versions of the indexer don't meet the users
requirements, the CDT should provide an extension point that will
allow users to write their own indexer that will populate
the index.
Provided the information is complete, all index-based CDT features
should work as normal.
R13
P4
Index Manager should allow for ongoing search while indexing
The indexer should be able to compare incoming index
entries with any
pending search queries and return the matches.
Bug 72803: [Performance][Usability][Indexer/Search] Waiting
policy for
index results <https://bugs.eclipse.org/bugs/show_bug.cgi?id=72803>
Bug 53792: [Scalability] Prioritizing the indexing when searching a
working set <https://bugs.eclipse.org/bugs/show_bug.cgi?id=53792>
R14
P1
Index should provide an interface to allow clients to
determine what
features are available with the current indexes
With the prospect of having various levels of detail available in a
project's indexes, the Index Manager needs to be able to provide an
interface that will answer any client whether there is sufficient
information in the index to run the client's service. If
the necessary
index detail is missing, it would be up to the client to
ask the user
if they wish to schedule a new index.
2.2 Index Content Requirements
Number
Priority
Description
R15
P1
Indexer will provide enough information to run searches
without a
second parse
Currently the index stores just the location information for the
entries
- this currently causes search to require two separate
parses: one for
the initial index and another to determine the offset
information. The
indexer should store this offset information in the initial
index and
thus avoid the second parse. Enough information must be
available in
the index to answer all of the possible MatchLocator queries.
Additional info that needs to be added to the indexer includes:
* function/method parameters
* if a variable has an initializer clause, extern specifier or
linkage specification (needed to determine if it is a
definition)
* if a field is static (needed to see if we need to check for
definition)
Bug 74427: Indexer needs to store more info
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=74427>
R16
P3
References in the index should be tied into their declarations
We need to be able to match up references to their declarations
(possible requirement for refactoring support).
Bug 69606: [Search] Match locator has to make sure that the
reference
belongs to the specified declaration
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=69606>
R17
P1
New Indexer needs to be written for new AST
As the CDT switches over to the new AST, the indexer will
have to be
rewritten to extract information from the AST.
2.3 Problem Markers
Number
Priority
Description
R18
P2
Problem Markers should be able to be removed manually
There are a number scenarios in which Index problem markers
get placed
on resources and cannot be removed. There needs to be some sort of
menu option that can manually delete selected problem markers.
Bug 74284: [IProblem] All Problem markers removed for
project when one
file is excluded
<https://bugs.eclipse.org/bugs/show_bug.cgi?id=74284>
3. UI Requirements
3.1 UI enhancements
Following UI enhancements are planned to support the feature:
Number
Description
UE1 Indexer Options Preference Page
This page will allow users to make changes to the Indexer
that affect
the entire workspace:
* Index Progress Bar displayed
* Indexer New Project Default Setting
UE2
Indexer Project Properties Page
This page will allow users to set indexer settings per project:
* Indexer to use: if a number of indexers are available (ie.
SourceIndexer, SourceIndexer2, CTagsIndexer) , users
can specify
which indexer should be used for the entire workspace.
* Indexing policy to use
4. References
1. Scanner Config Correctness Enhancement FDS
/Last Modified on Monday, January 10, 2005 /
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/cdt-dev
