[Dltk-dev] Apache Lucene indexer for DLTK

[Bartłomiej Łączkowski <bartlomiej.l@xxxxxxxx>]

Hi All,

I would like to open a discussion on indexing in DLTK and a possibility 
to contribute some additional improvements by means of adding a new 
indexing engine to DLTK. Currently we are using DLTK H2 indexer in our 
project (PHP Development Tools) and we've found out that it brings some 
limitations for some of our particular use cases and overall performance 
issues. Due to this fact we decided to implement an alternative indexer 
engine based on Apache Lucene. Currently we are testing the very first 
version of it but we have already noticed a significant performance 
improvement in comparison to H2 indexer. Just to shed some light on the 
performance topic - indexing projects is ~40% faster, in some of the 
cases searching the index can be faster even up to 50% (in general 
searching speed is comparable with H2 but Lucene indexer seems to be 
faster overall). What's more, Lucene has better support for 
multi-threaded environment and the size of the index data on the drive 
is only about 40-50% of the H2 database size.

As we find these results very promising, we are thinking about 
contributing our solution to DLTK so it could be used as an additional 
choice for indexing engine in DLTK.

So, what do you think about it?

Greetings,
Bartlomiej