Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [Dltk-dev] Apache Lucene indexer for DLTK
Hi All,

I've added & pushed some small changes to be compatible with Lucene 5.2.1. After this changes we need only two jars from Lucene: lucene-analyzers-common-5.2.1.jar and lucene-core-5.2.1.jar. Taking into account the list of jars that are listed in https://bugs.eclipse.org/bugs/show_bug.cgi?id=465874, it looks like it might be possible for the CQs to be approved in time for Neon.

Regards,
Bartlomiej

On 4/8/2016 10:26 AM, Kaloyan Raev wrote:

Hi,


I think Alex K is right - there is no chance for the CQs to be approved in time for Neon.


Our code is designed for Lucene 5.5. Although it is technically possible to change it to be compatible with 5.2.1, we still don't have all the required Lucene libraries already approved.


Therefore, it would be best to target this contribution for Neon.1.


For Neon.0, we will still implement bug 491142 [1], because this will help us (Zend) as adopters to force the new Lucene indexer to be the default one in our product.


I'll take care for submitting the CQs.


[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=491142

Thanks,
Kaloyan

From: dltk-dev-bounces@xxxxxxxxxxx <dltk-dev-bounces@xxxxxxxxxxx> on behalf of akurtakov <akurtakov@xxxxxxxxx>
Sent: Thursday, April 7, 2016 9:38 PM
To: DLTK Developer Discussions
Subject: Re: [Dltk-dev] Apache Lucene indexer for DLTK
 


On Thu, Apr 7, 2016 at 9:24 PM, Bartlomiej Laczkowski <bartlomiej.l@xxxxxxxx> wrote:
Hi All,

I have just pushed the change proposal to Gerrit:  https://git.eclipse.org/r/70162

@Alexey: In case of Lucene we use a few libraries from the most recent version (5.5.0). Unfortunately none of them are available in Orbit.

Although not available in Orbit. There are CQs for Lucene 5.2.1 e.g. core is handled via https://dev.eclipse.org/ipzilla/show_bug.cgi?id=9967 . Work is ongoing to make Orbit use the new bundle-recipes build mechanism and Lucene should be handled that way https://bugs.eclipse.org/bugs/show_bug.cgi?id=478337 . There are CQs for a number of other lucene jars so if your code can work with 5.2.1 that's probably the easiest way as DLTK will be able to get approval for reuse easily and DLTK can ship the jars even from its p2 repo if Orbit is not fixed in time. Otherwise it's way past CQs deadline for Neon so submitting CQs now would mean missing Neon release.
Please let me know if you need any clarification about state of these things.

Regards,
Alex

 

@Dawid: If you would like to play around with the new indexer in PDT, you need to register PDT indexer participant with the new id for Lucene indexer and disable H2 indexer plug-ins in your launch configuration.
Regards,
Bartlomiej


On 4/4/2016 1:57 PM, Alexey Panchenko wrote:
Sounds good.
Do we need the latest Lucene version? is it in Orbit? or should the effort be started with CQ and all that stuff?

Regards,
Alex

On Fri, Apr 1, 2016 at 8:45 PM, Dawid Pakuła <zulus@xxxxxxxxx> wrote:
Hi,

I like this idea. anyway current index have couple problems:
1. Additional abstraction layer (internal ORM), I hope lucene indexer haven’t this and can work on DeclarationInfo/ReferenceInfo directly.
2. Innability to add additional properties (real metadata) to indexed elements (and use it for filters), with lucene should be easier to introduce additional api
-- 
Dawid Pakuła
+48 795 996 064

From: Michal Niewrzal <michal.n@xxxxxxxx>
Reply: DLTK Developer Discussions <dltk-dev@xxxxxxxxxxx>
Date: 1 April 2016 at 13:21:20
To: DLTK Developer Discussions <dltk-dev@xxxxxxxxxxx>
Subject:  [Dltk-dev] Apache Lucene indexer for DLTK

Hi All,


In my opinion new indexer is a good idea because:

* it's faster than actual (and probably there's still place for improvement)

* it's based on mature, well known library

* fits actual indexer API


New JDT indexer is interesting but for now it looks to be not finished. Another thing is that until someone will extract core functionality to generic bundle to use it it needs to be copied, adjusted and maintained &#X1f60a 


Regards,

Michal



Od: dltk-dev-bounces@xxxxxxxxxxx <dltk-dev-bounces@xxxxxxxxxxx> w imieniu użytkownika akurtakov <akurtakov@xxxxxxxxx>
Wysłane: 1 kwietnia 2016 11:58
Do: DLTK Developer Discussions
Temat: Re: [Dltk-dev] Apache Lucene indexer for DLTK
 
That would be really welcome addition from my POV.

Regards,
Alex Kurtakov

On Fri, Apr 1, 2016 at 12:50 PM, Bartłomiej Łączkowski <bartlomiej.l@xxxxxxxx> wrote:
Hi All,

I would like to open a discussion on indexing in DLTK and a possibility to contribute some additional improvements by means of adding a new indexing engine to DLTK. Currently we are using DLTK H2 indexer in our project (PHP Development Tools) and we've found out that it brings some limitations for some of our particular use cases and overall performance issues. Due to this fact we decided to implement an alternative indexer engine based on Apache Lucene. Currently we are testing the very first version of it but we have already noticed a significant performance improvement in comparison to H2 indexer. Just to shed some light on the performance topic - indexing projects is ~40% faster, in some of the cases searching the index can be faster even up to 50% (in general searching speed is comparable with H2 but Lucene indexer seems to be faster overall). What's more, Lucene has better support for multi-threaded environment and the size of the index data on the drive is only about 40-50% of the H2 database size.

As we find these results very promising, we are thinking about contributing our solution to DLTK so it could be used as an additional choice for indexing engine in DLTK.

So, what do you think about it?

Greetings,
Bartlomiej
_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/dltk-dev

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/dltk-dev

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/dltk-dev



_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/dltk-dev


_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/dltk-dev



_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/dltk-dev

Back to the top