AW: [smila-dev] Datasource locking in Deltaindexing Manager

[<Daniel.Stucky@xxxxxxxxxxx>]

Hi all,

a locking mechanism is required to assure that only one client (in this case a CrawlerController crawling a specific data source) initializes and executes DeltaIndexing on this datasource. DeltaIndexing does not support parallel indexing of the same data source. This could be supported if DeltaIndexing-delete is disabeled (just adding/updating records).

Locking of the Thread id was the easiest solution without changing the interface I think.

One solution could be to use session IDs. A session ID would be created by calling initialize() and would have to passed with all other method calls.

Bye,
Daniel

> -----Ursprüngliche Nachricht-----
> Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> bounces@xxxxxxxxxxx] Im Auftrag von Juergen.Schumacher@xxxxxxxxxxx
> Gesendet: Mittwoch, 20. Mai 2009 09:37
> An: smila-dev@xxxxxxxxxxx
> Betreff: [smila-dev] Datasource locking in Deltaindexing Manager
> 
> Hi,
> 
> I'm currently experimenting with the Distributed OSGi (aka RFC 119)
> implementation of ECF 3.0
> for exposing SMILA services for remote calls. Specifically I tried to
> expose the Connectivity
> Manager service so that we can have Crawlers and Agents talk to a
> remote service instead of having
> them always in the same VM. In principal this works quite OK, but I
> encountered a problem in the
> Delta Indexing manager: When it initializes a data source for a crawl
> process, it locks it and
> associated it to the current thread. This works fine when the crawler
> is in the same VM because
> then the complete crawl is done in a single crawl thread. However, with
> remote access to Connectivity
> Manger this is not true anymore: servers have usually worker thread
> pools for executing remote
> calls and consecutive calls from a single client are not served by the
> same thread. Of course we
> could restrict the thread pool size to 1, but I suppose that would kill
> scalability just a bit (-;
> 
> Thus, if we really need this data source locking for a crawl thread, I
> think we need a different
> method than the current one. Any ideas?
> 
> Cheers,
> Jürgen.
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev