Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
AW: [smila-dev] Controlling Tasks Order Concept

Hi Ivan,

in the existing concepts the so called Buffer of ConnectivityManager (see http://wiki.eclipse.org/SMILA/Project_Concepts/Connectivity#Buffer_.28P2.29) was meant to deal with these problems.

Some more thoughts:
- do we really want to allow concurrent usage of agents and crawlers on the same datasource ? If so we also have to adopt the current usage of DeltaIndexinManager, as it blocks concurrent usage.

- I agree that there are scenarios where race conditions occur, but I also claim that these are special cases that do not happen all the time. So in my eyes the standard use case has the be optimized in regards to performance, these special cases have to be optimized in regards to robustness. The handling of these special cases should not have any (or as less as possible) impact on the standard cases.

- asynchronous processing of different records is OK, asynchronous processing of the same record is NOT OK (it may lead to corrupt data)

- this is a highly complex functionality, I think we have to discuss it in greater detail. We should list the uses cases and how we expect SMILA to handle them. Then we can discuss a technical solution.

- I also think that we need some mechanism to identify that the processing of a record has finished, either successfully or not (it then may be moved to a dead-letter-queue). E.g. it may be needed if events should be triggered after processing

Bye,
Daniel


> -----Ursprüngliche Nachricht-----
> Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> bounces@xxxxxxxxxxx] Im Auftrag von Ivan Churkin
> Gesendet: Donnerstag, 9. Oktober 2008 15:15
> An: Smila project developer mailing list
> Betreff: Re: [smila-dev] Controlling Tasks Order Concept
> 
> Hey guys,
> 
> Give me some feedback, please ;)
> This is very significant problem of architecture...
> Now problem is not visible because we only manually starts one Crawler.
> It becomes very actual when Agents will be added.
> 
> The page contains my ideas for solution only. Unfortunately
> documentation for every case will costs time.
> If my explanations was not good and its required to write complete
> documentation about also inform me.
> 

Back to the top