[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [smila-dev] Controlling Tasks Order Concept

Hi Ivan

One suggestion would be to attach state to record. State could be something like NEW, IN_PROGRESS, SUCCESS, ERROR ...
By tracking the record state it my indicate what action should be performed (if necessary ... for example in case of ERROR state).

Best Regards,

----- Original Message ----- From: <Daniel.Stucky@xxxxxxxxxxx>
To: <smila-dev@xxxxxxxxxxx>
Sent: Thursday, October 09, 2008 5:05 PM
Subject: AW: [smila-dev] Controlling Tasks Order Concept

Hi Ivan,

in the existing concepts the so called Buffer of ConnectivityManager (see http://wiki.eclipse.org/SMILA/Project_Concepts/Connectivity#Buffer_.28P2.29) was meant to deal with these problems.

Some more thoughts:
- do we really want to allow concurrent usage of agents and crawlers on the same datasource ? If so we also have to adopt the current usage of DeltaIndexinManager, as it blocks concurrent usage.

- I agree that there are scenarios where race conditions occur, but I also claim that these are special cases that do not happen all the time. So in my eyes the standard use case has the be optimized in regards to performance, these special cases have to be optimized in regards to robustness. The handling of these special cases should not have any (or as less as possible) impact on the standard cases.

- asynchronous processing of different records is OK, asynchronous processing of the same record is NOT OK (it may lead to corrupt data)

- this is a highly complex functionality, I think we have to discuss it in greater detail. We should list the uses cases and how we expect SMILA to handle them. Then we can discuss a technical solution.

- I also think that we need some mechanism to identify that the processing of a record has finished, either successfully or not (it then may be moved to a dead-letter-queue). E.g. it may be needed if events should be triggered after processing


-----UrsprÃngliche Nachricht-----
Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
bounces@xxxxxxxxxxx] Im Auftrag von Ivan Churkin
Gesendet: Donnerstag, 9. Oktober 2008 15:15
An: Smila project developer mailing list
Betreff: Re: [smila-dev] Controlling Tasks Order Concept

Hey guys,

Give me some feedback, please ;)
This is very significant problem of architecture...
Now problem is not visible because we only manually starts one Crawler.
It becomes very actual when Agents will be added.

The page contains my ideas for solution only. Unfortunately
documentation for every case will costs time.
If my explanations was not good and its required to write complete
documentation about also inform me.


smila-dev mailing list