[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [smila-dev] Controlling Tasks Order Concept
One suggestion would be to attach state to record. State could be something
like NEW, IN_PROGRESS, SUCCESS, ERROR ...
By tracking the record state it my indicate what action should be performed
(if necessary ... for example in case of ERROR state).
----- Original Message -----
Sent: Thursday, October 09, 2008 5:05 PM
Subject: AW: [smila-dev] Controlling Tasks Order Concept
in the existing concepts the so called Buffer of ConnectivityManager (see
was meant to deal with these problems.
Some more thoughts:
- do we really want to allow concurrent usage of agents and crawlers on
the same datasource ? If so we also have to adopt the current usage of
DeltaIndexinManager, as it blocks concurrent usage.
- I agree that there are scenarios where race conditions occur, but I also
claim that these are special cases that do not happen all the time. So in
my eyes the standard use case has the be optimized in regards to
performance, these special cases have to be optimized in regards to
robustness. The handling of these special cases should not have any (or as
less as possible) impact on the standard cases.
- asynchronous processing of different records is OK, asynchronous
processing of the same record is NOT OK (it may lead to corrupt data)
- this is a highly complex functionality, I think we have to discuss it in
greater detail. We should list the uses cases and how we expect SMILA to
handle them. Then we can discuss a technical solution.
- I also think that we need some mechanism to identify that the processing
of a record has finished, either successfully or not (it then may be moved
to a dead-letter-queue). E.g. it may be needed if events should be
triggered after processing
Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
bounces@xxxxxxxxxxx] Im Auftrag von Ivan Churkin
Gesendet: Donnerstag, 9. Oktober 2008 15:15
An: Smila project developer mailing list
Betreff: Re: [smila-dev] Controlling Tasks Order Concept
Give me some feedback, please ;)
This is very significant problem of architecture...
Now problem is not visible because we only manually starts one Crawler.
It becomes very actual when Agents will be added.
The page contains my ideas for solution only. Unfortunately
documentation for every case will costs time.
If my explanations was not good and its required to write complete
documentation about also inform me.
smila-dev mailing list