Hi Tom,
I see two drawbacks
of your proposed solution(s):
1) it will only work on one machine. In a
distributed environment it is not guaranteed, that a Resequencer will get all
the relevant messages concerning one Record. And as each Resequencer has it’s
own map of Ids and sequence numbers two competing operations would not be
recognized as such. The map has to be shared across all Resequencer instances
(e.g. by using another Queue, or a database).
The initial idea of the Buffer component in Connectivity
was to filter out and resolve competing operations before they enter the “system”,
that is before they are processed. Of course this Buffer would also have to
share its internal state across all instances (At the moment a Agent/Crawler is
bound to one instance of Connectivity, so this distribution is not relevant,
yet). In either case, the processing is left totally untouched by introducing
the Buffer component., which leads me to my second issue:
2) The workflow has to be adapted to architecture
changes. So in order to benefit from the Resequencer business logic, workflows
have to be designed in special ways (first do some processing, second store
thee processed data). I think this is hard to grasp by users. BTW: would the
actual storing be configurable, I mean will the Resequencer execute a BPEL
pipeline or is the LuceneIndexing hardcoded ? The latter is of course no valid
scenario, we have to be flexible in this regard, as users may want to store
their data in arbitrary stores/indexes/whatsoever
Perhaps you could
elaborate about your concerns with our initial Buffer idea ?
Bye,
Daniel
Von:
smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] Im
Auftrag von Thomas Menzel
Gesendet: Dienstag, 22. September 2009 07:28
An: Smila project developer mailing list
Betreff: RE: [smila-dev] Message Resequencer :: change to Agent
Interface
oops,
http://wiki.eclipse.org/SMILA/Specifications/ProcessingMessageResequencer
PS: all along writing
this draft I had this mail open but still managed to forget to add link….
Kind regards
Thomas Menzel @ brox IT-Solutions GmbH
From: smila-dev-bounces@xxxxxxxxxxx
[mailto:smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Igor.Novakovic@xxxxxxxxxxx
Sent: Montag, 21. September 2009 19:04
To: smila-dev@xxxxxxxxxxx
Subject: AW: [smila-dev] Message Resequencer :: change to Agent
Interface
Hi Thomas,
Could you please
provide us the link to your specification draft?
Cheers
Igor
Von:
smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] Im
Auftrag von Thomas Menzel
Gesendet: Montag, 21. September 2009 18:28
An: Smila project developer mailing list
Betreff: [smila-dev] Message Resequencer :: change to Agent Interface
Hi,
I wrote a
specification draft for this change. plz feel free to comment.
in order for this to
implement I will need to change the interface of the agent.
Kind regards
Thomas Menzel @ brox IT-Solutions GmbH
From: smila-dev-bounces@xxxxxxxxxxx
[mailto:smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Thomas Menzel
Sent: Montag, 21. September 2009 14:00
To: Smila project developer mailing list
Subject: [smila-dev] FYI :: new feature :: Message Resequencer
hi folks,
just wanted to announce and inform you that
I will be working on the problem that messages don’t get out of sync when
there are changes in close succession
this change will be tracked thru the bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=289995
Kind regards
Thomas Menzel @ brox IT-Solutions GmbH