AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer

[<Igor.Novakovic@xxxxxxxxxxx>]

Hi,


> I think that solving the problem after a Record has entered the Queue
is way too
> complicated, at least if we want to give the user the freedom to
design his
> pipelines (including any rerouting to follow up Queues) as he likes.

Fully ACK!
That is exactly what I meant!


> The most generic solution is to allow only one instance of a record to
be
> queued/processed in a given time frame. 

I think that this is a _key_ point!


> This time frame highly depends on the data
> and the processing workflow (processing time, timeouts, retries,
errors, ...) but
> otherwise it's not of concern how the pipelines and queues are
configured. This is
> was the Buffer was initially intended to be. So instead of racking our
brains how
> the desired functionality might be implemented on the right hand side
of the Queue
> (the workflow) it may be worth thinking about solving it on the left
hand side (in
> Connectivity), cause I think that no one has come up with a real
concept for the
> Buffer, yet.

Like you, I advocate the buffer approach.
But today, when I did some in depth analysis, I realized that the buffer
concept is the way to go when it comes to eliminating superfluous
operations _but_ it does not completely prevent (there is always a small
chance) having two or more operations for the same document in the
queue.

So to enhance the buffer concept I propose an enhancement:
Why not lock the record right after the router has added a new message
to the queue and unlock it when that message has been successfully
consumed (which would be done by listener I guess)?
Of course, this lock should be only obeyed by the connectivity module.
The pipelets in the pipeline will still be able to update/overwrite the
record.
This way we will prevent overwriting of the record by the connectivity
module while it has been processed.

We should consume events only when we are "ready". This would lead to
having agents only delivering notices about events. These
events/operations would then be buffered by the buffer in a dedicated
queue and only when the buffer decides to "let through" one operation
the actual content (and the rest of the metadata) would be requested
from the agent.

Cheers
Igor