RE: [smila-dev] RE: FYI :: new feature :: Message Resequencer

[Thomas Menzel <tmenzel@xxxxxxx>]

Hi all,

aahhh, finally some discussion and constructive feedback.

> I think that solving the problem after a Record has entered the Queue
> is way too complicated, at least if we want to give the user the
> freedom to design his pipelines (including any rerouting to follow up
> Queues) as he likes.

That would be really great!!

> So I think that all proposals assume a specific pipeline and Queue
> (listener/router) configuration to work. 
Only to the extend that the SRS must be called at the beginning with REGISTER and at the end with RESEQUENCE.
Other than that you are free to do what u want.

> ... and that Tom
> assumes exactly two pipelines, one for processing and one for indexing
> where the Resequencer is right between those pipelines).

That was originally so, but not anymore. The SRS can be just put as a pipelet in front of the indexing pipelet.

> What I think is also missing is the fact, that the content of a record
> may get overwritten and that processing is resumed with different data
> than expected. (remember, there are multiple record instances sharing
> the same record Id and we are in a multithreaded environment, never
> knowing what is executed at what point in time).

This very fact has led me to the SRS.

> The most generic solution is to allow only one instance of a record to
> be queued/processed in a given time frame. This time frame highly
> depends on the data and the processing workflow (processing time,
> timeouts, retries, errors, ...) but otherwise it's not of concern how
> the pipelines and queues are configured. This is was the Buffer was
> initially intended to be. So instead of racking our brains how the
> desired functionality might be implemented on the right hand side of
> the Queue (the workflow) it may be worth thinking about solving it on
> the left hand side (in Connectivity), cause I think that no one has
> come up with a real concept for the Buffer, yet.

For i=0 to infinity { echo +1 } 
;)

If have tried and don't see such a solution. Admittedly, I have been more concerned with the SRS than the buffer b/c I'm not its originator nor am I pushing it for lack of confidence in its viability.

Maybe I have some good thoughts/ideas tonight.

Kind regards
Thomas Menzel @ brox IT-Solutions GmbH


> -----Original Message-----
> From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> bounces@xxxxxxxxxxx] On Behalf Of Daniel.Stucky@xxxxxxxxxxx
> Sent: Mittwoch, 7. Oktober 2009 18:24
> To: smila-dev@xxxxxxxxxxx
> Subject: AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer
> 
> Hi all,
> 
> it seems that this discussion gets a little out of control :-)
> 
> I think that solving the problem after a Record has entered the Queue
> is way too complicated, at least if we want to give the user the
> freedom to design his pipelines (including any rerouting to follow up
> Queues) as he likes.
> 
> So I think that all proposals assume a specific pipeline and Queue
> (listener/router) configuration to work. (I guess Jürgen assumes one
> pipeline containing all processing including Lucene index and that Tom
> assumes exactly two pipelines, one for processing and one for indexing
> where the Resequencer is right between those pipelines).
> 
> What I think is also missing is the fact, that the content of a record
> may get overwritten and that processing is resumed with different data
> than expected. (remember, there are multiple record instances sharing
> the same record Id and we are in a multithreaded environment, never
> knowing what is executed at what point in time).
> 
> 
> The most generic solution is to allow only one instance of a record to
> be queued/processed in a given time frame. This time frame highly
> depends on the data and the processing workflow (processing time,
> timeouts, retries, errors, ...) but otherwise it's not of concern how
> the pipelines and queues are configured. This is was the Buffer was
> initially intended to be. So instead of racking our brains how the
> desired functionality might be implemented on the right hand side of
> the Queue (the workflow) it may be worth thinking about solving it on
> the left hand side (in Connectivity), cause I think that no one has
> come up with a real concept for the Buffer, yet.
> 
> 
> Bye,
> Daniel
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev
> 
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.420 / Virus Database: 270.14.4/2417 - Release Date:
> 10/07/09 05:18:00