[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer

Hi all,

it seems that this discussion gets a little out of control :-)

I think that solving the problem after a Record has entered the Queue is way too complicated, at least if we want to give the user the freedom to design his pipelines (including any rerouting to follow up Queues) as he likes.

So I think that all proposals assume a specific pipeline and Queue (listener/router) configuration to work. (I guess Jürgen assumes one pipeline containing all processing including Lucene index and that Tom assumes exactly two pipelines, one for processing and one for indexing where the Resequencer is right between those pipelines).

What I think is also missing is the fact, that the content of a record may get overwritten and that processing is resumed with different data than expected. (remember, there are multiple record instances sharing the same record Id and we are in a multithreaded environment, never knowing what is executed at what point in time).

The most generic solution is to allow only one instance of a record to be queued/processed in a given time frame. This time frame highly depends on the data and the processing workflow (processing time, timeouts, retries, errors, ...) but otherwise it's not of concern how the pipelines and queues are configured. This is was the Buffer was initially intended to be. So instead of racking our brains how the desired functionality might be implemented on the right hand side of the Queue (the workflow) it may be worth thinking about solving it on the left hand side (in Connectivity), cause I think that no one has come up with a real concept for the Buffer, yet.