RE: [smila-dev] RE: FYI :: new feature :: Message Resequencer

[Thomas Menzel <tmenzel@xxxxxxx>]

Hi all,

If i may share my 2 cents here:

a) Parallel workflows should be totally legitimate and not illegal.

Imagine that you want to process a record in two completely different ways ending up in diff indexes. Why should we force it to run in serial fashion if the customer provides the computing power?

Also the router explicitly allows several Send tasks in its config, which we would have to take out.


b) following your discussion it seems to me that you slowly approach the idea where you need to register first and unregister at the end, albeit you use the terms (un)lock and move the functionality into existing components.

Anyhow, reading your mails and thinking  more about the buffer I realized there is a serious concurrency design flaw in the SRS. The problem is that I have always assumed that older PRs will have moved on further than the newer PRs but this is not certain at all!  in a highly concurrent setup 
1. processing an older PR can linger at processing step 12 while a newer one has almost finished processing completely. With a persisting BB -- and thus a shared record instance -- this will cause inconsistent states of the record.
2. there may be already several PRs for the same item in the Q all waiting for the first processing step. With listeners and the use of selectors (ie. our rules) JMS stipulates no requirements on the order of messages taken from the Q. this means that the new PR can be taken from the Q before the older one.

To solve this, I have to agree with you that we need a buffer that allows us to queue and consolidate subsequent PRs as long as the item in question is being processed.


New idea:

An idea that I had (but not thought thru yet) was to have such a buffer in connectivity myself but I don't want to delay all PRs by a fixed amount of time. instead I want to have pipelets just before calling the PT to signal to connectivity that processing has completed (this could also be added to the impl. Of the PT itself, would make it more complicated though). However, this idea poses problems in a cluster setup:
- what kind of remote communication shall we use?
- is it safe enough? Must it be safe?
Thought: since we use an MQ anyhow we just could open up another Q to send such messages back. IMO these messages could be even UDP style: in case connectivity misses one or that Q dies, a timeout could solve the problem.

New idea2:
Take the core of juergen's idea and instead of opening up a buffer, map or a Q in addition to the recordstore, place additional information associated with the record not as part of it, so that it is not shared. I also think that we have to give each processing step a sequential number which is taken into account to determine who may still do smth.

Note: these ideas are just unfiltered ramblings of my brain and still need much more work and thinking. But maybe it triggers an idea with you...

Good night


Kind regards
Thomas Menzel @ brox IT-Solutions GmbH

> -----Original Message-----
> From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> bounces@xxxxxxxxxxx] On Behalf Of Igor.Novakovic@xxxxxxxxxxx
> Sent: Donnerstag, 8. Oktober 2009 20:40
> To: smila-dev@xxxxxxxxxxx
> Subject: AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer
> 
> > Yes, you do :-)
> > It's not a missconfiguration.
> >
> > Imagine that a workflow is not expressed with a single pipeline but
> by
> using
> > multiple pipelines. The pipelines are connected via different Queues.
> So you would
> > have the following walktrough:
> >
> > Connectivity -> Queue1 -> Pipeline1 -> Queue2 ->Pipeline2 -> ...
> QueueN ->
> > PipelineN
> >
> > Before each Queue is a Router and after each Queue is a Listener. You
> only want to
> > unlock the record after PipelineN has finished processing.
> I said that the misconfiguration is only the case where the _router_
> sends one record to more than one queue.
> What you are talking about is a "cascading" workflow.
> In this case you could easily (e.g. via listener configuration)
> instruct
> the listener of the PipelineN to remove the lock. All other listeners
> (listening on the pipelines 1 to N-1) would leave the lock just as it
> is.
> 
> 
> > Such a setup is useful to achieve a better scalability. And I guess
> you have
> > already seen such setups :-)
> Sure. We at Empolis are doing just that ;-)
> 
> Cheers
> Igor
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev
> 
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.421 / Virus Database: 270.14.7/2421 - Release Date:
> 10/08/09 06:39:00