RE: [smila-dev] RE: FYI :: new feature :: Message Resequencer

[Thomas Menzel <tmenzel@xxxxxxx>]

Hi igor,

> But I am not solely to blame on that since:
> 	- Your proposal is everything else but trivial. Actually you
> proposed 2 things...
> 	- The description is 8k plain text long.
> 	- You introduced almost a dozen of new terms and
> abbreviations...

The principle is **very trivial**. 
I just take time and space to explain it and see if all requirements are met. 
Sorry, if that is too much for you, but in order to be precise and complete these things just need to happen.

> 1. Your approach is not general. You concentrate on something you call "processing targets"
Sorry, but my approach is _very_ general IMO, or what do you think limits it to a specific use case?

> and completely neglect the price of preprocessing.
Where do I do this?

> Please forget the processing target 
> and assume, that each pipelet in
> the
> Pipeline fetches the record from the blackboard and stores it either in
> blackboard or in some own store (e.g. index file). 
If it has its own store and that requires correct order, then this pipelet is a PT, correct. 
IMO there shouldn't be many of these, most should be stateless.

> Also assume that
> every Pipelet takes some significant time to perform. Therefore we must
> prevent the unnecessary calls of _any_ pipelet.

+1

> 2. Both "full" and "smart" resequencer share the same problem: In a
> cluster environment you will have at least one resequencer instance per
> node. How will those instances share their ID-SN-Maps in a performant
> way?

I just updated that section to explain that better.


> Done. Please take a look at " Idea 1 - Connectivity
> Consolidation/Resequencer Buffer" section of
> http://wiki.eclipse.org/SMILA/Specifications/ProcessingMessageResequenc

I did. Unfortunately you only left a few comments on the things I pointed out. What I actually had asked you, is to explain for each listed requirement, i.e. these:

# 2.1 Basic Operations
# 2.2 compound management, splitting of records
    * 2.2.1 Composition
    * 2.2.2 Aggregation
    * 2.2.3 Parent/Descendants Ordering Requirement
# 2.3 support >1 processing targets
# 2.4 clustering, complex processing chain
# 2.5 single point of failure
# 2.6 scalability and performance

...how the buffer will solve it. For this you will have to rework the section a bit.
Remember: a requirement is only met when it works in all worst case scenarios otherwise it dosnt. In the latter case you need to state conditions where it works and where it does not.

--- snip ----

Other subject: 
I'm thinking of giving each solution an own page, e.g. one for buffer, FSR and SRS.
What do you think of that?

Kind regards
Thomas Menzel @ brox IT-Solutions GmbH


> -----Original Message-----
> From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> bounces@xxxxxxxxxxx] On Behalf Of Igor.Novakovic@xxxxxxxxxxx
> Sent: Mittwoch, 7. Oktober 2009 13:00
> To: smila-dev@xxxxxxxxxxx
> Subject: AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer
> 
> Hi,
> 
> > > scalability
> > Your argument against this point gives me the impression that you
> might not have
> > understood my proposal fully.
> Well, that might be true.
> But I am not solely to blame on that since:
> 	- Your proposal is everything else but trivial. Actually you
> proposed 2 things...
> 	- The description is 8k plain text long.
> 	- You introduced almost a dozen of new terms and
> abbreviations...
> 
> Nevertheless, I've just read it again (several times :-) and still see
> some serious flaws in it:
> 1. Your approach is not general. You concentrate on something you call
> "processing targets" and completely neglect the price of preprocessing.
> Please forget the processing target and assume, that each pipelet in
> the
> Pipeline fetches the record from the blackboard and stores it either in
> blackboard or in some own store (e.g. index file). Also assume that
> every Pipelet takes some significant time to perform. Therefore we must
> prevent the unnecessary calls of _any_ pipelet.
> 2. Both "full" and "smart" resequencer share the same problem: In a
> cluster environment you will have at least one resequencer instance per
> node. How will those instances share their ID-SN-Maps in a performant
> way?
> 
> 
> > > I'm confident that this is possible. The way to do this is by
> buffering
> > > document operations.
> > > Can we discuss buffering now?
> >
> > Do it, and show that all requirements are met.
> > I suggest you do this in the buffer section explaining after the
> manner I have
> > described the SRS, that is
> > - working principle:
> > - show how each requirements is or may be met. Plz also include all
> the items I
> > mentioned thus far.
> Done. Please take a look at " Idea 1 - Connectivity
> Consolidation/Resequencer Buffer" section of
> http://wiki.eclipse.org/SMILA/Specifications/ProcessingMessageResequenc
> e
> r
> 
> Regards
> Igor
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev
> 
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.420 / Virus Database: 270.14.4/2417 - Release Date:
> 10/07/09 05:18:00