[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [smila-dev] RE: FYI :: new feature :: Message Resequencer

Hi,

 

Here are my answers:

 

> This issue _does not_ occur when crawling some data source.

there might be rare cases where it could occur there (links to the same resource, e.g. the same document referenced from 2 websites)

 

> (Crawling is the most common use case.)

not sure if crawling really is the most common case. In the past I usually integrated our former product more in the agent style

 

> This issue only occurs if the data source has been monitored by an agent _and_ the user is doing “ADD” (or “UPDATE”) and subsequently almost instantly an “DELETE” operation on the document (data set that is represented later as a record in SMILA).

a)      doesn’t only affect ADD/DEL ops but also ADD/ADD as pointed out

b)      > Instantly

a.       Highly depends on the setup

b.      generalized: as long as the change on the same resource occurs within the time period a previous change event is being processed.

 

> The chances that this happens are very low.

Highly depends on the setup and where u get the data from and the frequency that this data changes.

 

> agreed that it should be addressed in the connectivity module by a component called “Buffer”.

As I pointed out, this solution is

a)      not safe

b)      might not meet the application needs/requirements

 

> Since this issue occurs very rare, it can be generally rated as “low”.

Well, with the assumptions you have made, yes. If those assumptions fail: it is not low IMO. As I said: It depends on the use case. Here are some (think agent):

 

Use cases:

·         a wiki that I used by many users concurrently:

o   Here it can happen fairly frequently that the same page is saved twice in fast succession. At least it happens to me that after saving I notice a typo or add a quick note, resulting on another "save".

·         web application

o   in order to ease the DB load,  the search is the primary means to access the data, especially those where a complex SQL query would be crafted.

o   To have always an accurate result, a minimum time diff. between resource change and index update is required.

 

So long,

 

tom

 

From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Igor.Novakovic@xxxxxxxxxxx
Sent: Montag, 5. Oktober 2009 20:54
To: smila-dev@xxxxxxxxxxx
Subject: AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer

 

Hi Tom,

 

Fist, sorry for not reacting any sooner to your mails. I was “on the road” last week, which means that I was offline almost all of the time L

 

As suggested by you (in our phone call today), I won’t be answering to any of your mails that you wrote last week. Instead, I refer here to the latest version of the wiki page (http://wiki.eclipse.org/SMILA/Specifications/ProcessingMessageResequencer) that you created.

 

Before we go deeper into the technical discussion, I would like to clear some things first:

1.       We are actually talking about the use case/issue which can be shortly defined as: “The execution order of operations on _one_ particular record _does_ matter.”

2.       We want to be highly scalable. This implies that in general we always have more than one queue consumer. (Single queue consumer is only a special case.)

3.       It is not important just to execute operations in the right order (ADD and then DELETE). Even more important is _not to execute superfluous operations_ at all: Do not add something if it should be deleted right after that!

4.       We want to buffer user’s actions for at least a _couple of minutes_.

5.       Some people reading this discussion may ask themselves “How important is this use case at all?”, so let’s rate it:

·         Occurrence: very rare

                                                               i.      This issue _does not_ occur when crawling some data source. (Crawling is the most common use case.)

                                                             ii.      This issue only occurs if the data source has been monitored by an agent _and_ the user is doing “ADD” (or “UPDATE”) and subsequently almost instantly an “DELETE” operation on the document (data set that is represented later as a record in SMILA). The chances that this happens are very low.

·         Awareness: quite high

                                                               i.      We have been aware of this issue for more than a year. We discussed it and (after a short analysis) agreed that it should be addressed in the connectivity module by a component called “Buffer”.

·         Relevance: low

                                                               i.      Since this issue occurs very rare, it can be generally rated as “low”.

 

Can we agree on these statements?

If not, please correct me or add what’s being missing.

 

Cheers

Igor

 

Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] Im Auftrag von Thomas Menzel
Gesendet: Freitag, 2.
Oktober 2009 22:48
An: Smila project developer mailing list
Betreff: [smila-dev] RE: FYI :: new feature :: Message Resequencer

 

my work is done on the wiki page, and i "saw that it was good." ;)

 

till monday


From: smila-dev-bounces@xxxxxxxxxxx [smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Thomas Menzel [tmenzel@xxxxxxx]
Sent: Friday, October 02, 2009 4:30 PM
To: Smila project developer mailing list
Subject: [smila-dev] RE: FYI :: new feature :: Message Resequencer

FYI:

i'm updating the concpet page b/c i heard you wanted to look at either today or on monday.

 

tom


From: smila-dev-bounces@xxxxxxxxxxx [smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Thomas Menzel [tmenzel@xxxxxxx]
Sent: Monday, September 21, 2009 2:00 PM
To: Smila project developer mailing list
Subject: [smila-dev] FYI :: new feature :: Message Resequencer

hi folks,

 

just wanted to announce and inform you that I will be working on the problem that messages don’t get out of sync when there are changes in close succession

 

this change will be tracked thru the bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=289995

 

 

Kind regards

Thomas Menzel @ brox IT-Solutions GmbH

 

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.14.3/2415 - Release Date: 10/05/09 06:19:00