Hi,
Here are my answers:
> This issue _does not_ occur when
crawling some data source.
there might be rare
cases where it could occur there (links to the same resource, e.g. the same
document referenced from 2 websites)
> (Crawling is the most common use case.)
not sure if crawling
really is the most common case. In the past I usually integrated our former product
more in the agent style
> This issue only
occurs if the data source has been monitored by an agent _and_ the user
is doing “ADD” (or “UPDATE”) and subsequently almost
instantly an “DELETE” operation on the document (data set that is
represented later as a record in SMILA).
a) doesn’t only affect ADD/DEL ops but also
ADD/ADD as pointed out
b) > Instantly
a.
Highly depends
on the setup
b.
generalized:
as long as the change on the same resource occurs within the time period a
previous change event is being processed.
> The chances that
this happens are very low.
Highly depends on the
setup and where u get the data from and the frequency that this data changes.
> agreed that it should be addressed in the
connectivity module by a component called “Buffer”.
As I pointed out,
this solution is
a) not safe
b) might not meet the application
needs/requirements
> Since this issue occurs very rare, it can be
generally rated as “low”.
Well, with the
assumptions you have made, yes. If those assumptions fail: it is not low IMO. As
I said: It depends on the use case. Here are some (think agent):
Use cases:
·
a wiki
that I used by many users concurrently:
o Here it can happen fairly frequently that the
same page is saved twice in fast succession. At least it happens to me that after
saving I notice a typo or add a quick note, resulting on another "save".
·
web
application
o in order to ease the DB load, the search
is the primary means to access the data, especially those where a complex SQL
query would be crafted.
o To have always an accurate result, a minimum time
diff. between resource change and index update is required.
So long,
tom
From: smila-dev-bounces@xxxxxxxxxxx
[mailto:smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Igor.Novakovic@xxxxxxxxxxx
Sent: Montag, 5. Oktober 2009 20:54
To: smila-dev@xxxxxxxxxxx
Subject: AW: [smila-dev] RE: FYI :: new feature :: Message Resequencer
Hi Tom,
Fist, sorry for not
reacting any sooner to your mails. I was “on the road” last week,
which means that I was offline almost all of the time L
As suggested by you
(in our phone call today), I won’t be answering to any of your mails that
you wrote last week. Instead, I refer here to the latest version of the wiki
page (http://wiki.eclipse.org/SMILA/Specifications/ProcessingMessageResequencer)
that you created.
Before we go deeper
into the technical discussion, I would like to clear some things first:
1. We are actually talking about the use case/issue
which can be shortly defined as: “The execution order of operations on _one_
particular record _does_ matter.”
2. We want to be highly scalable. This implies
that in general we always have more than one queue consumer. (Single queue
consumer is only a special case.)
3. It is not important just to execute operations
in the right order (ADD and then DELETE). Even more important is _not to
execute superfluous operations_ at all: Do not add something if it should
be deleted right after that!
4. We want to buffer user’s actions for at
least a _couple of minutes_.
5. Some people reading this discussion may ask
themselves “How important is this use case at all?”, so let’s
rate it:
·
Occurrence:
very rare
i.
This
issue _does not_ occur when crawling some data source. (Crawling is the
most common use case.)
ii.
This
issue only occurs if the data source has been monitored by an agent _and_
the user is doing “ADD” (or “UPDATE”) and subsequently
almost instantly an “DELETE” operation on the document (data set
that is represented later as a record in SMILA). The chances that this happens
are very low.
·
Awareness:
quite high
i.
We have
been aware of this issue for more than a year. We discussed it and (after a
short analysis) agreed that it should be addressed in the connectivity module
by a component called “Buffer”.
·
Relevance:
low
i.
Since
this issue occurs very rare, it can be generally rated as “low”.
Can we agree on these
statements?
If not, please
correct me or add what’s being missing.
Cheers
Igor
Von: smila-dev-bounces@xxxxxxxxxxx
[mailto:smila-dev-bounces@xxxxxxxxxxx] Im Auftrag von Thomas Menzel
Gesendet: Freitag, 2. Oktober 2009 22:48
An: Smila project developer mailing list
Betreff: [smila-dev] RE: FYI :: new feature :: Message Resequencer
my
work is done on the wiki page, and i "saw that it was good." ;)
From:
smila-dev-bounces@xxxxxxxxxxx [smila-dev-bounces@xxxxxxxxxxx] On Behalf Of
Thomas Menzel [tmenzel@xxxxxxx]
Sent: Friday, October 02, 2009 4:30 PM
To: Smila project developer mailing list
Subject: [smila-dev] RE: FYI :: new feature :: Message Resequencer
i'm updating the concpet page b/c i heard you wanted to look at
either today or on monday.
From:
smila-dev-bounces@xxxxxxxxxxx [smila-dev-bounces@xxxxxxxxxxx] On Behalf Of
Thomas Menzel [tmenzel@xxxxxxx]
Sent: Monday, September 21, 2009 2:00 PM
To: Smila project developer mailing list
Subject: [smila-dev] FYI :: new feature :: Message Resequencer
hi folks,
just wanted to announce and inform you that
I will be working on the problem that messages don’t get out of sync when
there are changes in close succession
this change will be tracked thru the bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=289995
Kind
regards
Thomas Menzel @ brox IT-Solutions GmbH
No
virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.14.3/2415 - Release Date: 10/05/09
06:19:00