|RE: [smila-dev] RE: FYI :: new feature :: Message Resequencer|
Here are my answers:
> This issue _does not_ occur when crawling some data source.
there might be rare cases where it could occur there (links to the same resource, e.g. the same document referenced from 2 websites)
> (Crawling is the most common use case.)
not sure if crawling really is the most common case. In the past I usually integrated our former product more in the agent style
> This issue only occurs if the data source has been monitored by an agent _and_ the user is doing “ADD” (or “UPDATE”) and subsequently almost instantly an “DELETE” operation on the document (data set that is represented later as a record in SMILA).
a) doesn’t only affect ADD/DEL ops but also ADD/ADD as pointed out
b) > Instantly
a. Highly depends on the setup
b. generalized: as long as the change on the same resource occurs within the time period a previous change event is being processed.
> The chances that this happens are very low.
Highly depends on the setup and where u get the data from and the frequency that this data changes.
> agreed that it should be addressed in the connectivity module by a component called “Buffer”.
As I pointed out, this solution is
a) not safe
b) might not meet the application needs/requirements
> Since this issue occurs very rare, it can be generally rated as “low”.
Well, with the assumptions you have made, yes. If those assumptions fail: it is not low IMO. As I said: It depends on the use case. Here are some (think agent):
· a wiki that I used by many users concurrently:
o Here it can happen fairly frequently that the same page is saved twice in fast succession. At least it happens to me that after saving I notice a typo or add a quick note, resulting on another "save".
· web application
o in order to ease the DB load, the search is the primary means to access the data, especially those where a complex SQL query would be crafted.
o To have always an accurate result, a minimum time diff. between resource change and index update is required.
Fist, sorry for not reacting any sooner to your mails. I was “on the road” last week, which means that I was offline almost all of the time L
As suggested by you (in our phone call today), I won’t be answering to any of your mails that you wrote last week. Instead, I refer here to the latest version of the wiki page (http://wiki.eclipse.org/SMILA/Specifications/ProcessingMessageResequencer) that you created.
Before we go deeper into the technical discussion, I would like to clear some things first:
1. We are actually talking about the use case/issue which can be shortly defined as: “The execution order of operations on _one_ particular record _does_ matter.”
2. We want to be highly scalable. This implies that in general we always have more than one queue consumer. (Single queue consumer is only a special case.)
3. It is not important just to execute operations in the right order (ADD and then DELETE). Even more important is _not to execute superfluous operations_ at all: Do not add something if it should be deleted right after that!
4. We want to buffer user’s actions for at least a _couple of minutes_.
5. Some people reading this discussion may ask themselves “How important is this use case at all?”, so let’s rate it:
· Occurrence: very rare
i. This issue _does not_ occur when crawling some data source. (Crawling is the most common use case.)
ii. This issue only occurs if the data source has been monitored by an agent _and_ the user is doing “ADD” (or “UPDATE”) and subsequently almost instantly an “DELETE” operation on the document (data set that is represented later as a record in SMILA). The chances that this happens are very low.
· Awareness: quite high
i. We have been aware of this issue for more than a year. We discussed it and (after a short analysis) agreed that it should be addressed in the connectivity module by a component called “Buffer”.
· Relevance: low
i. Since this issue occurs very rare, it can be generally rated as “low”.
Can we agree on these statements?
If not, please correct me or add what’s being missing.
virus found in this incoming message.