| RE: [smila-user] RE: JDBC-Crawling Phenomenon |
|
on a side note: i dont like pushing
all kinds of records into the same Q. I always would open up an own Q for each
kind of record and processing state to keep things separated that way changes
wont mess up other parts so easily. I really think we should
make this more known to the community and establish that as a good practice. on
the other hand: things will become more complicated in regard to config and
setup. Kind regards Thomas Menzel @ brox IT-Solutions GmbH From: smila-user-bounces@xxxxxxxxxxx
[mailto:smila-user-bounces@xxxxxxxxxxx] On Behalf Of Thomas Menzel hi anderas, yes, but condition
for AddPipeline is: operation == ADD &&
datasource not like (%feeds% or %xmldump%). that condition is
true for ur kinkon cases on an ADD op!! Mit freundlichen Grüßen / Kind regards Thomas Menzel brox IT-Solutions
GmbH ================================== From: smila-user-bounces@xxxxxxxxxxx
[mailto:smila-user-bounces@xxxxxxxxxxx] On Behalf Of Andreas.Schultz@xxxxxxxxxxx Hi Thomas, obviously one of us
still misunderstands either the problem or the hints. The condition which
decides which Rule à Pipeline will be
chosen is defined within the QueueWorkerListenerConfig -config: <Rule
Name="ADD JDBC Rule" WaitMessageTimeout="10"
Threads="4" MaxMessageBlockSize="20">
<Source BrokerId="broker1"
Queue="SMILA.connectivity"/>
<Condition>Operation='ADD' and DataSourceID LIKE
'%kinkon%'</Condition>
<Task>
<Process Workflow="KinKonAddPipeline"/>
</Task> </Rule> So in this case, all
DataSourceIDs containing %kinkon% should be routed through a specific
KinKonAddPipeline. But the same data (an
identical set of data!; same DS, same DB, same whatever!) sometimes is
routed through <Rule
Name="ADD Rule" WaitMessageTimeout="10" Threads="4"
MaxMessageBlockSize="20">
<Source BrokerId="broker1"
Queue="SMILA.connectivity"/>
<Condition>Operation='ADD' and NOT(DataSourceID LIKE '%feeds%') and
NOT(DataSourceID LIKE '%xmldump%')</Condition>
<Task>
<Process Workflow="AddPipeline"/>
</Task> </Rule> But: They are never
routed through both of them! Best Andreas Schultz - - - - Bitte
beachten Sie meine neuen Kontaktdaten - - - -
www.empolis.com ……………………………………………………………………………………………………………………………………………………………………………………………………….. Know. Right.
Now. Das ist unsere Philosophie.
Empolis, an Attensity Group Company, bietet eine integrierte Suite von
Geschäftsanwendungen, die mit Hilfe
patentierter semantischer Informations-Technologien die exponentiell wachsende
Menge unstrukturierter ……………………………………………………………………………………………………………………………………………………………………………………………………….. Abonnieren Sie
unseren monatlichen Newsletter: http://www.empolis.de/newsletter.html
Von:
smila-user-bounces@xxxxxxxxxxx [mailto:smila-user-bounces@xxxxxxxxxxx] Im
Auftrag von Thomas Menzel hi, > what I meant
was, that identical data goes through the [ADD Rule] sometimes and through the
[ADD JDBC Rule] sometimes. > And there is no
obvious rule when which rule is chosen. That’s the problem. exactly! u need to - mark ur records
distinctly so there is a condition that only one rule will select them and not
the other OR - put them into diff.
Qs and have the listeners listen on their respective Qs. Kind regards Thomas Menzel @ brox IT-Solutions GmbH From: smila-user-bounces@xxxxxxxxxxx
[mailto:smila-user-bounces@xxxxxxxxxxx] On Behalf Of Andreas.Schultz@xxxxxxxxxxx Hi Thomas, what I meant was,
that identical data goes through the [ADD Rule] sometimes and through the [ADD
JDBC Rule] sometimes. And there is no obvious
rule when which rule is chosen. That’s the problem. At 2009-09-29
15:40:34,799: - Record is routed
with rule [Default Route Rule] and operation [null], record
id=177c250f8e116110396aaa5b1dd51662d633f6517dab42801d98be7f1765f6
- Closing JdbcCrawler...
- Unregistering
crawling thread
kinkon_bookmark_jdbc
- Crawling thread
kinkon_bookmark_jdbc
unregistered
- Crawling thread
kinkon_bookmark_jdbc
stopped.
- Record is processed
by Listener with rule: [ADD Rule] and operation [ADD], record
id=177c250f8e116110396aaa5b1dd51662d633f6517dab42801d98be7f1765f6 At 2009-09-29
15:40:58,391: Record is routed with
rule [Default Route Rule] and operation [null], record
id=177c250f8e116110396aaa5b1dd51662d633f6517dab42801d98be7f1765f6
Closing
JdbcCrawler...
Record is processed
by Listener with rule: [ADD JDBC Rule] and operation [ADD], record
id=177c250f8e116110396aaa5b1dd51662d633f6517dab42801d98be7f1765f6 As you may have
recognized, there are about 15 sec. between the operations. As I mentioned, I
put exactly the same data (a single set) into the process. I tried it several
times afterwards to get a glimpse of an rule of it, but it reacts totally
heuristic. Always the same data! Best Andreas Schultz - - - - Bitte
beachten Sie meine neuen Kontaktdaten - - - -
www.empolis.com ……………………………………………………………………………………………………………………………………………………………………………………………………….. Know. Right.
Now. Das ist unsere
Philosophie. Empolis, an Attensity Group Company, bietet eine integrierte Suite
von Geschäftsanwendungen, die mit Hilfe patentierter
semantischer Informations-Technologien die exponentiell wachsende Menge
unstrukturierter ……………………………………………………………………………………………………………………………………………………………………………………………………….. Abonnieren Sie
unseren monatlichen Newsletter: http://www.empolis.de/newsletter.html
Von:
smila-user-bounces@xxxxxxxxxxx [mailto:smila-user-bounces@xxxxxxxxxxx] Im
Auftrag von Thomas Menzel hi andreas, i'm not entirely sure
as what ur problem or error is that u see: > both listeners
take the record this not a bug it’s a
feature ;) both conditions fit,
so both can take on the records. on the concurrent system you cant tell which
gets what. > mimetype error ,
line 17 the default
addpipline invokes the MIME type detection service that needs a file extension
to do its work, which is contained in a field as defined in
config/../MimeTypeConfig.xml if the detection fails the rest of the processing is
skipped (see <if name="conditionIsText">… )
and hence nothing is added to the index since I guess u read
from the DB and u don’t need to detect mime type this can be ignored Kind regards Thomas Menzel @ brox IT-Solutions GmbH From: smila-user-bounces@xxxxxxxxxxx
[mailto:smila-user-bounces@xxxxxxxxxxx] On Behalf Of Andreas.Schultz@xxxxxxxxxxx Hi all, I have a really nice phenomenon using a
JDBC DS: After having succeeded to connect to the DB
(MSSQL with authorization via Windows-Domain) which was really hard work, I added an entry to the Listener-config to
call my pipeline: <Rule
Name="ADD JDBC Rule" WaitMessageTimeout="10"
Threads="4" MaxMessageBlockSize="20">
<Source BrokerId="broker1"
Queue="SMILA.connectivity"/>
<Condition>Operation='ADD' and DataSourceID LIKE
'%kinkon%'</Condition>
<Task>
<Process Workflow="KinKonAddPipeline"/>
</Task>
</Rule> <Rule
Name="ADD Rule" WaitMessageTimeout="10"
Threads="4" MaxMessageBlockSize="20">
<Source BrokerId="broker1"
Queue="SMILA.connectivity"/>
<Condition>Operation='ADD' and NOT(DataSourceID LIKE '%feeds%') and
NOT(DataSourceID LIKE '%xmldump%')</Condition>
<Task>
<Process Workflow="AddPipeline"/>
</Task>
</Rule> The new pipeline has been a striped down
copy of the normal addpipeline. Funny was the behavior of the
indexing-process: Sometimes it succeeded, sometimes not! If you look at the attached log-file, you
will discover 2 sections, first of failed to put the content to the index,
second succeeded! Obviously, the first one took its way
through the ADD Rule, “Record is processed by Listener with rule:
[ADD Rule]” The second one through the expected “Record is processed by Listener with rule:
[ADD JDBC Rule]” Is this a misuse/ misconfiguration of mine
or a bug? Best Andreas Schultz - - - - Bitte
beachten Sie meine neuen Kontaktdaten - - - -
www.empolis.com ……………………………………………………………………………………………………………………………………………………………………………………………………….. Know. Right.
Now. Das ist unsere
Philosophie. Empolis, an Attensity Group Company, bietet eine integrierte Suite
von Geschäftsanwendungen, die mit Hilfe
patentierter semantischer Informations-Technologien die exponentiell wachsende
Menge unstrukturierter ……………………………………………………………………………………………………………………………………………………………………………………………………….. Abonnieren Sie
unseren monatlichen Newsletter: http://www.empolis.de/newsletter.html
|