Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[smila-dev] WG: [smila-user] SMILA JDBC-crawling

Hi,

 

could please someone with some knowledge of the JDBCCrawler take care of this ?

 

Thanks!

 

Bye,

Daniel

 

 

Von: smila-user-bounces@xxxxxxxxxxx [mailto:smila-user-bounces@xxxxxxxxxxx] Im Auftrag von Andreas.Schultz@xxxxxxxxxxx
Gesendet: Donnerstag, 7. Januar 2010 10:08
An: smila-user@xxxxxxxxxxx
Betreff: [smila-user] SMILA JDBC-crawling

 

Hi all,

 

todays problem could be described as following:

 

The crawling of a MySQL-DB delivers 4398 rows but only 218 of them are inserted into the Lucene-index.

No relevant errors occur within the log-file, even not in debug-mode.

I added the crawling configuration and the INFO-logfile to this mail.

 

Thanks for your help

 

Andreas Schultz
Senior Software Developer

- - - - Bitte beachten Sie meine neuen Kontaktdaten - - - -


Empolis GmbH  |  Meisenstr. 90 | 33607 Bielefeld  |  Germany
AN ATTENSITY GROUP COMPANY
Phone +49 (0)521 55 785 413|  Fax +49 (0)521 55 785 121
andreas.schultz@xxxxxxxxxxx

 

www.empolis.com
Sitz Kaiserslautern  |  Amtsgericht Kaiserslautern HRB 30711  |  Geschäftsführer: Dr. Stefan Wess, Dr. Peter Tepassé

 

………………………………………………………………………………………………………………………………………………………………………………………………………..

Know. Right. Now.

Das ist unsere Philosophie. Empolis, an Attensity Group Company, bietet eine integrierte Suite von Geschäftsanwendungen,

die mit Hilfe patentierter semantischer Informations-Technologien die exponentiell wachsende Menge unstrukturierter
Daten analysiert, interpretiert und automatisiert verarbeitet. Entscheider, Experten, Mitarbeiter und Kunden erhalten so
stets situations- und aufgabengerecht genau das Wissen, das für ihre Arbeit relevant ist.

………………………………………………………………………………………………………………………………………………………………………………………………………..

Abonnieren Sie unseren monatlichen Newsletter: http://www.empolis.de/newsletter.html

 

<?xml version="1.0" encoding="UTF-8"?>
<!--
/***********************************************************************************************************************
 * Copyright (c) 2008 empolis GmbH and brox IT Solutions GmbH. All rights reserved. This program and the accompanying
 * materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution,
 * and is available at http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors: Ivan Churkin (brox IT Solutions GmbH) - initial creator
 **********************************************************************************************************************/
-->
<DataSourceConnectionConfig
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
  xsi:noNamespaceSchemaLocation="../org.eclipse.smila.connectivity.framework.crawler.jdbc/schemas/JdbcDataSourceConnectionConfigSchema.xsd">
	<DataSourceID>kinkon_jdbc</DataSourceID>
	<SchemaID>org.eclipse.smila.connectivity.framework.crawler.jdbc</SchemaID>
	<DataConnectionID>
		<Crawler>JdbcCrawler</Crawler>
	</DataConnectionID>
	<!--CompoundHandling>No</CompoundHandling-->
	<!--DeltaIndexing>disabled</DeltaIndexing-->
	<DeltaIndexing>full</DeltaIndexing>
	<Attributes>
		<Attribute Name="Url" 
      HashAttribute="true"
			KeyAttribute="true" 
      Type="String">
			<ColumnName>URL</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="Leading_ID" 
      HashAttribute="false"
			KeyAttribute="false" 
      Type="String">
			<ColumnName>Leading_ID</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="Source" 
      HashAttribute="false"
			KeyAttribute="false" 
      Type="String">
			<ColumnName>Source</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="Category" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>Category</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="Content" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>Content</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="View_1" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>View_1</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="View_2" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>View_2</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="View_3" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>View_3</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="View_4" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>View_4</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
		<Attribute Name="UserId" 
			HashAttribute="false" 
			KeyAttribute="false"
			Type="String">
			<ColumnName>UserId</ColumnName>
			<SqlType>string</SqlType>
		</Attribute>
	</Attributes>
	<Process>
		<Selections>
			<!--
			<Grouping>
					<Stepping>20</Stepping>
					<SQL>SELECT CONVERT(varchar, [nMessageID]) as Leading_ID FROM [KinkonGmbHTestDB].[dbo].[SMILA_NewsView] ORDER BY Leading_ID ASC</SQL>
			</Grouping>
			-->
			<SQL>
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_bookmarks) 	
				UNION ALL
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_contacts)	  							
				UNION ALL
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_my_contacts)	  							
				UNION ALL
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_messages)
				UNION ALL
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_groups) 	
				UNION ALL
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_forums)
				UNION ALL
				(SELECT URL, Leading_ID as Leading_ID, Source, Category, Content, View_1, View_2, View_3, View_4, UserId FROM smila_forum_group)
			</SQL>
		</Selections>
		<Database Connection="jdbc:mysql://localhost:3306/kinkon"
			FetchSize="100000"
			User="kinkon"
			Password="kinkon" 
			JdbcDriver="com.mysql.jdbc.Driver" />		
	</Process>
</DataSourceConnectionConfig>

Attachment: SMILA_JConsole.jpg
Description: SMILA_JConsole.jpg

Attachment: SMILA.zip
Description: SMILA.zip

_______________________________________________
smila-user mailing list
smila-user@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-user

Back to the top