Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[smila-user] performance degredation with the new processing

Hi folks,

 

I have done a little perf. test with the new processing and solr and it seems to be slower than before (from 31 mins to 50 min for a subset of the german Wikipedia) .

 

It seems that scale ain’t doing the trick as it is supposed to (on one machine only) – or, very likely,  I don’t know how to config it.

 

The setup is as follows:

Just one box with quad core and 4GB ram.

I used our (brox) standard AddPipeline that does some aperture like conversion of docs (not really needed in this case but always good to run test including this) and then puts it into the solr index, which is configured the same as in smila (except that I had to switch the date field to be string instead of date due to our open bug). So I pretty much  used smila’s default setup as described in 5min to success.

 

The rest of the config is the same as it was before the processing change except with regard to the Q worker etc not present anymore and the associated mandatory changes.

 

Now, maxScaleUp  was for the 1st run  4 and the 2nd  6. The 2nd run was even 2 mins slower, although that can be neglected and could be due to having started it with crawlW!?

 

At the same time, CPU utilization was rather low, e.g. only 30-40%.

 

Any hints? Or does the new processing incure just more overhead but pays off when u also scale wide?

 

Thomas Menzel @ brox IT-Solutions GmbH

 



Taglocity Tags: smila

Back to the top