[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [smila-user] performance degredation with the new processing
|
Hi,
Ad 1: Yes it's quite possible that the Bulkbuilder/ObjectStore combination has some ... uhm ...
potential for optimization. For example, I think that increasing the crawlers buffer size will not
change much, because the Bulkbuilder appends the records one by one. Maybe I can have a look at this
tomorrow.
Yes, an in-Memory solution would quite certainly improve performance. And the whole purpose of
separating SMILA into independent (OSGi) services is to make it easy to exchange service
implementations. Just do a new service implementation and put it in config.ini instead of
o.e.s.objectstore.filesystem and you should be done.
Ad 2: The scaleUpLimits for pipelineProcessor and pipelineProcessor are OK. The first one for the
"_finishingTasks" is not a global one, but one for a "system worker" and it's OK that it's 1. You'll
find the global scaleUp limit for the node at the end of /smila/tasks. The clusterconfig.json looks
OK to me, too. But if the tasks are created to slowly, scaleUp cannot help anyway.
/smila/debug is currently not documented, it's kind of an "experimental and inofficial sandbox API"
anyway and may change often. If parts of it prove to be very important for monitoring, we should
rather move it to an "official" URLs instead of documenting /smila/debug (;
Cheers,
Juergen
