| Re: [smila-user] Bucket size on SMILA |
Hi,
besides the "bulkLimitTime" parameter the bulkbuilder worker has another parameter "bulkLimitSize". Maybe you have to increase its default value.
(see http://wiki.eclipse.org/SMILA/Documentation/Bulkbuilder)
If this doesn't help, try to increase the 10 (seconds) of the "bulkLimitTime" to make sure that this doesn't cause your problem.
You can also have a look at the job monitoring API to see which worker processed how many records and how many bulks.
(Go to http://localhost:8080/smila/jobmanager/jobs and click on your job run(s) to see the details. Have a look at the input/ouput "recordCount" and "dataObjectCount" (= bulks) for each worker)
If the problem still remains, please send the job run details for the crawlFilesystemPortraits and the PortraitProcessing job.
Regards,
Andreas
Von: smila-user-bounces@xxxxxxxxxxx [mailto:smila-user-bounces@xxxxxxxxxxx] Im Auftrag von Chiara Pasini
Gesendet: Dienstag, 14. Mai 2013 15:05
An: Smila project user mailing list
Betreff: [smila-user] Bucket size on SMILA
_______________________________________________
Hi guys,
we're having some trouble with our SMILA pipeline...
The workflow (PortraitProcessing) reads the output of the FileCrawler as defined in attachment.
We would like to maximise the bucket size in input so we used the parameters:
"parameters": {
"pipelineRunBulkSize": "20" },
"parameters": {
"bulkLimitTime": "10" }
as we did for an older pipeline.
Unfortunately even if the folder to crawl contains only 4 files, the pipeline receives (almost simultaneously) 2 separate buckets (2 files each).
Is there a way to force smila to build only one bucket with all the files in the directory?
Thanks,
C
smila-user mailing list
smila-user@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-user