Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [smila-user] Blackboard can't handle BIG attachments

Hi,

in general, the BinaryStorage can be used to keep the attachments of Blackboard records on disc instead of in-memory. 
However, in synchronous calls from the REST API we currently don't support this option, because we expected BIG data
to be processed via asynchronous workflows where you can enable this option for the PipelineProcessorWorker. And small
attachments in synchronous calls should be kept in-memory for better performance. So yes, for the moment you probably 
need to work around this yourself using tempfiles. Or you can patch the
org.eclipse.smila.processing.httphandler.PipelineProcessHandler: in method process() it says

  final Blackboard blackboard = _blackboardFactory.createTransientBlackboard();

You could change this to 

  final Blackboard blackboard = _blackboardFactory.createBlackboard(false, true);

to enable the use of BinaryStorage for all your calls. In this case you should remove all the attachments at the end of 
your pipeline to prevent that they are loaded in-memory when the engine tries to return the result record to the caller.

Having said that: Yes, the attachment handling is still a bit rudimentary and should much less rely on storing them in-
memory. We have some ideas about that, but I don't think we will get to implement that soon.

Regards,
Juergen.


Back to the top