Summary: | [Workspace Management] Use multiple threads for loading the files | ||
---|---|---|---|
Product: | [Automotive] Sphinx | Reporter: | Robert Kiss <robert.kiss> |
Component: | Core | Assignee: | Project Inbox <sphinx-inbox> |
Status: | CLOSED WONTFIX | QA Contact: | |
Severity: | enhancement | ||
Priority: | P3 | CC: | balazs.grill, chetankumar |
Version: | 0.7.0 | ||
Target Milestone: | --- | ||
Hardware: | All | ||
OS: | All | ||
Whiteboard: |
Description
Robert Kiss
2011-06-17 04:27:48 EDT
Clearly a good idea. I'm however wondering based on which criteria we should decide how many jobs we should create when being face to a given number of files. Once the know that we also need to consider the file sizes so as to make sure that a first job loads 100 little files and a second hundred big files. It's not necessary to split the initial pool of files from beginning. We can, for example, create a pool of daemon threads that listen for some notification. When such a notification is received, it will take one request from a queue, execute it, store the response into another queue.... you got the idea. The initial number of threads can be derived from the number of available CPU's Does parallel file load also include proxy resolution that happens after load ? I don't think it can. A resolved proxy could refer to an object that is not yet loaded. However, it can initialize the index for the loaded resource. That operation is also time consuming and can be executed in parallel. See ModelLoadManager.forceProxyResolution; the call to lookupResolver.init(resources) I might have time to work on this in the July/August. Closed stale issue before migration |