Community
Participate
Working Groups
Build Identifier: Large models made from hundreds of files each of them a couple of MB large take quite some time to load. However, loading a particular file does not depend on the loading of another file so we should try to harness the multiple cores that most of us have. This is definitely not a trivial task and could have a lot of implication but the multi cores are here to stay and each piece of software should try to benefit from it. Reproducible: Always
Clearly a good idea. I'm however wondering based on which criteria we should decide how many jobs we should create when being face to a given number of files. Once the know that we also need to consider the file sizes so as to make sure that a first job loads 100 little files and a second hundred big files.
It's not necessary to split the initial pool of files from beginning. We can, for example, create a pool of daemon threads that listen for some notification. When such a notification is received, it will take one request from a queue, execute it, store the response into another queue.... you got the idea. The initial number of threads can be derived from the number of available CPU's
Does parallel file load also include proxy resolution that happens after load ?
I don't think it can. A resolved proxy could refer to an object that is not yet loaded. However, it can initialize the index for the loaded resource. That operation is also time consuming and can be executed in parallel. See ModelLoadManager.forceProxyResolution; the call to lookupResolver.init(resources)
I might have time to work on this in the July/August.
Closed stale issue before migration
