Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [smila-dev] Re: Problems with BinStorage

Hi Daniel,

- the BinStorage should be totally independent of the BlackboardService as well as Record and Record IDs (you already use String as key)
The Binary storage is totally independent of BlackboardService from the design point of view (on a imaginary sequence diagram the binary storage appears at the right of the blackboard service. blackboard being the client/caller). The thing is that binary storage uses as key, the passed string from blackboard service (passed string = calculated key <attachment id> from blackboard service which is based on record Id and file name - in order to provide unique keys). This unique key (received from blackboard) is used inside of binary storage service, in order to unique identify the persisted folder/files.
So, in case of the blackboard service is the only client of binary storage, the passed key ensures unique & identifying of persisted files....otherwise the binary storage shall provide the unique/identifier mechanism

- don't stick to much to the concept of a "file", BinStorage is used to store large (mostly binary) data

- I suggest shorter method names like "load", "store" and "delete"
Ok. The names were just inherited from current implementation

- I also suggest to use Streams instead of byte[] for store and load
Ok. This is also inherited ...

- it is not required to have a 1:1 mapping of attachment to file/folder in the filesystem. In fact, I think that this approach is not feasible for high volume of data, even with a tree structure. So perhaps there are other techniques needed on top of final storage in the filesystem. But maybe this can be done in a more advanced implementation.
yes, I totally agree with you, related the current implementation - 1:1 mapping in flat structure which must be changed. Anyway, at the moment there are obviously two important aspects that are very important for near future of Binary storage:
1. "Virtual File System vs. other data object persisting solutions" - I believe at the moment is very important to know all specific requirements (like: "the need to offer hierarchical storage to clients", "there is the requirement (from georg) to partition the storage via config", clusterize the physical storage ) which could make the difference in taking the right decision. Also, aspects like hierarchically structure for clients and clustering are very important in choosing the appropriate framework. So, I would kindly ask for any available requirements (if any :) ) related to Binary storage ... that make subject of this point.

Existing APIs:
http://java-source.net/open-source/database-engines
http://java-source.net/open-source/persistence
and others

2. "Hierarchical structure vs. Flat structure" - we already have an answer, in case at the previously point the VFS is preferred. So, in case the binary storage shall base on virtual file system then the current persistence implementation must be changed in order to persist data in tree/hierarchical structure (for performance impact). I am currently developing a concept of hierarchical/configurable binary storage in file system ... I will post this concept on the wiki page right after I finish it and if the VFS is the adopted solution.

Best Regards,
Marius

----- Original Message ----- From: <Daniel.Stucky@xxxxxxxxxxx>
To: <smila-dev@xxxxxxxxxxx>
Sent: Wednesday, October 08, 2008 2:54 PM
Subject: AW: [smila-dev] Re: Problems with BinStorage


Hi Marius,
some remarks to the wiki page:

- the BinStorage should be totally independent of the BlackboardService as well as Record and Record IDs (you already use String as key) - don't stick to much to the concept of a "file", BinStorage is used to store large (mostly binary) data
- I suggest shorter method names like "load", "store" and "delete"
- I also suggest to use Streams instead of byte[] for store and load
- it is not required to have a 1:1 mapping of attachment to file/folder in the filesystem. In fact, I think that this approach is not feasible for high volume of data, even with a tree structure. So perhaps there are other techniques needed on top of final storage in the filesystem. But maybe this can be done in a more advanced implementation.


Bye,
Daniel

-----Ursprüngliche Nachricht-----
Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
bounces@xxxxxxxxxxx] Im Auftrag von Marius Cimpean
Gesendet: Mittwoch, 8. Oktober 2008 00:03
An: smila-dev@xxxxxxxxxxx
Betreff: [smila-dev] Re: Problems with BinStorage

Hi all

1. I completely restructured the binstorage bundles (not in the SVN
yet) -
from the design&architecture point of view (the persistence logic is
still
the same - flat fs). My intention was, first to restructure the
binstorage
bundles (done) and then improve the persistence mechanism (backend -
persistence logic) - based on the discussion list. The reported test
case by
Daniel (huge amount of data) sounds like "it is a must" related to
changing
the persistence mechanism, which currently was implemented (as
described by
Daniel in flat fs way).
2. Wiki page for binstorage
http://wiki.eclipse.org/SMILA/Project_Concepts/Binary_Storage

Best regards,
Marius

_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev





Back to the top