Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
AW: [smila-dev] Re: Problems with BinStorage

Hi all,

It seems to me that not all of us are very clear what the major requirements that we have posed on our framework are.
So I thought this is a good occasion to move (and actualize) SMILA requirements from confluence to http://wiki.eclipse.org/SMILA/General_Framework_Requirements ;-)

Beside that, I've also wrote down some component requirements (http://wiki.eclipse.org/SMILA/Component_Requirements) that I've talk about with Georg last Friday.
Please read them carefully and keep them in mind while designing and implementing components.

BTW: I will also create a wiki page dedicated to Store(s) today.

Regards
Igor


> -----Ursprüngliche Nachricht-----
> Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] Im
> Auftrag von Marius Cimpean
> Gesendet: Freitag, 10. Oktober 2008 17:42
> An: Smila project developer mailing list
> Betreff: Re: [smila-dev] Re: Problems with BinStorage
> 
> Hi all
> 
> I would kindly ask you to have a look over the above email and let me know
> your suggestions, in order to be able "build the part of the requirements"
> for binstorage.
> 
> I'm most interesting in knowing if the binstorage shall be hierarchically in
> file system OR the binary objects could be saved in other way (by using an
> existing API) - like xmlstorage, lucene index.
> 
> If no suggestion or no idea about the perfect requirements ... than ... I
> suggest implement both solutions so binary storage will be smart enough to
> use (depending on some config) hierarchically structure in file system (by
> using commons-vfs ... or ) and some existing persistence API. The business
> of binstorage will not depend on the "DAO - abstract interf." implementation
> of the persistence.
> 
> Thanks.
> 
> Best Regards,
> Marius
> 
> ----- Original Message -----
> From: "Marius Cimpean" <marius.cimpean@xxxxxxxxxxx>
> To: "Smila project developer mailing list" <smila-dev@xxxxxxxxxxx>
> Sent: Wednesday, October 08, 2008 4:18 PM
> Subject: Re: [smila-dev] Re: Problems with BinStorage
> 
> 
> > Hi Daniel,
> >
> > - the BinStorage should be totally independent of the BlackboardService as
> > well as Record and Record IDs (you already use String as key)
> >>>> The Binary storage is totally independent of BlackboardService from the
> >>>> design point of view (on a imaginary sequence diagram the binary
> >>>> storage appears at the right of the blackboard service. blackboard
> >>>> being the client/caller). The thing is that binary storage uses as key,
> >>>> the passed string from blackboard service (passed string = calculated
> >>>> key <attachment id> from blackboard service which is based on record Id
> >>>> and file name - in order to provide unique keys). This unique key
> >>>> (received from blackboard) is used inside of binary storage service, in
> >>>> order to unique identify the persisted folder/files.
> > So, in case of the blackboard service is the only client of binary
> > storage, the passed key ensures unique & identifying of persisted
> > files....otherwise the binary storage shall provide the unique/identifier
> > mechanism
> >
> > - don't stick to much to the concept of a "file", BinStorage is used to
> > store large (mostly binary) data
> >
> > - I suggest shorter method names like "load", "store" and "delete"
> >>>> Ok. The names were just inherited from current implementation
> >
> > - I also suggest to use Streams instead of byte[] for store and load
> >>>> Ok. This is also inherited ...
> >
> > - it is not required to have a 1:1 mapping of attachment to file/folder in
> > the filesystem. In fact, I think that this approach is not feasible for
> > high volume of data, even with a tree structure. So perhaps there are
> > other techniques needed on top of final storage in the filesystem. But
> > maybe this can be done in a more advanced implementation.
> >>>> yes, I totally agree with you, related the current implementation - 1:1
> >>>> mapping in flat structure which must be changed. Anyway, at the moment
> >>>> there are obviously two important aspects that are very important for
> >>>> near future of Binary storage:
> > 1. "Virtual File System vs. other data object persisting solutions" - I
> > believe at the moment is very important to know all specific requirements
> > (like: "the need to offer hierarchical storage to clients", "there is the
> > requirement (from georg) to partition the storage via config", clusterize
> > the physical storage ) which could make the difference in taking the right
> > decision. Also, aspects like hierarchically structure for clients and
> > clustering are very important in choosing the appropriate framework.
> > So, I would kindly ask for any available requirements (if any :)  )
> > related to Binary storage ... that make subject of this point.
> >
> > Existing APIs:
> > http://java-source.net/open-source/database-engines
> > http://java-source.net/open-source/persistence
> > and others
> >
> > 2. "Hierarchical structure vs. Flat structure" - we already have an
> > answer, in case at the previously point the VFS is preferred. So, in case
> > the binary storage shall base on virtual file system then the current
> > persistence implementation must be changed in order to persist data in
> > tree/hierarchical structure (for performance impact). I am currently
> > developing a concept of hierarchical/configurable binary storage in file
> > system ... I will post this concept on the wiki page right after I finish
> > it and if the VFS is the adopted solution.
> >
> > Best Regards,
> > Marius
> >
> > ----- Original Message -----
> > From: <Daniel.Stucky@xxxxxxxxxxx>
> > To: <smila-dev@xxxxxxxxxxx>
> > Sent: Wednesday, October 08, 2008 2:54 PM
> > Subject: AW: [smila-dev] Re: Problems with BinStorage
> >
> >
> > Hi Marius,
> > some remarks to the wiki page:
> >
> > - the BinStorage should be totally independent of the BlackboardService as
> > well as Record and Record IDs (you already use String as key)
> > - don't stick to much to the concept of a "file", BinStorage is used to
> > store large (mostly binary) data
> > - I suggest shorter method names like "load", "store" and "delete"
> > - I also suggest to use Streams instead of byte[] for store and load
> > - it is not required to have a 1:1 mapping of attachment to file/folder in
> > the filesystem. In fact, I think that this approach is not feasible for
> > high volume of data, even with a tree structure. So perhaps there are
> > other techniques needed on top of final storage in the filesystem. But
> > maybe this can be done in a more advanced implementation.
> >
> >
> > Bye,
> > Daniel
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> >> bounces@xxxxxxxxxxx] Im Auftrag von Marius Cimpean
> >> Gesendet: Mittwoch, 8. Oktober 2008 00:03
> >> An: smila-dev@xxxxxxxxxxx
> >> Betreff: [smila-dev] Re: Problems with BinStorage
> >>
> >> Hi all
> >>
> >> 1. I completely restructured the binstorage bundles (not in the SVN
> >> yet) -
> >> from the design&architecture point of view (the persistence logic is
> >> still
> >> the same - flat fs). My intention was, first to restructure the
> >> binstorage
> >> bundles (done) and then improve the persistence mechanism (backend -
> >> persistence logic) - based on the discussion list. The reported test
> >> case by
> >> Daniel (huge amount of data) sounds like "it is a must" related to
> >> changing
> >> the persistence mechanism, which currently was implemented (as
> >> described by
> >> Daniel in flat fs way).
> >> 2. Wiki page for binstorage
> >> http://wiki.eclipse.org/SMILA/Project_Concepts/Binary_Storage
> >>
> >> Best regards,
> >> Marius
> >>
> > _______________________________________________
> > smila-dev mailing list
> > smila-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/smila-dev
> >
> >
> >
> > _______________________________________________
> > smila-dev mailing list
> > smila-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/smila-dev
> >
> >
> 
> 
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev


Back to the top