RE: [smila-dev] binstorage redesign

[Thomas Menzel <tmenzel@xxxxxxx>]

1. yes, its use is to store binary objects
2. we use the VFS to as it provides all needed support to clusterize the physical storage
3. I think there will be the need to offer hierarchical storage to clients, even if BS doesn’t need them (right now)
4. there is the requirement (from georg) to partition the storage via config (but that hasn’t been fully defined yet)
5. automatically decide to partition data could be done but might prove difficult or less efficient. question is here: how does this affect performance?

lets wait for marius to think and reply to all these concerns.

Kind regards
Thomas Menzel @ brox IT-Solutions GmbH


-----Original Message-----
From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Dmitry Hazin
Sent: Dienstag, 30. September 2008 09:57
To: Smila project developer mailing list
Subject: Re: [smila-dev] binstorage redesign

hm, i thought that binstorage was aimed to be just storage for binary 
objects and not virtual filesystem; is there any requirement about 
virtual filesystem? Regarding performance aspect, binstorage should 
decide internally how to keep files most effectively and it should be 
able to organize them into hierarchy if needed/distribute in cluster etc.

Thomas Menzel wrote:
> well, my thinking is that we provide a virtual file system as a binary storage. 
>
> hence a client (not now, but it is thought of that the bin storage is an infrastructure item that may offer his service to other clients as well) may want/need to organize his data hierarchically, as if in normal file system.
>
> if that is not needed for now in BS then we can limit the impl. currently to just offer a hashset-style storage but the API should support hierarchical (ie tree) structures and an FS is the most natural IMO.
>
> we may also name the child just flat bin storage and later do a hierarchical bin storage if u like that better.
>
> there is also most likely a performance aspect to full flat vs. hierarchical, as the lookup will be faster in a tree structure. that is not noticeable now with few items but when u have millions it will. 
> also: when distributing the data in a cluster, nodes in a tree are a very welcome entity that can be mapped directly a hardware nodes.
>
> all these are details for the (near) future but need to taken into consideration.
>
> Kind regards
> Thomas Menzel @ brox IT-Solutions GmbH
>
>
> -----Original Message-----
> From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-bounces@xxxxxxxxxxx] On Behalf Of Ivan Churkin
> Sent: Dienstag, 30. September 2008 09:04
> To: Smila project developer mailing list
> Subject: Re: [smila-dev] binstorage redesign
>
> I completely agree with.
> Imho, binstorage API should be very simple. I can imagine that some 
> customer will want to use database for storing binary data.
> As I understand there is only one problem - some text miner engine have 
> to get real file access to extract data.
>
> --
> Ivan
>
>
>
> Dmitry Hazin wrote:
>   
>> Why do you think that that binstorage should mimic the file system? 
>> What is the reason for that? The simplest will be just manage files in 
>> binstorage by Id (ie by name), and binstorage should decide how to 
>> store files physically, in file system or using any other way. So I 
>> don't see any the reason why binstorage should mimic filesystem, 
>> please explain your point.
>> _______________________________________________
>> smila-dev mailing list
>> smila-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/smila-dev
>>     
>
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> smila-dev mailing list
> smila-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/smila-dev
>   
_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev