Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [smila-dev] Re: Problems with BinStorage

Hi,

It's not really clear to me, what this discussion is about currently (API? backend?), so I'll try to sum up my point of view on the complete package, and hopefully we'll get some answers from it (-;

The purpose of BinStorage is to store millions (or billions) of (possibly large) binary objects and make them available to a relatively large number of record processors running distributed in a "cluster" of computers. Currently I do not know what kind of backend might be the best to match these requirements, I rather think that it should be exchangeable in order to be able to match different requirements: In one application it could be simply a shared file system, because performance is sufficient and administration is easy, in other applications distributed file systems might be necessary, or some database technology or whatever. I don't think that we can decide this now for good. Thus the API must be as generic as possible to be implementable on any kind of backend.
And the details of the storage backend must not be of relevance to a client: One blackboard just writes the attachments of a record to bin storage, another one retrieves it again. They should not have to care about where to put the objects in a hierarchy for good performance, it's the task of BinStorage to create such hierarchies internally if a flat storage is not sufficient.

On the other side, I'm not against having a BinStorage API that enables different kinds of clients to use different "namespaces" in BinStorage to separate their data. These namespaces could be hierarchical: a "file system" metaphor behind the API might be helpful for developers using BinStorage, because most users are accustomed to it, even if the actual storage has nothing to with a file system.  On the other hand this might enable the configurator/administrator of BinStorage to distribute partitions of data to different storage mediums (just dreaming now ;-) for better performance. But BinStorage must not rely only on structures provided by the clients, but must organize the stored data for optimal performance even if a client does not provide any structure at all.

Hopefully this has not increased the confusion even more (-; 

Yours,
Juergen.


Back to the top