Re: [science-iwg] org.eclipse.dawnsci.dataset.IDataset Re: science-iwg D

[Jay Jay Billings <jayjaybillings@xxxxxxxxx>]

I should clarify that the IDataProvider that I mentioned is the one from ICE, not the one from SWT XYGraph.

Jay

On Wed, Aug 27, 2014 at 9:26 AM, Kasemir, Kay <kasemirk@xxxxxxxx> wrote:

Hi:

Does your org.eclipse.dawnsci.dataset.IDataset support non-array data?

I suppose it does since I think you want to support lazy or sparse data, where one can provide

    int size();                              // number of samples
    Sample getSample(int i); // get one of the samples

but you can't necessarily support

    Sample[] getSamples();
or
   List<Sample> getSamples(); // Similar because of List.toArray()

since you might not have all the data in memory at the same time.


My concern is dealing with live control system data, which I may internally store as

MyDataForOneControlSystemChannel:
        List<Sample>              history;     // N1 historic samples
        RingBuffer<Sample> live_data; // N2 samples that have

Fetching the history is slow and is only done when the plot is started, then updated at long periods.
Live data can arrive at any time and is added to the ring buffer. To allow running "forever" without running out of memory, it's a limited length ring buffer.

At any time, I know that I have N1+N2 samples, and I can provide any single sample[i] for 0<= i < (N1+N2).
I want to update the plot say once per second, but I don't want to copy all samples into a plain array once per second.
So similar to a lazy data provider, I can implement

    int size()
    {  return history.size() + live_data.size(); }

    Sample getSample(int i)
    {
      int end_of_history = history.size();
      return i < end_of_history ? history.getSample(i) : live_data.getSample(i - end_of_history);
    }

but I would like to avoid implementing anything that needs to return a complete array.


Another concern is locking:
When you deal with live data, the data set isn't static.
It starts out empty.
Then the first live sample is added.
Then the next live sample is added.
Then you may finally receive the historic data, which took longer to fetch.
Now a 3rd live sample arrives.

With the XYGraph, this was supported by agreeing to always lock on the IDataProvider.
It "works", but it is very hard to enforce. The  XYGraph itself locks on the IDataProvider, but if a user doesn't read the Javadoc for org.csstudio.swt.xygraph.dataprovider.IDataProvider and updates the data without locking, the xygraph could access inconsistent values.

So overall, would the new org.eclipse.dawnsci.dataset.IDataset support concurrent access to lazy/sparse/segmented data?

Thanks,
Kay

On Aug 27, 2014, at 7:11 AM, <science-iwg-request@xxxxxxxxxxx>
 wrote:
> ..
> From: Gerring, Matt (DLSLtd,RAL,LSCI)
> Sent: 27 August 2014 09:49
> To: 'Jay Jay Billings'
> Subject: RE: uk.ac.diamond.scisoft.analysis.dataset.IDataset -> org.eclipse.dawnsci.dataset.IDataset
> ..
> Firstly using a DoubleDataset. It has a constructor which takes the double[] and the shape. If 1D data, the shape is simply the length. Peter Chang<mailto:peter.chang@xxxxxxxxxxxxx> is the architect of the data system and can also advise you of more complex data needs. Mostly people get IDatasets from the LoaderService as in the examples but instantiating the concrete class is allowed.
> ..

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

--

Jay Jay Billings

Oak Ridge National Laboratory

Twitter Handle: @jayjaybillings