Re: [science-iwg] Eclipse Datasets for Science

[Jonah Graham <jonah@xxxxxxxxxxxxxxxx>]

Hi Ian,

Excellent, thanks for starting to review the code. Peter has already
addressed some of the technical issues and I'll add in my 2 cents.


> [...] but the
> other examples complained about the lack of an slf4j binding [2]
I put in a "hack" to workaround the slf4j bindings missing for
BasicExample. Since the idea of slf4j is that the integrator decides
on the binding I have left dependency out completely from the examples
and platform.

My feeling is that some of the error handling within datasets needs a
review, with items such as methods that are declared throw Exception
[1] or some inconsistencies about what is logged vs what is thrown
[2].

[1] org.eclipse.dataset.ILazyDataset.getSlice(IMonitor, int[], int[],
int[]) throws Exception, however the only exception ever actually
thrown appears to be when the IMonitor is cancelled.
[2] Most places raise a RuntimeException at the same time as they
log.error the same message. However there are some exceptions to this,
such as  org.eclipse.dataset.AbstractDataset.get1DIndex(int, int)
which logs the error, and then tries to correct for it and
org.eclipse.dataset.LazyDatasetBase.getMetadata() which catches an
Exception and logs it.

> >    - what's missing?
>
> * From the DAWN code, I think there is value in the IOperation
> framework being pulled over. The IOperation classes appear
> implementation agnostic - so it should be possible to add generic
> operations to this project for common use.
I agree that IOperation should be made accessible. Presumably it
should happen at the same time. At the moment my goal was to slim down
DAWNSci to only things that did not depend on DAWN. The reality is
that all the IOperation concrete classes are not part of DAWNSci yet,
so weren't really under my consideration for extracting. I do hope
someone has the time and money to extract it though, possibly into a
new project that depends on org.eclipse.dataset.

> >    - suggestions for improvements.
> * Deeper support for data change listeners will allow the UI to update
> when data changes, or for a chain of dependent datasets & operations
> that dynamically update on changes to precedent datatset.
I am with Peter on this, unfortunately I am not fully familiar with
the requirements thus far, but any concept of listeners inside the
data objects gives me concern. Perhaps there is a need for a unified
layer that consumes IDatasets to provide the correct notifications.

> * As Peter Chang suggested in email yesterday, it may be worth
> removing the ROI classes - since these seem specific to one usage of
> IDataset.  Other applications/domains will have their own libraries of
> geometric shapes (such as the JTS libraries used in GeoTools [3])
I had tried to leave everything I could in the original
org.eclipse.dawnsci.analysis.dataset plug-in and then add to in only
the relevant parts of org.eclipse.dawnsci.analysis.api. So it could
very well be that removing ROIs from datasets and adding them back
somewhere else makes sense. To see everything that is dependent on
ROIs in the dataset bundle, have a look at this branch:
https://github.com/jonahkichwacoders/org.eclipse.dataset/commits/remove_rois

I think this opens up a great debate. Should org.eclipse.dataset be
synonymous with numpy? What is the dividing line between numpy and
scipy? Should we have the same dividing line?