| Re: [science-iwg] Eclipse Datasets for Science |
Send science-iwg mailing list submissions to
science-iwg@xxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg
or, via email, send a message with subject or body 'help' to
science-iwg-request@xxxxxxxxxxx
You can reach the person managing the list at
science-iwg-owner@xxxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of science-iwg digest..."
Today's Topics:
1. E4 (UOMo)
2. Re: Eclipse Datasets for Science (Peter.Chang@xxxxxxxxxxxxx)
3. Re: Eclipse Datasets for Science (Jonah Graham)
4. Re: Eclipse Datasets for Science (Jay Jay Billings)
----------------------------------------------------------------------
Message: 1
Date: Wed, 9 Sep 2015 10:53:31 +0200
From: UOMo <uomo@xxxxxxxxxxx>
To: science-iwg@xxxxxxxxxxx
Subject: [science-iwg] E4
Message-ID:
<CAAGawe2o4t51ZuUPS4OYyBvYqDg5kkQuwBmrHepk88Pbx9SEsA@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"
Hi,
As I noticed, Eclipse 4 was also mentioned in a recent post here, if any
member of the WG has experience with E4 they're able to discuss with the
STEM team (https://wiki.eclipse.org/The_STEM_Development_Team), especially
Jamie or Stefan would appreciate sharing it with STEM.
Eclipse STEM started using E4 in certain areas already and would be happy
to hear what others can say about their usage.
Thanks,
Werner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/science-iwg/attachments/20150909/bc79b283/attachment.html>
------------------------------
Message: 2
Date: Wed, 9 Sep 2015 09:30:37 +0000
From: <Peter.Chang@xxxxxxxxxxxxx>
To: <science-iwg@xxxxxxxxxxx>
Subject: Re: [science-iwg] Eclipse Datasets for Science
Message-ID:
<7E83BBEFAFDA644FBB249281490DAC1BDBF307C1@xxxxxxxxxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset=us-ascii
Hi all,
Ian wrote:
...
> - suggestions for improvements.
>
> * It would seem fairly straightforward to introduce TimeDataset and
> PeriodDataset classes to store instants in time and periods of elapsed
> time respectively. Representing time and period in dedicated classes
> offers the opportunity for the respective APIs to provide additional
> methods (e.g. before, after, overlaps). Dedicated classes also reduces
> the chance of integration troubles caused by teams misunderstanding
> how, for example, an elapsed period is being represented in a
> DoubleDataset (one team presumes floating point seconds, another
> team presumes floating point millis).
This ambiguity can be resolved with defining units metadata.
> * Deeper support for data change listeners will allow the UI to update
> when data changes, or for a chain of dependent datasets & operations
> that dynamically update on changes to precedent datatset.
There hasn't been much thought on this type of usage for datasets: dynamically updating items in a source dataset and subsequently calculating the updates of its descendants. Perhaps this should be left to the processing operation infrastructure.
>
> * I accidentally conflated the MetadataType and IMetadata types
> handled by ILazyDataset & IDataset whilst familiarising myself with
> the code. Since IMetadata actually is an instance of MetadataType,
> I guess my confusion would be avoided by removing getMetadata()
> from IDataset (replacing it with getMetadata(IMetadata.class).
It was legacy that may be worth removing...
>
> * As Peter Chang suggested in email yesterday, it may be worth
> removing the ROI classes - since these seem specific to one usage of
> IDataset. Other applications/domains will have their own libraries of
> geometric shapes (such as the JTS libraries used in GeoTools [3])
I don't think I suggested removing it just other things but we can utterly strip down to just dataset and metadata.
Regards,
Peter
>
> Cheers,
> Ian
>
>
> [1]
> https://github.com/jonahkichwacoders/org.eclipse.dataset.examples#g
> etting-started
> [2] http://pastebin.com/raw.php?i=PeYGxRp1
> [3] http://docs.geotools.org/stable/userguide/library/jts/index.html
>
> On 8 September 2015 at 23:27, Tracy Miranda
> <tracy@xxxxxxxxxxxxxxxx> wrote:
> > Following on from our previous email[1] Jonah has now created a fork
> > of DAWNSci's datasets which is available here:
> > https://github.com/jonahkichwacoders/org.eclipse.dataset
> >
> > Details of how you can try it out for yourself are here:
> > https://github.com/jonahkichwacoders/org.eclipse.dataset.examples
> >
> > In particular this fork:
> > * Is a subset of dataset functionality
> > * Has a p2 site available to try out the code
> > * p2 site includes source bundles
> > * Depends only on current Orbit approved bundles
> > * Has no pre-compiled JARs
> > * Works with Eclipse 4
> > * Works as a standalone JAR
> > * Includes all relevant tests from Dawn
> > * Buildable with maven/tycho (e.g. run mvn verify and get an update
> > site)
> > * using Travis CI as build machine
> >
> > Datasets features & functionality includes:
> > * Various types: doubles, compound, complex, float, etc
> > * Array creation
> > * Universal functions e.g. sin, cos, exp
> > * Linear algebra
> > * Indexing & slicing
> > * Regions of interest e.g. rectangular
> >
> > This fork is intended to be a springboard for SWG members to try out
> > DAWN datasets, and start to discuss how it applies to their projects.
> > We will maintain it in the short term.
> >
> > When you next have an opportunity we hope all interested parties
> could
> > 1.Try it out
> > - can you run the example?
> > - can you integrate it with your codebase?
> >
> > 2. Get Involved in architecture discussions:
> > - how does it fit within your framework?
> > - what works well?
> > - what's missing?
> > - suggestions for improvements.
> >
> > Ultimately the goal is to have a standardised dataset API &
> > implementation shared by all.
> >
> > Regards,
> > Tracy
> >
> > [1] http://dev.eclipse.org/mhonarc/lists//science-iwg/msg01074.html
> >
> > _______________________________________________
> > science-iwg mailing list
> > science-iwg@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or
> > unsubscribe from this list, visit
> > https://dev.eclipse.org/mailman/listinfo/science-iwg
>
>
>
> --
> Ian Mayo
> Deep Blue C Technology Ltd
> - New technologies, implemented well
> _______________________________________________
> science-iwg mailing list
> science-iwg@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit https://dev.eclipse.org/mailman/listinfo/science-
> iwg
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
------------------------------
Message: 3
Date: Wed, 9 Sep 2015 11:17:51 +0100
From: Jonah Graham <jonah@xxxxxxxxxxxxxxxx>
To: Science Industry Working Group <science-iwg@xxxxxxxxxxx>
Subject: Re: [science-iwg] Eclipse Datasets for Science
Message-ID:
<CAPmGMvhrZUbOpoMdMYiPjjouq6YSgc6bOJQ6UbE_q1wZ_5xwVA@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset=UTF-8
Hi Ian,
Excellent, thanks for starting to review the code. Peter has already
addressed some of the technical issues and I'll add in my 2 cents.
> [...] but the
> other examples complained about the lack of an slf4j binding [2]
I put in a "hack" to workaround the slf4j bindings missing for
BasicExample. Since the idea of slf4j is that the integrator decides
on the binding I have left dependency out completely from the examples
and platform.
My feeling is that some of the error handling within datasets needs a
review, with items such as methods that are declared throw Exception
[1] or some inconsistencies about what is logged vs what is thrown
[2].
[1] org.eclipse.dataset.ILazyDataset.getSlice(IMonitor, int[], int[],
int[]) throws Exception, however the only exception ever actually
thrown appears to be when the IMonitor is cancelled.
[2] Most places raise a RuntimeException at the same time as they
log.error the same message. However there are some exceptions to this,
such as org.eclipse.dataset.AbstractDataset.get1DIndex(int, int)
which logs the error, and then tries to correct for it and
org.eclipse.dataset.LazyDatasetBase.getMetadata() which catches an
Exception and logs it.
> > - what's missing?
>
> * From the DAWN code, I think there is value in the IOperation
> framework being pulled over. The IOperation classes appear
> implementation agnostic - so it should be possible to add generic
> operations to this project for common use.
I agree that IOperation should be made accessible. Presumably it
should happen at the same time. At the moment my goal was to slim down
DAWNSci to only things that did not depend on DAWN. The reality is
that all the IOperation concrete classes are not part of DAWNSci yet,
so weren't really under my consideration for extracting. I do hope
someone has the time and money to extract it though, possibly into a
new project that depends on org.eclipse.dataset.
> > - suggestions for improvements.
> * Deeper support for data change listeners will allow the UI to update
> when data changes, or for a chain of dependent datasets & operations
> that dynamically update on changes to precedent datatset.
I am with Peter on this, unfortunately I am not fully familiar with
the requirements thus far, but any concept of listeners inside the
data objects gives me concern. Perhaps there is a need for a unified
layer that consumes IDatasets to provide the correct notifications.
> * As Peter Chang suggested in email yesterday, it may be worth
> removing the ROI classes - since these seem specific to one usage of
> IDataset. Other applications/domains will have their own libraries of
> geometric shapes (such as the JTS libraries used in GeoTools [3])
I had tried to leave everything I could in the original
org.eclipse.dawnsci.analysis.dataset plug-in and then add to in only
the relevant parts of org.eclipse.dawnsci.analysis.api. So it could
very well be that removing ROIs from datasets and adding them back
somewhere else makes sense. To see everything that is dependent on
ROIs in the dataset bundle, have a look at this branch:
https://github.com/jonahkichwacoders/org.eclipse.dataset/commits/remove_rois
I think this opens up a great debate. Should org.eclipse.dataset be
synonymous with numpy? What is the dividing line between numpy and
scipy? Should we have the same dividing line?
------------------------------
Message: 4
Date: Wed, 9 Sep 2015 07:15:15 -0400
From: Jay Jay Billings <jayjaybillings@xxxxxxxxx>
To: Science Industry Working Group <science-iwg@xxxxxxxxxxx>
Subject: Re: [science-iwg] Eclipse Datasets for Science
Message-ID:
<CAE3ybv5yj6pnKbrZc9jg6WU=bntq1F0CRRSK5A=0-AJM2idfEA@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"
Great news! Looking forward to trying it.
Jay
On Sep 8, 2015 6:28 PM, "Tracy Miranda" <tracy@xxxxxxxxxxxxxxxx> wrote:
> Following on from our previous email[1] Jonah has now created a fork of
> DAWNSci's datasets which is available here:
> https://github.com/jonahkichwacoders/org.eclipse.dataset
>
> Details of how you can try it out for yourself are here:
> https://github.com/jonahkichwacoders/org.eclipse.dataset.examples
>
> In particular this fork:
> * Is a subset of dataset functionality
> * Has a p2 site available to try out the code
> * p2 site includes source bundles
> * Depends only on current Orbit approved bundles
> * Has no pre-compiled JARs
> * Works with Eclipse 4
> * Works as a standalone JAR
> * Includes all relevant tests from Dawn
> * Buildable with maven/tycho (e.g. run mvn verify and get an update site)
> * using Travis CI as build machine
>
> Datasets features & functionality includes:
> * Various types: doubles, compound, complex, float, etc
> * Array creation
> * Universal functions e.g. sin, cos, exp
> * Linear algebra
> * Indexing & slicing
> * Regions of interest e.g. rectangular
>
> This fork is intended to be a springboard for SWG members to try out DAWN
> datasets, and start to discuss how it applies to their projects. We will
> maintain it in the short term.
>
> When you next have an opportunity we hope all interested parties could
> 1.Try it out
> - can you run the example?
> - can you integrate it with your codebase?
>
> 2. Get Involved in architecture discussions:
> - how does it fit within your framework?
> - what works well?
> - what's missing?
> - suggestions for improvements.
>
> Ultimately the goal is to have a standardised dataset API & implementation
> shared by all.
>
> Regards,
> Tracy
>
> [1] http://dev.eclipse.org/mhonarc/lists//science-iwg/msg01074.html
>
> _______________________________________________
> science-iwg mailing list
> science-iwg@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://dev.eclipse.org/mailman/listinfo/science-iwg
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/science-iwg/attachments/20150909/81d3cfa0/attachment.html>
------------------------------
_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg
End of science-iwg Digest, Vol 32, Issue 48
*******************************************