Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [science-iwg] January - New Eclipse Project Proposal

Matt,

I don't have a good write up on them. (At least I thought the write up that we do have was much better. Interns...) At any rate, here is the code:

https://github.com/eclipse/ice/tree/master/org.eclipse.ice.datastructures

The goal of this package is to create general purpose data classes, structures and pattern realizations that can be mapped to a wide range of scientific problems while also maintaining metadata about that information. They are also all bound with JAXB so that they can be persisted to XML. Their design is verbose so that developers can almost immediately know how to pack their data into the classes. 

Consider, for example, a battery. If the state of that battery would be represented on disk by five quantities - say a string, two integers and two floats - and each of those quantities has associated metadata such as descriptions, ids, names, etc., then we could map them as follows:

Battery --> 1 instance DataComponent
Quantities 1-5 --> 5 instances of Entry

Then any collection of DataComponents, etc. are stored in a Form that is processed by the workflow engine and the UI. They are, in a sense, the exact opposite of IDataSet because they store rather macroscopic quantities and ignore large amounts of n-dimensional data, (which we leave on disk and store in ResourceComponents).

Our long term goals with this are to switch this to an EMF model, optimize the way metadata is stored, use IDataSet to back structures like MatrixComponent, and allow developers to create their own versions via annotations.

Jay

On Wed, Jan 27, 2016 at 8:27 AM, <Matt.Gerring@xxxxxxxxxxxxx> wrote:

“However, if it is the latter then it is bigger than simply being a numpy alternative...”

 

Sounds like you have a real extension. I don’t understand what it is (because I haven’t looked). Where should I look to understand what the extension is please? Or can you detail it on the email string?

 

Cheerio,

 

Matt

 

From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Tracy Miranda
Sent: 27 January 2016 13:17


To: Science Industry Working Group
Subject: Re: [science-iwg] January - New Eclipse Project Proposal

 

Agreed.

 

When we started this discussion nd-arrays & IDatasets were one of the first things talked through and prioritised. As the conversations continued, other dimensions & use-cases opened up. While the spec was updated to try to reflect this, it looks like we need to take an in-depth look to make the scope & priorities clear. Part of the problem is that different things get discussed at different events, meetings - so let's collate those all now so we're all on the same page.  

 

I'm going to kick-off a new thread to try to frame those discussions.

 

Tracy

 

On Wed, Jan 27, 2016 at 12:36 PM, Jay Jay Billings <jayjaybillings@xxxxxxxxx> wrote:

Well, I suppose that is what we need to figure out... is this a numpy in Java project or the data structures project we discussed on the list and at the conference?

If it is the former, then, sure, it has nothing to do with ICE and I retract my suggestions. However, if it is the latter then it is bigger than simply being a numpy alternative... or was Cormen's data structures book wrong and the only data structures that matter are n-dimensional arrays?

I'm happy whichever way it goes because we need the IDataset code, but I think the scope needs to be completely clear.

Jay

On Jan 27, 2016 4:21 AM, <Matt.Gerring@xxxxxxxxxxxxx> wrote:

Hi folks,

 

I don’t think I agree with Jay on this. For me January is ‘numpy for Java’ (although this is up to the project leaders, I thought that is what they were doing). If the ICE data-structures replicate numpy functionality not present, they should be in the IC. If not they should be either added later if the project leaders agree or be in a separate project.

 

Mathematics follows where elegance leads, as they say.

 

All the best,

 

Matt

 

 

 

From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Jay Jay Billings
Sent: 26 January 2016 19:31
To: Science Industry Working Group
Subject: Re: [science-iwg] January - New Eclipse Project Proposal

 

Jonah,

 

That is part of the reason you should have more committers: so you don't have to know everything about the code! :-P

 

Supposing that you do wait to include the code for ICE, it is still better to put the committers you want on the project now. Then, when it is time to start moving the code from ICE or any other project it can just happen without waiting on elections. It takes quite a long time for a committer to address a few bugs and then go through the election. At the same time, it is important to have existing committers from the old parent projects on the data structures project because it is unreasonable to expect it to all be dumped on you and Peter.

 

Another thing I just remembered: Didn't we decide that we wanted the name to include "data" in it somewhere? While I really like "January," I'm worried that no one will have any clue what the org.eclipse.january packages are all about.

 

Jay

 

On Tue, Jan 26, 2016 at 1:30 PM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:

Hi Jay,

Thanks for raising your concerns, lets keep this discussion going.

Regarding the first concern, while the intention was to try to include
ICE data structures in the initial contribution, I simply have not had
the time I hoped to be able to review these. With DAWNSci structures
it took a reasonable amount of discussion as well as work on
dependencies, restructuring and to & fro with Peter to get them into
the current state. Conscious of the fact the project proposal
mechanics take time, I opted to go with this 'minimum viable product'
and schedule the integrations with ICE, EAVP and others after the
fact.

Likewise, with the second concern, again I have opted for the lean
approach and gone for the bare minimum number of developers.

I share your desire to get the community involved as much as possible.
In this case my approach is to start with getting a minimal feature
set out  and build up from there.

Thanks,
Jonah

~~~
Jonah Graham
Kichwa Coders Ltd.
www.kichwacoders.com

On 26 January 2016 at 15:24, Jay Jay Billings <jayjaybillings@xxxxxxxxx> wrote:
> Jonah,
>
> This is great news! Awesome!
>
> My first concern is that during our previous discussion about this both at
> ECE, on the list, and during the SC call a couple of months ago we concluded
> that it was best to include ICE's data structures as part of the initial
> contribution. From the proposal it looks like you want to deliver an IC and
> then come back around to EAVP and ICE. I think that is the correct approach
> for EAVP, but I think ICE's data structures should be part of the IC. They
> are ready to go and can just be dropped in.
>
> My second concern is that the project has only two committers, you and
> Peter. I think there are other people in the group that would have
> justifiable roles in this (Matt, Baha, etc.) and it is better to list them
> now. If you list them now then they will not have to be elected later.
>
> Both of these concerns boil down to getting as much of the community
> involved as soon as possible. I think that is really important for all the
> science projects.
>
> I'll copy this into a comment on the proposal.
>
> Jay
>
> On Tue, Jan 26, 2016 at 5:25 AM, Jonah Graham <jonah@xxxxxxxxxxxxxxxx>
> wrote:
>>
>> Dear all,
>>
>> Peter and I are pleased to announce that the January project proposal
>> (https://projects.eclipse.org/proposals/january) has now gone live.
>>
>> We are now soliciting community feedback so that we can progress the
>> proposal to the creation review.
>>
>> We have written a short blog post
>> (http://kichwacoders.com/2016/01/26/introducing-project-january/) with
>> some background on naming too.
>>
>> Best Regards,
>> Jonah
>>
>> ~~~
>> Jonah Graham
>> Kichwa Coders Ltd.
>> www.kichwacoders.com
>> _______________________________________________
>> science-iwg mailing list
>> science-iwg@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe
>> from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/science-iwg
>
>
>
>
> --
> Jay Jay Billings
> Oak Ridge National Laboratory
> Twitter Handle: @jayjaybillings
>
> _______________________________________________
> science-iwg mailing list
> science-iwg@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/science-iwg
_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg



 

--

Jay Jay Billings

Oak Ridge National Laboratory

Twitter Handle: @jayjaybillings

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

 

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg



--
Jay Jay Billings
Oak Ridge National Laboratory
Twitter Handle: @jayjaybillings

Back to the top