[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: AW: [equinox-dev] Re: [prov] Deltas for delivering updates

Awesome!
You are correct:
-  the required of the artifact descriptor is here to indicate whether this
is a mandatory step or not.
- the artifact descriptor objects need to be exposed (this was never done
'cause we never implemented the post processing)

I like the idea of piping processors. Let's see how far we can take it.
I like even more the idea that you start working on the post processing :-)

For the download in the repo I was hoping that the artifact repository
would reuse some helpers to avoid the problems you mentioned.  The reason
why I was suggesting to do the download in the repo was mostly to give more
flexibility about how things could be processed, but let's see how far we
can go with the current approach before changing things.

Another note:
- You may want to give the full artifact key in the processor data

Thx for your involvement,

PaScaL



                                                                       
             Stefan Liebig                                             
             <Stefan.Liebig@co                                         
             mpeople.de>                                                To
             Sent by:                  Equinox development mailing list
             equinox-dev-bounc         <equinox-dev@xxxxxxxxxxx>       
             es@xxxxxxxxxxx                                             cc
                                                                       
                                                                   Subject
             07/19/2007 04:42          Re: AW: [equinox-dev] Re: [prov]
             AM                        Deltas for delivering updates   
                                                                       
                                                                       
             Please respond to                                         
             Stefan.Liebig@com                                         
             people.de; Please                                         
                respond to                                             
                  Equinox                                              
                development                                            
               mailing list                                            
             <equinox-dev@ecli                                         
                 pse.org>                                              
                                                                       
                                                                       




Agreed!
So, e.g. a post processing step would be something like
(processor-id="apply-xdelta",data="from-version=1.0",required=true).
Is the intended semantic of ´required==false´ that a
ProcessorManager/Executer may decide on some context whether or execute
or not, e.g. md5 check required or not may be a decision of the ´user´?
Currently ProcessingStep is ´only´ part of the ArtifactDescriptor and so
neither available to requests and repositories because it is not part of
their interface. They only know IArtifactKey. It is only used inside
LocalWrittableRepo and meta data generation.
Given that a request can not execute post processors. As a consequence
ArtifactDescriptor should be made visible/accessible/query-able from the
repository such as
ArtifactDescriptors[] getArtifactDescriptors( IArtifactKey key )
With this a request can than decide for which of the returned
descriptors it is capable of processing the required post processing steps.

Another question that comes into my mind is: Should the post processing
be capable of piping the output of one post processor to the next post
processor?

I could start working on this post processing stuff, if that is ok?

I am not sure whether a repository should be responsible for download at
all. It´s responsibility should be storing/updating/deleting and
retrieving/querying, it should delegate this to requests. Even the
initial feed of the artifact descriptors should be handled by a request,
otherwise dealing with stuff like (internet) proxies, etc. has to be
done in multiple places.

Stefan

Pascal Rapicault wrote:
> Answers embedded <pascal></pascal>
> PaScaL
>
>
>

>              "Liebig, Stefan "

>              <Stefan.Liebig@co

>              mpeople.de>
To
>              Sent by:                  "Equinox development mailing list"

>              equinox-dev-bounc         <equinox-dev@xxxxxxxxxxx>

>              es@xxxxxxxxxxx
cc
>

>
Subject
>              07/18/2007 08:39          AW: [equinox-dev] Re: [prov]
Deltas
>              AM                        for delivering updates

>

>

>              Please respond to

>                   Equinox

>                 development

>                mailing list

>              <equinox-dev@ecli

>                  pse.org>

>

>

>
>
>
>
> First to provide some clarification for me:
> The content (repositoryDescriptor) of an artifact repo is described
within
> the artifacts.xml.
> This maps to a set of artifact descriptors where each artifact descriptor
> contains an artifact
> key, the artifacts download size and a list of post processing steps. The
> post processing steps
> seam to be currently unused currently.
>
> <pascal> This is correct. The idea behind the post processing step was to
> describe operations that had to be performed on the artifact once it had
> been downloaded. The canonical example being unpack (from pack200). It
> could be used for MD5 checks and things like that but I'm less sure about
> this.
> Also these processing steps should probably be used as negotiation
between
> repositories, since for example I should not download a packed200 jar if
I
> don't have the support to unpack it.</pascal>
>
> Additionaly the repo descriptor contains a mapping definition to create a
> url for accessing an
> artifact. This url is than used to fetch the artifact.
> <pascal> The mapper is an implementation detail of the simple artifact
> repository. Other repository implementers may do the mapping another
> way.</pascal>
>
> How could deltas (the plural is for different delta algorithms) be
> described here?
> Is a delta an artifact in itself?
> <pascal> Yes, however it is never explicitly requested for mirror by the
> creator of a request.</pascal>
>
> Maybe an artifact key (of a delta) contains an optional version property
to
> define the ´from version´?
> The ´from version´ and the type of alg. are necessary to apply the delta.
> The Mapper has than to consider the ´from version´ and the processing
step
> id for creating the url.
> Would the ´from version´ better be part (data field) of a processing
step?
> The Mapper has than to consider also the processing steps for creating
the
> url.
> <pascal>I like the approach where the type of delta is indicated as a
post
> processing step. It fits well in the model where first you check that the
> target artifact repo can receive the artifact being downloaded, then
> download, then apply the post processing steps, because downloading a
delta
> is just like downloading another artifact that could have its own
> processing step, etc. Then the only thing that needs special code for is
> how to find if a delta can be downloaded instead of the complete artifact
> being requested.
> Maybe a first thing would be to implement the support for post
processing.
> This will likely question how the download is currently handled (I'm not
> really in love with what we have now) and may help move some
> responsibilities around. For example, maybe the download should be
handled
> in the target repo itself.
> </pascal>
>
>
> Anyway, in a repository there may be stored a complete artifact, a zipped
> artifact, a xdelta artifact
> or a jbdiff artifact,.. of one and the same target artifact.
>
> With that the MirrorRequest could be enhanced such that it can query the
> source repo for different
> variants (complete,delta,..) of the target artifact, choose the
appropriate
> one, download it, apply
> the post processing steps and store it in the target repo.
>
> Stefan
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: equinox-dev-bounces@xxxxxxxxxxx
>> [mailto:equinox-dev-bounces@xxxxxxxxxxx] Im Auftrag von
>> Pascal Rapicault
>> Gesendet: Mittwoch, 11. Juli 2007 15:50
>> An: Equinox development mailing list
>> Cc: equinox-dev@xxxxxxxxxxx; equinox-dev-bounces@xxxxxxxxxxx
>> Betreff: [equinox-dev] Re: [prov] Deltas for delivering updates
>>
>> As you correctly identified the separation of responsibility
>> between ECF and the provisioning code is done as follow:
>> - ECF is used to do the transports (e.g http, ftp,
>> bittorrent). We leverage, its API, its transport pluggability
>> (and eventually its ability to restart downloads).
>> - The provisioning code (DownloadManager) is responsible to
>> cause the download of all the requested artifacts using the
>> artifact repos. It will be responsible for picking the best
>> mirror (hopefully we can reuse what is in Maya) and also
>> doing the parallelization.
>>
>>
>>> Can/should delta handling be part of ECF - as a kind of smart
>>>
>> transport/download handling? Or is delta handling so
>> important to provisioning that it should be part of provisioning?
>> Given our usage of ECF, I don't think that ECF has enough
>> context to be able to handle delta (it would not have the
>> identification information that you are talking about).
>> Therefore I think the ability to do delta depends on the
>> repository in which the artifact is being mirrored to and
>> from and it should be transparent for the download manager.
>> Therefore with the current code I think the decision or not
>> to use delta can be made in the MirrorRequest by checking the
>> characteristics of the artifact repositories at hand, and the
>> actual logic of getting the right delta could be left to the
>> implementation of the repositories.
>>
>> It is also important to note that this code did not receive a
>> lot of attention and anything could be changed.
>>
>> PaScaL
>>
>>
>>
>>
>>              "Liebig, Stefan "
>>
>>              <Stefan.Liebig@co
>>
>>              mpeople.de>
>>           To
>>              Sent by:
>> <equinox-dev@xxxxxxxxxxx>
>>              equinox-dev-bounc
>>           cc
>>              es@xxxxxxxxxxx
>>
>>
>>      Subject
>>                                        [equinox-dev] Deltas
>> for delivering
>>              07/11/2007 04:49          updates
>>
>>              AM
>>
>>
>>
>>
>>
>>              Please respond to
>>
>>                   Equinox
>>
>>                 development
>>
>>                mailing list
>>
>>              <equinox-dev@ecli
>>
>>                  pse.org>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> I am currently investigating how the delta stuff (binary
>> deltas) could fit into the ongoing provisioning work.
>>
>> When working with deltas there are certain constraints such as:
>> - when requesting a delta for an artifact it is essential
>> that the request contains an identification (version, md5,
>> ..) of the artifact that is already available on the client,
>> so that is possible to regenerate the new artifact from the
>> old artifact and the delta.
>> - the client should not rely on the fact that the server(s)
>> can always respond with a delta. In some way the client needs
>> to have some flexibility on handling the data (deltas,
>> complete ´files´) which the server can respond.
>>
>> Currently provisioning uses ECF to handle the transport and download.
>> Can/should delta handling be part of ECF - as a kind of smart
>> transport/download handling? Or is delta handling so
>> important to provisioning that it should be part of provisioning?
>>
>> Any thoughts?
>> _______________________________________________
>> equinox-dev mailing list
>> equinox-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/equinox-dev
>>
>>
>> _______________________________________________
>> equinox-dev mailing list
>> equinox-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/equinox-dev
>>
>>
> _______________________________________________
> equinox-dev mailing list
> equinox-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/equinox-dev
>
>
> _______________________________________________
> equinox-dev mailing list
> equinox-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/equinox-dev
>

_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev