[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: AW: [equinox-dev] Re: [prov] Deltas for delivering updates

Agreed!
So, e.g. a post processing step would be something like (processor-id="apply-xdelta",data="from-version=1.0",required=true).
Is the intended semantic of ´required==false´ that a ProcessorManager/Executer may decide on some context whether or execute or not, e.g. md5 check required or not may be a decision of the ´user´?
Currently ProcessingStep is ´only´ part of the ArtifactDescriptor and so neither available to requests and repositories because it is not part of their interface. They only know IArtifactKey. It is only used inside LocalWrittableRepo and meta data generation.
Given that a request can not execute post processors. As a consequence ArtifactDescriptor should be made visible/accessible/query-able from the repository such as
ArtifactDescriptors[] getArtifactDescriptors( IArtifactKey key )
With this a request can than decide for which of the returned descriptors it is capable of processing the required post processing steps.


Another question that comes into my mind is: Should the post processing be capable of piping the output of one post processor to the next post processor?

I could start working on this post processing stuff, if that is ok?

I am not sure whether a repository should be responsible for download at all. It´s responsibility should be storing/updating/deleting and retrieving/querying, it should delegate this to requests. Even the initial feed of the artifact descriptors should be handled by a request, otherwise dealing with stuff like (internet) proxies, etc. has to be done in multiple places.

Stefan

Pascal Rapicault wrote:
Answers embedded <pascal></pascal>
PaScaL


"Liebig, Stefan " <Stefan.Liebig@co mpeople.de> To Sent by: "Equinox development mailing list" equinox-dev-bounc <equinox-dev@xxxxxxxxxxx> es@xxxxxxxxxxx cc Subject 07/18/2007 08:39 AW: [equinox-dev] Re: [prov] Deltas AM for delivering updates Please respond to Equinox development mailing list <equinox-dev@ecli pse.org>





First to provide some clarification for me: The content (repositoryDescriptor) of an artifact repo is described within the artifacts.xml. This maps to a set of artifact descriptors where each artifact descriptor contains an artifact key, the artifacts download size and a list of post processing steps. The post processing steps seam to be currently unused currently.

<pascal> This is correct. The idea behind the post processing step was to
describe operations that had to be performed on the artifact once it had
been downloaded. The canonical example being unpack (from pack200). It
could be used for MD5 checks and things like that but I'm less sure about
this.
Also these processing steps should probably be used as negotiation between
repositories, since for example I should not download a packed200 jar if I
don't have the support to unpack it.</pascal>

Additionaly the repo descriptor contains a mapping definition to create a
url for accessing an
artifact. This url is than used to fetch the artifact.
<pascal> The mapper is an implementation detail of the simple artifact
repository. Other repository implementers may do the mapping another
way.</pascal>

How could deltas (the plural is for different delta algorithms) be
described here?
Is a delta an artifact in itself?
<pascal> Yes, however it is never explicitly requested for mirror by the
creator of a request.</pascal>

Maybe an artifact key (of a delta) contains an optional version property to
define the ´from version´?
The ´from version´ and the type of alg. are necessary to apply the delta.
The Mapper has than to consider the ´from version´ and the processing step
id for creating the url.
Would the ´from version´ better be part (data field) of a processing step?
The Mapper has than to consider also the processing steps for creating the
url.
<pascal>I like the approach where the type of delta is indicated as a post
processing step. It fits well in the model where first you check that the
target artifact repo can receive the artifact being downloaded, then
download, then apply the post processing steps, because downloading a delta
is just like downloading another artifact that could have its own
processing step, etc. Then the only thing that needs special code for is
how to find if a delta can be downloaded instead of the complete artifact
being requested.
Maybe a first thing would be to implement the support for post processing.
This will likely question how the download is currently handled (I'm not
really in love with what we have now) and may help move some
responsibilities around. For example, maybe the download should be handled
in the target repo itself.
</pascal>


Anyway, in a repository there may be stored a complete artifact, a zipped artifact, a xdelta artifact or a jbdiff artifact,.. of one and the same target artifact.

With that the MirrorRequest could be enhanced such that it can query the
source repo for different
variants (complete,delta,..) of the target artifact, choose the appropriate
one, download it, apply
the post processing steps and store it in the target repo.

Stefan

-----Ursprüngliche Nachricht-----
Von: equinox-dev-bounces@xxxxxxxxxxx
[mailto:equinox-dev-bounces@xxxxxxxxxxx] Im Auftrag von
Pascal Rapicault
Gesendet: Mittwoch, 11. Juli 2007 15:50
An: Equinox development mailing list
Cc: equinox-dev@xxxxxxxxxxx; equinox-dev-bounces@xxxxxxxxxxx
Betreff: [equinox-dev] Re: [prov] Deltas for delivering updates

As you correctly identified the separation of responsibility
between ECF and the provisioning code is done as follow:
- ECF is used to do the transports (e.g http, ftp,
bittorrent). We leverage, its API, its transport pluggability
(and eventually its ability to restart downloads).
- The provisioning code (DownloadManager) is responsible to
cause the download of all the requested artifacts using the
artifact repos. It will be responsible for picking the best
mirror (hopefully we can reuse what is in Maya) and also
doing the parallelization.

Can/should delta handling be part of ECF - as a kind of smart
transport/download handling? Or is delta handling so
important to provisioning that it should be part of provisioning?
Given our usage of ECF, I don't think that ECF has enough
context to be able to handle delta (it would not have the
identification information that you are talking about).
Therefore I think the ability to do delta depends on the
repository in which the artifact is being mirrored to and
from and it should be transparent for the download manager.
Therefore with the current code I think the decision or not
to use delta can be made in the MirrorRequest by checking the
characteristics of the artifact repositories at hand, and the
actual logic of getting the right delta could be left to the
implementation of the repositories.

It is also important to note that this code did not receive a
lot of attention and anything could be changed.

PaScaL




"Liebig, Stefan "

             <Stefan.Liebig@co

             mpeople.de>
          To
             Sent by:
<equinox-dev@xxxxxxxxxxx>
             equinox-dev-bounc
          cc
             es@xxxxxxxxxxx


Subject [equinox-dev] Deltas for delivering 07/11/2007 04:49 updates

             AM





             Please respond to

                  Equinox

                development

               mailing list

             <equinox-dev@ecli

                 pse.org>









I am currently investigating how the delta stuff (binary
deltas) could fit into the ongoing provisioning work.

When working with deltas there are certain constraints such as:
- when requesting a delta for an artifact it is essential
that the request contains an identification (version, md5,
..) of the artifact that is already available on the client,
so that is possible to regenerate the new artifact from the
old artifact and the delta.
- the client should not rely on the fact that the server(s)
can always respond with a delta. In some way the client needs
to have some flexibility on handling the data (deltas,
complete ´files´) which the server can respond.

Currently provisioning uses ECF to handle the transport and download.
Can/should delta handling be part of ECF - as a kind of smart
transport/download handling? Or is delta handling so
important to provisioning that it should be part of provisioning?

Any thoughts?
_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev


_______________________________________________ equinox-dev mailing list equinox-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/equinox-dev

_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev


_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev