[epp-dev] AWS S3 integration

[Markus Knauer <mknauer@xxxxxxxxxxxxxx>]

Sorry for cross-posting and spamming this list, but I think everything related 
to Amazons Simple Storage Service should be discussed (or at least mentioned) 
here in this mailing list.


Markus


----------  Forwarded Message  ----------

Subject: [g-Eclipse] AW: [g-Eclipse] AW: [g-Eclipse] M3 Release
Date: Wednesday 05 March 2008
From: "Stuempert, Mathias IWR"

Hi Pawel,

> Have we ever agreed voting rules? Should not it be unilateral?

[...]

> -1 for including S3 (or even -2)

Hey, now this is unilateral ;-)

> 1. it is not tested. My first test:
>   - create AWS token - ok, but AWS Credential Info is empty (minor
problem)

Yepp, the info dialog is in fact an optional feature of an auth token
implementation and therefore I did not include it in the first
implementation. Nevertheless somebody decided having an empty info
dialog would be better than having no info dialog, right Thomas :-P

>   - create new S3 connection - Scheme s3, Access Key and Bucket not
empty, 
> but Finish button is disabled and and no warning or error appeared.

Well, the same behaviour as for any other EFS implementation (besides
the fact that those already show error messages). I think your error was
just to forget the leading "/" in the path name, i.e. the bucket name.
Nevertheless, directly mounting buckets may not be a good idea at the
moment. So if we decide to include S3 into the release I will either
remove the field for directly mounting buckets or have another look into
this part of the code.

> 2. jets3 is using Axis (Axis1 version 1.4). As we know, different 
> plugins using Axis tends to influence each other. So I would not dare
to 
> put another Axis plugin into release without extensive tests.

It doesn't, at least as long as you are not making use of the
SoapS3Service. In fact jets3t recommends to make not use of this
implementation but to instead make use of the RestS3Service which does
only rely on commons.httpclient. All functionalities within our plug-ins
are based on REST and therefore no need to worry about axis.

> What about axis

Not necessarily needed as stated above.

> and bouncycastle?

Not necessarily needed. This is only needed if you would like to store
your AWSCredentials to an encrypted file on disk. As we are not doing
this at the moment BC is not needed.
 
> What about activation.jar (important for 
> java 1.5, for java1.6 it is in rt.jar), mail.jar (if other axis sets 
> attachements support),

Axis not needed -> these are also not needed.

> log4j (I cannot find reference to it, but jets3t 
> is distributed with this jar)

Log4j is just an optional cap on the head of commons-logging. You may
configure your commons-logging to make use of log4j by specifying

org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogg
er

in your commons-logging.properties. If this is not configured the
standard logging facility is used that ships with Java and/or
commons-logging. Therefore -> not needed.

You can have a simple test which jars are really needed to access S3.
Have a look at the cockpit application which can be found in the bin
folder of the jets3t distribution. Edit your OS-dependent start-script
(.sh or .bat) to only include:

set CP=%CP%;%MY_JETS3T_HOME%/jars/jets3t-0.6.0.jar
set CP=%CP%;%MY_JETS3T_HOME%/jars/jets3t-gui-0.6.0.jar
set CP=%CP%;%MY_JETS3T_HOME%/jars/cockpit-0.6.0.jar
set
CP=%CP%;%MY_JETS3T_HOME%/libs/commons-logging/commons-logging-1.1.1.jar
set CP=%CP%;%MY_JETS3T_HOME%/libs/commons-codec/commons-codec-1.3.jar
set
CP=%CP%;%MY_JETS3T_HOME%/libs/commons-httpclient/commons-httpclient-3.1.
jar

in the classpath. Furthermore go to the configs folder and comment the
above mentioned line specified log4j in the commons-logging.properties.
Then start cockpit and you will see that you still have full control
over your S3 account.

BTW, maybe some words about how S3 is organised and how to make use of
the g-Eclipse implementation. Your S3 account may contain a number of
buckets, these are your folders. These buckets may contain objects, i.e.
files. That's it about the structure. You can not have a file outside a
bucket or a bucket inside a bucket. Therefore you only have two levels,
the bucket level and the object level. Therefore trying to create a
folder within a folder will fail within g-Eclipse's S3 implementation
(still have to provide better error messages). Furthermore creating a
file directly in your connection, i.e. outside any bucket will also fail
(also here I have to provide better error messages). Another issue with
creating folders/buckets refers to the allowed names of a bucket. Note
that bucket names are global, that means there is only one namespace for
buckets. Therefore it will most likely fail if you try to create a
bucket named "MyBucket" since most probably anybody else in the world
has already created such a bucket (yes, it is REALLY GLOBAL). This is
not a g-Eclipse limitation but directly comes from S3. Therefore it is
recommended to prefix your bucket's name with your access key id, i.e.
instead of "MyBucket" -> "lasro7a3r7aw34.MyBucket" if "lasro7a3r7aw34"
is your access key id. In the future we may talk about making this
easier for the user but at the moment it is most S3-complient to keep it
as it is.

Sorry for the somewhat longer mail, but I thought giving a short
introduction into S3 would be a good idea :)

Cheers, Mathias

-------------------------------------------------------