How to access EODATA and Object Storage using s3cmd on Linux on CREODIAS
You can install s3cmd using Python PIP or from your Linux repository. This article is written for Ubuntu 22.04.
Installation from your system repository on Debian/Ubuntu systems:
Check for updates:
$ sudo apt update
Install s3cmd:
$ sudo apt install s3cmd
Installation from the Python repository (on most Linux distributions with python and pip preinstalled):
Installing with PIP:
Check if you have PIP installed:
$ pip3
If it is not installed (Ubuntu):
$ sudo apt install python3-pip
$ pip3 --version
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Now you should be able to install s3cmd using pip3:
$ sudo pip3 install s3cmd
s3cmd configuration
First, obtain access and secret key you use for accessing EODATA. You will need these credentials later. The following article contains instructions how to do it: How to get credentials used for mounting EODATA on a cloud VM on CREODIAS
Enter the following command:
$ s3cmd --configure
Refer to the user manual for detailed description of all options.
You will now be prompted for providing certain details. Below are the values which you should use during the configuration. If your access key is different than CLOUDFERRO and your secret key is different than PUBLIC, replace these values with appropriate keys you obtained previously.
Access Key [access]:CLOUDFERRO
Secret Key [access]:PUBLIC
Default Region [RegionOne]: default
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [data.cloudferro.com:] data.cloudferro.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: <ENTER>
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: <ENTER>
Path to GPG program [/usr/bin/gpg]: <ENTER>
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: False
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: <ENTER>
New settings:
Access Key: CLOUDFERRO
Secret Key: PUBLIC
Default Region: default
S3 Endpoint: data.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name: _____
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] <ENTER>
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y <ENTER>
Configuration saved to '/home/eouser/.s3cfg'
Access Key [access]:CLOUDFERRO
Secret Key [access]:PUBLIC
Default Region [RegionOne]: default
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [eodata.cloudferro.com:] eodata.cloudferro.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: <ENTER>
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: <ENTER>
Path to GPG program [/usr/bin/gpg]: <ENTER>
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: Yes
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: <ENTER>
New settings:
Access Key: CLOUDFERRO
Secret Key: PUBLIC
Default Region: default
S3 Endpoint: eodata.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name: _____
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] <ENTER>
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y <ENTER>
Configuration saved to '/home/eouser/.s3cfg'
Access Key [access]:CLOUDFERRO
Secret Key [access]:PUBLIC
Default Region [RegionOne]: default
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [eodata.cloudferro.com:] eodata.cloudferro.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: <ENTER>
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: <ENTER>
Path to GPG program [/usr/bin/gpg]: <ENTER>
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: Yes
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: <ENTER>
New settings:
Access Key: CLOUDFERRO
Secret Key: PUBLIC
Default Region: default
S3 Endpoint: eodata.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name: _____
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] <ENTER>
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y <ENTER>
Configuration saved to '/home/eouser/.s3cfg'
Access Key [access]:CLOUDFERRO
Secret Key [access]:PUBLIC
Default Region [RegionOne]: default
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [data.cloudferro.com:] data.cloudferro.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: <ENTER>
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: <ENTER>
Path to GPG program [/usr/bin/gpg]: <ENTER>
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: False
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: <ENTER>
New settings:
Access Key: CLOUDFERRO
Secret Key: PUBLIC
Default Region: default
S3 Endpoint: data.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name: _____
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] <ENTER>
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y <ENTER>
Configuration saved to '/home/eouser/.s3cfg'
Now you can use s3cmd commands (additional information about s3cmd: http://s3tools.org/usage).
$ s3cmd ls
2017-12-11 15:30 s3://DIAS
2017-12-11 15:30 s3://EOCLOUD
2017-12-11 15:30 s3://EODATA
$ s3cmd ls s3://EODATA/
DIR s3://EODATA/Envisat/
DIR s3://EODATA/Landsat-5/
DIR s3://EODATA/Landsat-7/
DIR s3://EODATA/Landsat-8/
DIR s3://EODATA/Sentinel-1/
DIR s3://EODATA/Sentinel-2/
DIR s3://EODATA/Sentinel-3/
DIR s3://EODATA/Sentinel-5P/
In order to acquire access to Object Storage buckets via s3cmd, first you have to generate your own ec2 credentials with this tutorial How to generate and manage EC2 credentials on CREODIAS.
After the creation of credentials please remove the file .s3cfg from your Home folder and then reconfigure s3cmd by entering:
s3cmd --configure
and the following values:
New settings:
Access Key: (your EC2 credentials)
Secret Key: (your EC2 credentials)
Default Region: none
S3 Endpoint: s3.waw3-1.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw3-1.cloudferro.com
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
New settings:
Access Key: (your EC2 credentials)
Secret Key: (your EC2 credentials)
Default Region: none
S3 Endpoint: s3.waw3-2.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw3-2.cloudferro.com
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
New settings:
Access Key: (your EC2 credentials)
Secret Key: (your EC2 credentials)
Default Region: none
S3 Endpoint: s3.fra1-2.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: s3.fra1-2.cloudferro.co
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
New settings:
Access Key: (your EC2 credentials)
Secret Key: (your EC2 credentials)
Default Region: none
S3 Endpoint: s3.waw2-1.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw2-1.cloudferro.com
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
After this operation, you should be allowed to list and access your Object Storage.