Remote transfer for EODATA on CREODIAS

The CREODIAS platform provides several ways to download EODATA (Earth Observation Data) products, depending on your environment and workflow needs:

Browse and download via Data Explorer

Use the CREODIAS web interface to explore and retrieve products interactively.

First step in working with Data Explorer is login, then download product or process products.

Download to a Virtual Machine (VM)

Set up a VM on CREODIAS, store the data locally on it, and process it within the cloud environment.

Access EODATA with boto3, s3cmd, curl and wget.

Get credentials to access EODATA via VM – alternatively, mount EODATA with Goofys or s3fs.

Download via remote transfer

Retrieve data directly to your own infrastructure using the S3 protocol.

This article explains that method in detail.

What we are going to cover

Prerequisites

No. 1 Activated dashboard services

To use services in the account at link https://new.cloudferro.com/, you need to enter organization data and top your wallet with adequate funds. See How to start using dashboard services on CREODIAS.

For a brand new account, the Dashboard will look like this:

No. 2 Commercial conditions for EODATA transfer

This article sets up a general overview for remote EODATA transfers:

How to access EODATA from your own infrastructure on CREODIAS

If you already know which datasets you would like to download, check the list of available EODATA collections.

Step 1 Request EODATA transfer

In Dashboard, an option called Eodata Transfers will be available in the Billing and Reporting group.

The first time you click on it, the following window will appear:

Click on the only link provided and create a ticket, requesting that you want the operator to enable EODATA transfer for your account.

After the transfer is approved, you will get an email message similar to this:

Click on the link in it to start using the new options in the Dashboard.

Step 2 Use EODATA transfer

Assuming the request is approved, the account options and content will appear different in the following ways:

1. The contents of the browser window changes to this view:

   

   You will see the state of your current items and wallets (if you have activated more than one wallet).

2. Under Configuration group, a new option will appear — S3 Credentials.

   

   This is the main option to work with the credentials – create new ones, edit or delete the existing ones.

3. Under Billing and Reporting, the option Wallets/Contracts will show the credits allocated to the account for this new service. Note that remote transfer of EODATA is always billed in PAY AS YOU GO mode:

   

The amount will change every two hours, depending on the amount of transfers you do. Be sure to check the remaining state of the account before large transfers.

Step 3 Obtain S3 credentials for transfer

To access data in S3 format, you need to create a special credential. Click on option S3 Credentials to see the corresponding window:

Next, click on Add Credential button and the modal window to add the credential will appear:

The only parameter to define is the date at which the credential will expire. Click on calendar icon to define it using GUI or enter the data in format day.month.year:

The next modal window shows the access and secret keys as the S3 credentials generated:

You may have two or more S3 credentials, like here:

Step 4 (Optional) Special options for each credential

For each credential you can Edit, Reset Secret and Delete it.

Edit credential

Editing credential means changing its expiration date:

Reset Secret Key

You have to confirm that you want to reset the secret key:

The next window will contain the old access key and a new secret key:

Note

The key will be reset immediately and using the old key will stop functioning. If you have used the old secret key in software or some manual procedures, be sure to replace it everywhere with the newly generated secret key.

Delete Credential

The window at which you need to confirm that you want to delete the credential is this:

The consequences:

• A balloon window at the top to confirm that the credential was deleted.

• One credential less in the list of credentials.

• No more downloading EODATA with that credential.

Security best practices

You can have several credentials active at the same time. If other members of your project are using them, you will want to rotate keys regularly. To that end, change the expiry date and/or reset secret key and – in the direst circumstances – consider deleting the credentials altogether.

Do not store credentials in plain text, especially not somewhere in the cloud or printed on paper at the office.

Troubleshooting

If you cannot see the S3 Credentials option after approval:

• Refresh the dashboard page.

• Log out and log in again.

• Contact support through article Helpdesk and Support if the option is still missing.

Sanity check

As a proof of concept, you can use s3cmd from article How to download EODATA files using s3cmd on CREODIAS.

Here is how to deploy the access and secret credentials and then use them in Terminal to list the buckets:

export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""

# Helpful defaults
export AWS_DEFAULT_REGION="us-east-1"
export AWS_EC2_METADATA_DISABLED=true

# EODATA endpoint (required with every command)
export ENDPOINT="https://eodata.cloudferro.com"

s3cmd \
  --access_key="$AWS_ACCESS_KEY_ID" \
  --secret_key="$AWS_SECRET_ACCESS_KEY" \
  --host="eodata.cloudferro.com" \
  --host-bucket="eodata.cloudferro.com/%(bucket)" \
  ls s3://DIAS/
                          DIR  s3://DIAS/C3S/
                          DIR  s3://DIAS/CAMS/
                          DIR  s3://DIAS/CEMS/
                          DIR  s3://DIAS/CLMS/
                          DIR  s3://DIAS/CLMS_archive/
                          DIR  s3://DIAS/CMEMS/
                          DIR  s3://DIAS/Envisat/
                          DIR  s3://DIAS/Envisat-ASAR/
                          DIR  s3://DIAS/Global-Mosaics/
                          DIR  s3://DIAS/Jason-3/
                          DIR  s3://DIAS/Landsat-5/
                          DIR  s3://DIAS/Landsat-7/
                          DIR  s3://DIAS/Landsat-8/
                          DIR  s3://DIAS/Landsat-8-ESA/
                          DIR  s3://DIAS/Landsat-9/
                          DIR  s3://DIAS/SMOS/
                          DIR  s3://DIAS/Sentinel-1/
                          DIR  s3://DIAS/Sentinel-1-RTC/
                          DIR  s3://DIAS/Sentinel-2/
                          DIR  s3://DIAS/Sentinel-3/
                          DIR  s3://DIAS/Sentinel-5P/
                          DIR  s3://DIAS/Sentinel-6/
                          DIR  s3://DIAS/Suomi-NPP/
                          DIR  s3://DIAS/Terra/
                          DIR  s3://DIAS/auxdata/

What To Do Next

This is the link to EODATA section: EODATA. Try the following articles as well:

• How to download EODATA file using boto3 on CREODIAS

• How to mount EODATA as a filesystem using Goofys in Linux on CREODIAS

• Using curl and wget to download EODATA products from CREODIAS