How to access EODATA using s3cmd on Creodias
When you want to access EODATA on Creodias cloud, one option is to mount it as a file system so that the cloud data appears as part of the local Linux directory tree. Another option, described in this article, is to access the EODATA repository directly with s3cmd on a virtual machine hosted on Creodias cloud.
This method is useful when you want to list directories, inspect product paths, and download selected files or full products without mounting the repository as a file system.
What We Are Going To Cover
Configuring s3cmd for access to the EODATA repository
Listing buckets and exploring the EODATA repository
Downloading single files and full products
Using sync to download products and restore missing files
Prerequisites
No. 1 Account
You need a Creodias hosting account with access to the Horizon interface: https://horizon.cloudferro.com/auth/login/?next=/.
No. 2 Obtained EC2 credentials
You need credentials used specifically for accessing the EODATA repository. These credentials are not the same as the ones used for user-created object storage containers. To obtain them, follow this article: How to get credentials used for accessing EODATA on a cloud VM on Creodias
No. 3 A virtual machine
You need a virtual machine hosted on Creodias cloud. This article uses Ubuntu 24.04 as an example. Other operating systems are outside the scope of this article.
The virtual machine should already have access to the EODATA network.
To create a virtual machine hosted on Creodias cloud, follow one of these articles:
How to create a Linux VM and access it from Windows desktop on Creodias
How to create a Linux VM and access it from Linux command line on Creodias
No. 4 s3cmd installed
You need to have s3cmd installed on your virtual machine.
To install it, follow this article: How to install s3cmd on Linux on Creodias
No. 5 Understanding the s3cmd configuration file
s3cmd stores its connection settings in configuration files. In this article, one configuration file is used only for EODATA access.
To learn more about configuration files, see: Configuration files for s3cmd command on Creodias
Configuring s3cmd for access to the EODATA repository
In this article, the example configuration file is /home/eouser/eodata-access. You can store it in a different location if needed.
Start the configuration process with:
s3cmd -c /home/eouser/eodata-access --configure
You will be asked a series of questions. Enter your own access key and secret key obtained in Prerequisite No. 2.
The endpoint and HTTPS values depend on the cloud region you are using.
Answer the questions as follows:
Access Key [access]: CLOUDFERRO
Secret Key [access]: PUBLIC
Default Region [US]: us-east-1
S3 Endpoint [s3.amazonaws.com]: data.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:
Encryption password:
Path to GPG program [/usr/bin/gpg]:
Use HTTPS protocol [Yes]: No
HTTP Proxy server name:
Answer the questions as follows:
Access Key [access]: YOUR_ACCESS_KEY
Secret Key [access]: YOUR_SECRET_KEY
Default Region [US]: us-east-1
S3 Endpoint [s3.amazonaws.com]: eodata.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:
Encryption password:
Path to GPG program [/usr/bin/gpg]:
Use HTTPS protocol [Yes]: Yes
HTTP Proxy server name:
Leave the DNS-style bucket+hostname:port template empty. This is important. If you set a DNS-style bucket template such as %(bucket)s.%(host_base)s, s3cmd may try to build hostnames such as eodata.eodata.cloudferro.com, which will not work for EODATA access in these regions.
Once you have answered all questions, s3cmd will ask whether you want to test the connection:
Test access with supplied credentials? [Y/n]
Answer Y and press Enter.
If the configuration is correct, you should see output similar to this:
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
You will then be asked whether you want to save the settings:
Save settings? [y/N]
Answer y and press Enter.
The configuration file should then be saved:
Configuration saved to '/home/eouser/eodata-access'
Since this file contains access credentials, it is a good idea to make it readable only by the current user:
chmod 600 /home/eouser/eodata-access
Checking the saved configuration file
After saving the configuration, open the file and verify the most important lines:
nano /home/eouser/eodata-access
For WAW3-1, the relevant part should look like this:
host_base = data.cloudferro.com
host_bucket = data.cloudferro.com
bucket_location = us-east-1
use_https = False
signature_v2 = False
For WAW3-2, FRA1-2, and WAW4-1, the relevant part of the file should look like this:
host_base = eodata.cloudferro.com
host_bucket = eodata.cloudferro.com
bucket_location = us-east-1
use_https = True
signature_v2 = False
In these EODATA configurations, host_bucket should match host_base. Do not use a bucket-prefixed DNS template for EODATA access in these regions.
Exploring the EODATA repository
To list available buckets, execute:
s3cmd -c /home/eouser/eodata-access ls
The output depends on the region and endpoint used by the s3cmd configuration file.
In WAW3-1, the command:
s3cmd -c /home/eouser/eodata-access ls
may return several buckets, similar to these:
2017-11-15 10:40 s3://DIAS
2017-11-15 10:40 s3://EOCLOUD
2017-11-15 10:40 s3://EODATA
2017-11-15 10:40 s3://HRSI
2017-11-15 10:40 s3://HRVPP
In this output, s3://DIAS, s3://EOCLOUD, and s3://EODATA are legacy or compatibility bucket names exposed by the same service for EODATA access.
The s3://HRSI and s3://HRVPP buckets are additional top-level data buckets exposed by the same service.
Although the top-level bucket list shows s3://EODATA, the standard EODATA paths used in this article can still be accessed with s3://eodata/.
In WAW3-2, FRA1-2, and WAW4-1, the command:
s3cmd -c /home/eouser/eodata-access ls
should return output similar to this:
2017-11-15 10:40 s3://eodata
In these regions, the EODATA bucket is shown directly as s3://eodata.
To list the contents of the eodata bucket, execute:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/
You should get output similar to this:
DIR s3://eodata/Aqua/
DIR s3://eodata/C3S/
DIR s3://eodata/CAMS/
DIR s3://eodata/CEMS/
DIR s3://eodata/CLMS/
DIR s3://eodata/CLMS_archive/
DIR s3://eodata/Envisat/
DIR s3://eodata/Envisat-ASAR/
DIR s3://eodata/Global-Mosaics/
DIR s3://eodata/Jason-3/
DIR s3://eodata/Landsat-5/
DIR s3://eodata/Landsat-7/
DIR s3://eodata/Landsat-8/
DIR s3://eodata/Landsat-8-ESA/
DIR s3://eodata/Landsat-9/
DIR s3://eodata/SMOS/
DIR s3://eodata/SRTM/
DIR s3://eodata/Sentinel-1/
DIR s3://eodata/Sentinel-1-RTC/
DIR s3://eodata/Sentinel-2/
DIR s3://eodata/Sentinel-3/
DIR s3://eodata/Sentinel-5P/
DIR s3://eodata/Sentinel-6/
DIR s3://eodata/Suomi-NPP/
DIR s3://eodata/Terra/
DIR s3://eodata/TerraAqua/
DIR s3://eodata/auxdata/
You can continue exploring the repository by listing deeper paths. For example, to list the contents of the Sentinel-1 directory, execute:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/Sentinel-1/
The output should look similar to this:
DIR s3://eodata/Sentinel-1/AUX/
DIR s3://eodata/Sentinel-1/SAR/
Ending paths with a slash when using s3cmd ls
When you use s3cmd ls on a directory-like path, end the path with a slash.
For example, this command:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/CLMS/
returns the contents of the CLMS path:
DIR s3://eodata/CLMS/bio-geophysical/
DIR s3://eodata/CLMS/land_cover_use_in_priority_areas/
DIR s3://eodata/CLMS/landcover_landuse/
If you omit the final slash:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/CLMS
the output will show:
DIR s3://eodata/CLMS/
DIR s3://eodata/CLMS_archive/
Listing the contents of a product
To list all files in a product such as S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635, execute:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/
The output should look similar to this:
2023-02-20 07:06 DIROBJ s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/
This output shows that the path behaves like a directory-like object. To inspect its contents or download the whole product, continue by using that path in the examples below. Before relying on a specific internal structure, it is a good idea to verify the actual downloadable contents of that path with s3cmd ls, because different EODATA resources may be exposed in slightly different ways.
Downloading files
The examples below show how to download one existing object and how to download a full product.
Example 1: Downloading a single file
First, list the available files in the directory from which you want to download data:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/Sentinel-5P/AUX/AUX_CTMANA/2023/08/12/
You should get output similar to this:
2025-01-24 00:24 1717236175 s3://eodata/Sentinel-5P/AUX/AUX_CTMANA/2023/08/12/S5P_OPER_AUX_CTMANA_20230812T000000_20230813T000000_20230820T143624.nc
In this case, the product is stored as a single file rather than as a directory containing multiple files. To download it, use the exact object path returned by the ls command:
s3cmd -c /home/eouser/eodata-access get s3://eodata/Sentinel-5P/AUX/AUX_CTMANA/2023/08/12/S5P_OPER_AUX_CTMANA_20230812T000000_20230813T000000_20230820T143624.nc
The download process should begin immediately. The file will be saved to your current working directory.
After the download is complete, you can use the ls command to verify that the file is now present locally.
Note
Do not copy the file size value, such as 1717236175, into the get command. That number is only part of the output of s3cmd ls. Use only the object path that starts with s3://.
This approach is safer than relying on a hard-coded example path, because the structure of some EODATA resources may change over time. By listing the directory first and then copying an existing object path, you make sure that the file you are trying to download is really available.
Example 2: Downloading a full product
To download all files from a product, use s3cmd get with the -r parameter, which allows recursive downloads.
For instance, to download all files from product S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635, execute this command:
s3cmd -c /home/eouser/eodata-access get -r s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635
The download process should begin. The files should be saved to your current working directory. Depending on how the selected EODATA path is exposed, the downloaded result may appear as one or more local files, directories, or both.
If you end the directory path in the command above with a slash, the contents of that directory will be placed in your current working directory. The directory itself will not be created. Therefore, if you want to download the directory together with its contents, do not end the path with a slash.
Using the sync command
You can also use s3cmd sync to download data from the EODATA repository to a selected local directory. This is useful when you want to keep a local copy of a product and restore any missing files later.
Downloading a product with sync
Assume that you want to download the product S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635 from the EODATA repository to a local directory named product-path.
First, create the local directory:
mkdir -p product-path
Then execute:
s3cmd -c /home/eouser/eodata-access sync s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635 ./product-path
The output should contain information about the download progress, like so:
download: 's3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc' -> './product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc' [1 of 2]
2156950172 of 2156950172 100% in 12s 168.37 MB/s done
mkdir: 's3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/' -> './product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/' [2 of 2]
Done. Downloaded 2156950172 bytes in 12.2 seconds, 168.10 MB/s.
Once the operation is completed, you can list the local directory to verify the downloaded content:
ls -R ./product-path
The output should look similar to this:
./product-path:
S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635
S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc
./product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635:
In this example, the synchronized result contains both a local .nc file and a directory created under product-path.
Syncing missing files
You can use the same sync command later to restore files that are missing from the local copy.
To test this behavior, remove the downloaded .nc file:
rm -i ./product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc
Using the -i parameter is optional. It asks for confirmation before removing the file.
To approve the deletion, answer the prompt with y and press Enter.
Then execute the same sync command again:
s3cmd -c /home/eouser/eodata-access sync s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635 ./product-path
The missing file should be downloaded again.
By default, sync restores files that exist in the remote EODATA location but are missing locally. Local files that do not exist in the remote source are left intact unless you explicitly use options that remove them.
Troubleshooting
If this command works:
s3cmd -c /home/eouser/eodata-access ls
but this one fails:
s3cmd -c /home/eouser/eodata-access ls s3://eodata/
check the configuration file carefully.
The most common cause is an incorrect host_bucket value that makes s3cmd build a hostname such as:
eodata.eodata.cloudferro.com
This hostname is incorrect for EODATA access.
For WAW3-2, FRA1-2, and WAW4-1, make sure the configuration uses:
host_base = eodata.cloudferro.com
host_bucket = eodata.cloudferro.com
and for WAW3-1:
host_base = data.cloudferro.com
host_bucket = data.cloudferro.com
If two ls commands that differ only by the final slash return different-looking results, that is usually normal. In s3cmd, the presence or absence of the trailing slash affects how the path is interpreted.
To inspect the connection process in more detail, use debug mode:
s3cmd -d -c /home/eouser/eodata-access ls s3://eodata/
What To Do Next
s3cmd can also be used to access object storage from Creodias.
Learn more here: How to access object storage from Creodias using s3cmd
Also, access EODATA programmatically through Python boto3 library: How to access EODATA using boto3 on Creodias.
Other articles of interest:
Cannot access EODATA on Creodias
How to mount EODATA as a filesystem using Goofys in Linux on Creodias