How to download EODATA files using s3cmd on CREODIAS
At first, please verify your connection with your EODATA buckets by using the list command.
eouser@vm01:~$ s3cmd ls
You should obtain this output.
2017-11-15 10:40 s3://DIASac 2017-11-15 10:40 s3://EOCLOUD 2017-11-15 10:40 s3://HRSI
If you cannot see these buckets, please complete all the steps from the following tutorial: How to access EODATA and Object Storage using s3cmd on Linux on CREODIAS.
Downloading an EOData product
An entire EOData product is available to download as a whole directory, not just a single file. Therefore, you have to use –recursive parameter in order to avoid skipping of any files.
Example of downloading a Sentinel-2 product:
s3cmd get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/
The above command will download the S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE product to your local folder from which you are issuing the command.
Files can be also redirected to a specific local folder. For our purposes it would be a downloads directory. Add the path to your local destination folder to the command.
eouser@vm01:~$ mkdir downloads eouser@vm01:~$ s3cmd -d get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/ downloads/
If nothing happens or you could not download a product properly, please use -d switch to see a debug output for further diagnosis:
s3cmd -d get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/
For syncing, you can use the s3cmd sync command. In synchronizes a local directory tree with S3.
Firstly we can create a new directory for our product and download that product using the s3cmd sync command on that directory.
Now we will present the sync method in practice.
Go to your target directory and list the files.
For our purposes we will remove the measurement directory.
In order to sync the data with your local folder, you have to put a dot (. → location of the current directory) as a last character.
After invoking the s3cmd sync command all the missing files will be found and restored, as presented below.
Done. Downloaded 5347560568 bytes in 23.7 seconds, 214.86 MB/s. eouser@vm01:~/example-product$ ls S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE-report-20181002T024920.pdf annotation manifest.safe measurement preview support