How to access or list EODATA using boto3 on Creodias
Note
If you are accessing EODATA using NFS, see this article:
Planned NFS shutdown on CF2 and WAW3-1 clouds, starting on 31st of August 2023.
Warning
We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is available here: How to install Python virtualenv or virtualenvwrapper on Creodias
If virtualenv is activated:
(myvenv) [email protected]:~$ pip3 install boto3
If we want to install the package globally:
[email protected]:~$ sudo pip3 install boto3
Two examples for listing products:
Client
low-level service access
generated from service description
exposes botocore client to the developer
typically maps 1:1 with the service API
snake-cased method names (e.g. ListBuckets API => list_buckets method)
import boto3
access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'
host='http://data.cloudferro.com'
s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)
for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2020/11/09/',MaxKeys=30000)['CommonPrefixes']:
print(i)
Resource:
higher-level, object-oriented API
generated from resource description
uses identifiers and attributes
has actions (operations on resources)
exposes subresources and collections
import boto3
access_key='16054e693d864fcc99f3345b9bac1881'
secret_key='d76bef9a48864fa885d0ea68671a166b'
host='http://data.cloudferro.com'
s3 = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')
# list all files in /eodata
for bucket_object in bucket.objects.all():
print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
print(bucket_object.key)
Save your file with the .py extension and run it with the ‘python3 <filename.py>’ command in your terminal. For example:
(boto3) [email protected]:~$ python3 boto_list.py