How to access or list EODATA using boto3 on Creodias

Note

If you are accessing EODATA using NFS, see this article:

Planned NFS shutdown on CF2 and WAW3-1 clouds, starting on 31st of August 2023.

Warning

We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is available here: How to install Python virtualenv or virtualenvwrapper on Creodias

If virtualenv is activated:

(myvenv) [email protected]:~$ pip3 install boto3

If we want to install the package globally:

[email protected]:~$ sudo pip3 install boto3

Two examples for listing products:

Client

  • low-level service access

  • generated from service description

  • exposes botocore client to the developer

  • typically maps 1:1 with the service API

  • snake-cased method names (e.g. ListBuckets API => list_buckets method)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://data.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2020/11/09/',MaxKeys=30000)['CommonPrefixes']:
                print(i)

Resource:

  • higher-level, object-oriented API

  • generated from resource description

  • uses identifiers and attributes

  • has actions (operations on resources)

  • exposes subresources and collections

import boto3

access_key='16054e693d864fcc99f3345b9bac1881'
secret_key='d76bef9a48864fa885d0ea68671a166b'

host='http://data.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
       print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
       print(bucket_object.key)

Save your file with the .py extension and run it with the ‘python3 <filename.py>’ command in your terminal. For example:

(boto3) [email protected]:~$ python3 boto_list.py