How to access or list EODATA using boto3 on CREODIAS

LEGACY ARTICLE

This article is marked as a legacy document and may not reflect the latest information. Please refer to the following article: How to access EODATA using boto3 on CREODIAS

Warning

We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is available here: How to install Python virtualenv or virtualenvwrapper on CREODIAS

If virtualenv is activated:

(myvenv) eouser@vm01:~$ pip3 install boto3

If we want to install the package globally:

eouser@vm01:~$ sudo pip3 install boto3

Two examples for listing products are presented below. You will need EODATA access key and secret key to execute that code. Obtain them by following this article: How to get credentials used for accessing EODATA on a cloud VM on CREODIAS. In each example below, replace the contents of the access_key variable with your access key and the contents of the secret_key variable with your secret key, respectively.

Client

low-level service access

generated from service description

exposes botocore client to the developer

typically maps 1:1 with the service API

snake-cased method names (e.g. ListBuckets API => list_buckets method)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='https://eodata.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2022/05/15/',MaxKeys=30000)['CommonPrefixes']:
   print(i)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://data.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2022/05/15/',MaxKeys=30000)['CommonPrefixes']:
   print(i)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://eodata.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2022/05/15/',MaxKeys=30000)['CommonPrefixes']:
   print(i)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://eodata.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2022/05/15/',MaxKeys=30000)['CommonPrefixes']:
   print(i)

Resource:

higher-level, object-oriented API

generated from resource description

uses identifiers and attributes

has actions (operations on resources)

exposes subresources and collections

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='https://eodata.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
   print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
   print(bucket_object.key)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://data.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
   print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
   print(bucket_object.key)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://eodata.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
   print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
   print(bucket_object.key)

import boto3

access_key='YOUR_ACCESS_KEY'
secret_key='YOUR_SECRET_KEY'

host='http://eodata.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
   print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
   print(bucket_object.key)

Save your file with the .py extension and run it with the ‘python3 <filename.py>’ command in your terminal. For example:

(boto3) eouser@vm01:~$ python3 boto_list.py