Using Python to send EODATA Catalogue requests on CREODIAS

This article covers an example way of sending EODATA Catalogue requests described in the article:

URLs used for sending these requests might cause problems when using with, say, curl command on Bash. For example, the following URL:

https://datahub.creodias.eu/odata/v1/Products?$filter=not contains(Name,'S2') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T00:10:00.000Z

contains among others a dollar sign, spaces and brackets. They need to be appropriately escaped and/or encoded to make sure that the request is valid. For instance, Bash might interpret $filter as variable $filter when it is, in fact, part of URL.

In this article, we circumvent this problem by

  • pasting the URL to one file,

  • pasting the body of request to another file (if needed),

  • creating another file with Python code

  • and then executing it.

Prerequisites

No. 1 Installed Python environment

This article is written for simple Python environments running on Ubuntu 22.04 and Windows Server 2022. Other operating systems (such as Windows 11) and environments (such as PyCharm) might also work, but are outside of scope of this article.

The following articles contain sections on how to install Python:

If you are running Ubuntu, you can also use virtualenvwrapper: How to install Python virtualenv or virtualenvwrapper on CREODIAS

No. 2 Python library called requests

You need to install Python package called requests. You can do that using pip:

pip install requests

On Ubuntu 22.04, you can also install this library globally using apt:

sudo apt install python3-requests

What We Are Going To Cover

  • Sending GET requests

  • Sending POST requests

Sending GET requests

Most links found in API manuals are GET requests which do not have any body.

Create script named send_get.py and enter the following content to it:

Send GET request to server
import requests
from requests.utils import requote_uri
# Open URL from file
url = open('link1.txt').read()
# Encode URL
url_encoded = requote_uri(url)
# Remove unnecessasary characters from encoded URL
url_encoded_cleared = url_encoded.replace('%0A', '')
# Obtain and print the response
response = requests.get(url_encoded_cleared)
print(response.json())

Create file named link1.txt in the same folder as the script send_get.py which contains the link you want to access. This file can have only one URL.

Navigate to the folder in which both script and link1.txt file are located using your terminal and run the script:

python3 send_get.py

Output should consist of the response from Catalogue API.

Example 1: OData API

Enter the link below to file link1.txt (note that this file must have exactly one URL):

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z

This link searches for products which

  • contain phrase S1A in their name, and

  • have their starting sensing date is later than 2022-05-03T00:00:00.000Z and earlier than 2022-05-21T00:00:00.000Z.

It should return top 20 results.

Execute the script. In this example, the GNOME Terminal and gedit on Ubuntu 22.04 were used. Note that gedit describes this code as Python 2 even though actually Python 3 was used.

../_images/using-python-eodata-catalogue-requests-01_creodias.png

Note that screenshot above contains only the beginning of the response.

Example 2: Deleted Products ODATA API

Enter the link below to file link1.txt (note that this file must have exactly one URL):

https://datahub.creodias.eu/odata/v1/DeletedProducts?$filter=not contains(Name,'S1') and DeletionDate gt 2023-04-01T00:00:00.000Z and DeletionDate lt 2023-05-30T23:59:59.999Z&$orderby=DeletionDate

This link searches for deleted products which

  • have a name that does not contain phrase S1, and

  • have deletion date set to later than 2023-04-01T00:00:00.000Z and earlier than 2023-05-30T23:59:59.999Z

The results are sorted by deletion date. Top 20 results should be returned.

Execute the script:

../_images/using-python-eodata-catalogue-requests-02_creodias.png

Note that screenshot above contains only the beginning of the response.

Example 3: OpenSearch API

Enter the link below to file link1.txt (note that this file must have exactly one URL):

https://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2021-07-01&completionDate=2021-07-31

This link searches for Sentinel-2 products which have 2021-07-01 and 2021-07-31 as their starting and finishing observation date, respectively.

Execute the script:

../_images/using-python-eodata-catalogue-requests-03_creodias.png

Note that screenshot above contains only the beginning of the response.

Sending POST requests

Create a file named send_post.py and enter the following code to it:

Send POST request to server
import requests
import json
from requests.utils import requote_uri
# Open URL from file
url = open('link2.txt').read()
# Open body from file and convert it to JSON object
query = open('body.json')
body = json.load(query)
# Encode URL
url_encoded = requote_uri(url)
# Remove unnecessasary characters from encoded URL
url_encoded_cleared = url_encoded.replace('%0A', '')
# Obtain and print the response
response = requests.post(url_encoded_cleared, json=body)
print(response.json())

Script presented above works similarly as the one presented in the previous section.

The URL should, however, be this time put to file link2.txt and not link1.txt.

Body of the request should be put in file body.json.

All files mentioned in this section should be in the same folder.

Navigate to that folder using your terminal and run Python code:

python3 send_post.py

Output should consist of the response from Catalogue API.

Example: OData API

Let’s use link from section Query by list of /eodata/OData-API-Manual-for-Creodias-EODATA-Catalogue to search for multiple product

names at once:

https://datahub.creodias.eu/odata/v1/Products/OData.CSC.FilterList

Place this link to file link2.txt.

Put the body found in that article to file body.json:

{
    "FilterProducts":
    [
        {"Name": "S1A_IW_GRDH_1SDV_20141031T161924_20141031T161949_003076_003856_634E.SAFE"},
        {"Name": "S3B_SL_1_RBT____20190116T050535_20190116T050835_20190117T125958_0179_021_048_0000_LN2_O_NT_003.SEN3"},
        {"Name": "xxxxxxxx.06.tar"}
    ]
}

Execute the script:

../_images/using-python-eodata-catalogue-requests-04_creodias.png