Push subscriptions to CREODIAS EODATA catalogue

Push and pull subscriptions allow you to receive notifications notifications about products published in the EODATA catalogue or a subset of such products.

This article covers push subscriptions which are delivered to endpoint specified by the user.

The other type, pull subscriptions, is covered in Pull subscriptions to CREODIAS EODATA catalogue

What We Are Going To Cover

  • Introduction

    • What should a HTTP request contain?

    • Headers

    • Escaping special characters

    • Response to the HTTP call

  • HTTP request - Python example

  • Limitations of subscriptions

  • Push subscription entity description

  • Creating push subscription

  • Status of created subscription

  • What happens after a push subscription has been created?

  • Editing push subscription

  • Listing current subscriptions

  • Deleting a subscription

Prerequisites

No. 1 Account

You need an active CREODIAS account https://new.cloudferro.com/.

No. 2 Ability to send HTTP requests

You need to be able to send HTTP requests, including POST and PATCH, containing headers and body.

No. 3 Knowledge how to filter products

You need to filter products only if you want to get notifications about a subset of the products in the catalogue.

Product filtering uses parameter $filter described in EOData Catalogue API Manual on CREODIAS.

No. 4 Generated Keycloak token

To prove your identity while using push subscriptions, you need a Keycloak token. Here is how to generate it: Using curl and wget to download EODATA products from CREODIAS

Keycloak token is valid only for a limited time; you can generate new token if the current one expires and you want to continue using the subscriptions API.

No. 5 Endpoint which will receive notifications

You need to have a server which will serve as your endpoint. It needs to have a public IP address and a domain attached to it. You also need an open port to which the notifications will be sent.

To read incoming notifications, you need to have a basic HTTP server with support for Basic Authentication. It must be secured via HTTPS, with a certificate provided by a trusted entity. This entity could be for instance one of trusted commercial providers, or Let’s Encrypt (which provides free certificates).

You can configure such a server on, among others, a Linux virtual machine running on CREODIAS cloud which has a Floating IP attached to it.

Introduction

What should a HTTP request contain?

You can create and manage your push subscriptions by sending HTTP requests.

When sending such a HTTP request, always use:

  • Type of request (usually GET, PATCH, DELETE)

  • Headers

  • URL

All these three parts of a HTTP request must be present together. Certain types of HTTP requests must also contain a body.

Headers

Requests in this article should contain the following headers:

{
   "Content-Type": "application/json",
   "Authorization": "Bearer <<access_token>>"
}

where <<access_token>> needs to be replaced with Keycloak token from Prerequisite No. 4 in order to authenticate you.

We will call them “the obligatory headings” and will refer to them as such throughout the article.

Escaping special characters

Requests in this article often contain special characters, for example, ‘ and &. Within a HTTP call, such characters must be escaped properly but the exact way of doing it will depend on the tool used. For example:

URL

Use URL encoding (%27 for ‘ and %22 for “).

JSON

Escape double quotes with backslashes (").

Bash & curl

To escape single quotes, combine single and double quotes. To escape double quotes, use backslash inside double quotes.

PHP

Use json_encode() and urlencode() functions.

Response to the HTTP call

Responses to these requests are either empty or consist of JSON data. Variations also exist: some JSON tools use “ to indicate strings, while others use ‘. Line formatting (or lack thereof) may vary.

HTTP request - Python example

In Python, the best way to send a HTTP request is using Python module requests. For JSON, you usually use pprint instead of print.

Make sure to have the requests module installed.

As mentioned, in this article we will show excerpts of code and you should add the mandatory rest of the code. For your reference, here is full code for request described in section Listing current subscriptions later in this article:

import pprint
import requests
import json

access_token = "<<your_access_token>>"

headers = {
     "Content-Type": "application/json",
     "Authorization": f"Bearer {access_token}",
 }

url = "https://datahub.creodias.eu/odata/v1/Subscriptions/Info"

response = requests.get(url=url, headers=headers)

try:
    pprint.pprint(response.json())
except requests.exceptions.JSONDecodeError:
    print(response.text)

Assign Keycloak token obtained by following Prerequisite No. 5 to variable access_token.

Save the file as http_request_test.py.

To run the script in that file, execute the following command (assuming you are in the same folder as the script):

python3 http_request_test.py

The script should return

  • response code,

  • response reason and

  • response text (if any).

Limitations of subscriptions

Your account can only have 1 running subscription. It can be either a push subscription (which is described in this article) or a pull subscription (which is outside of scope of this article).

Total number of subscriptions (both running and paused) is 10.

Push subscription entity description

pull subscription entity description table
Property name Type Description Example
Id Guid It is a universally unique identifier (UUID). The Id is a local identifier for a Subscription instance within the Catalogue, assigned upon Subscription creation. 9249dde5-4838-4a34-8925-bac8c1d53f09
Status Subscription Status enumeration The allowed values of a subscription status are:
  • running (notifications are being sent)
  • paused (notifications are temporarily not being sent)
  • cancelled (suspended and will be deleted)
The default value set to 'Status', if not provided by the user, is 'running'. 		running
SubscriptionEvent Subscription Event enumeration The subscription event to be monitored and for which notification is provided:
  • created
The default value set to SubscriptionEvent, if not provided by the user, is "created". For "SubscriptionEvent" = "created" the notifications about newly added products to the EOData Catalogue will be sent. 		created
FilterParam String The filter parameters of the Subscription (refers to the $filter= parameter of any The same filtering parameters as described for EOData Catalogue are available. Collection/Name eq ‘SENTINEL-1’ and Attributes/OData.CSC.StringAttribute/any (att:att/Name eq ‘productType’ and att/OData.CSC.StringAttribute/Value eq ‘IW_SLC__1S’)
SubmissionDate DateTimeOffset Date and time at which the Subscription was received by the Catalogue. Time is in UTC in the format YYYY-MMDDThh:mm:ss.sssZ 2024-01-17T09:13:04.654Z
LastNotificationDate DateTimeOffset Date and time at which the Subscription was received by the Catalogue. Time is in UTC in the format YYYY-MMDDThh:mm:ss.sssZ 2024-01-17T09:50:10.654Z
StageOrder Boolean Automatically orders the staging of products fulfilling the subscription. Only used if SubscriptionEvent = created Currently, the order of staging products is not feasible as all products in EOData Catalogue have status set to ‘Online’ and can be accessed immediately without setting order. true
Priority Int64 Priority of the created orders resulting from the subscription. Currently automatically fixed to ‘1’ without the possibility to change the value. 1
NotificationEndpoint String User's Endpoint used by the EOData Catalogue for subscription notifications. https://notification-endpoint.example.com:4000
NotificationEpUsername String The username associated with the Endpoint URI. It is mandatory if NotificationEndpoint requires authentication. mylocalusername
NotificationEpPassword String The password associated with the Endpoint URI. It is mandatory if NotificationEndpoint requires authentication. My%Password#789


Creating push subscription

Send a POST request to the following URL:

https://datahub.creodias.eu/odata/v1/Subscriptions

REMINDER: Include the obligatory headers.

Include the following body to this request (replace the values there as instructed):

{
    "StageOrder": true,
    "FilterParam": "Collection/Name eq 'SENTINEL-1' and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq 'IW_SLC__1S')",
    "Priority": 1,
    "Status": "running",
    "SubscriptionEvent": [
        "created"
    ],
    "NotificationEndpoint": "https://example.com:8000/",
    "NotificationEpUsername": "NotificationEndpointUser",
    "NotificationEpPassword": "My@!Passw0rd"
}

In this body, modify the values of keys as follows:

Key

Explanation

FilterParam

Appropriate filter query, as described in Prerequisite No. 3.

NotificationEndpoint

The URL of your endpoint, alongside with the port under which it is available.

NotificationEpUsername

The username used to authenticate on your notification endpoint.

NotificationEpPassword

The username used to authenticate on your notification endpoint.

You can also get notifications about all products published in the catalogue, not only a subset of them. To that end, you can either

  • leave the value of key FilterParam empty, or

  • do not include that key with your request at all.

NotificationEpUsername and NotificationEpPassword should be secure and not the same as credentials used for your CREODIAS account.

Query provided as value of key FilterParam in example above should be valid, so you can use it to test push notifications.

REMINDER: Include the obligatory headers.

When you’re ready, send the request.

Response code to that request should be 201 Created. Response should contain JSON data like this:

{
    "StageOrder": true,
    "FilterParam": "Collection/Name eq 'SENTINEL-2' and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq 'S2MSI2A')",
    "Priority": 1,
    "NotificationEndpoint": "https://example.com/catalogue-notifications:8000",
    "NotificationEpUsername": "NotificationEndpointUser",
    "NotificationEpPassword": "My@!Passw0rd",
    "Status": "running",
    "SubscriptionEvent": [
      "created"
    ]
}

Status of created subscription

A subscription (push or pull) can have one of the following statuses:

  • running - the subscription receives new notifications

  • paused - the subscription temporarily does not receive new notifications

  • cancelled - the subscription no longer receives new notifications and will be removed

If you create a subscription without specifying its status, the subscription will be created with the following Status: running.

If you already have a subscription with this status, this operation will fail since one user can only have one active subscription at any time.

To resolve, you can pause your current subscription (more information about it is later on in this article) and then create a new subscription.

If you, however, don’t want to pause your current subscription but want to create a new, inactive subscription, you can create a new subscription with status paused. To do that, while following section Creating push subscription, use paused for Status, instead of running.

"Status": "paused",

What happens after a push subscription has been created?

If you send a successful HTTP request, the subscription should be created. Each time a new product is published in the catalogue (assuming it meets your search criteria), your notification endpoint (see Prerequisite No. 5) should receive a notification.

The notification should contain information about a product and could look like this:

{
  "@odata.context": "$metadata#Notification/$entity",
  "SubscriptionEvent": "created",
  "ProductId": "0dd083d4-360e-4693-9791-1215da02c3a3",
  "ProductName": "S1A_S3_GRDH_1SDV_20230801T152912_20230801T152931_049683_05F970_0441_COG.SAFE",
  "SubscriptionId": "05d02462-6685-46e6-a1a4-bd1749330e43",
  "NotificationDate": "2024-04-03T10:08:38.507Z",
  "Value": {
    "@odata.context": "$metadata#Products(Attributes())(Assets())",
    "@odata.mediaContentType": "application/octet-stream",
    "Id": "0dd083d4-360e-4693-9791-1215da02c3a3",
    "Name": "S1A_S3_GRDH_1SDV_20230801T152912_20230801T152931_049683_05F970_0441_COG.SAFE",
    "ContentType": "application/octet-stream",
    "ContentLength": 161991897,
    "OriginDate": "2023-08-01T16:54:32.762Z",
    "Checksum": [
      {}
    ],
    "ContentDate": {
      "Start": "2023-08-01T15:29:12.689Z",
      "End": "2023-08-01T15:29:31.839Z"
    },
    "Footprint": "geography'SRID=4326;POLYGON ((43.033947 -12.177707, 43.756836 -12.014938, 43.49361 -10.859819, 42.773415 -11.021481, 43.033947 -12.177707))'",
    "GeoFootprint": {
      "type": "Polygon",
      "coordinates": [
        [
          [
            43.033947,
            -12.177707
          ],
          [
            43.756836,
            -12.014938
          ],
          [
            43.49361,
            -10.859819
          ],
          [
            42.773415,
            -11.021481
          ],
          [
            43.033947,
            -12.177707
          ]
        ]
      ]
    },
    "Attributes": [
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "origin",
        "Value": "CLOUDFERRO",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.IntegerAttribute",
        "Name": "datatakeID",
        "Value": 391536,
        "ValueType": "Integer"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "timeliness",
        "Value": "Fast-24h",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.IntegerAttribute",
        "Name": "cycleNumber",
        "Value": 298,
        "ValueType": "Integer"
      },
      {
        "@odata.type": "#OData.CSC.IntegerAttribute",
        "Name": "orbitNumber",
        "Value": 49683,
        "ValueType": "Integer"
      },
      {
        "@odata.type": "#OData.CSC.IntegerAttribute",
        "Name": "sliceNumber",
        "Value": 1,
        "ValueType": "Integer"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "productClass",
        "Value": "S",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "processorName",
        "Value": "Sentinel-1 IPF",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "orbitDirection",
        "Value": "ASCENDING",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.DateTimeOffsetAttribute",
        "Name": "processingDate",
        "Value": "2023-08-01T16:43:32.478663+00:00",
        "ValueType": "DateTimeOffset"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "operationalMode",
        "Value": "SM",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "processingLevel",
        "Value": "LEVEL1",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "swathIdentifier",
        "Value": "S3",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "processingCenter",
        "Value": "Production Service-SERCO",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "processorVersion",
        "Value": "003.61",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "platformShortName",
        "Value": "SENTINEL-1",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "instrumentShortName",
        "Value": "SAR",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.IntegerAttribute",
        "Name": "relativeOrbitNumber",
        "Value": 86,
        "ValueType": "Integer"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "polarisationChannels",
        "Value": "VV&VH",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "platformSerialIdentifier",
        "Value": "A",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.DoubleAttribute",
        "Name": "startTimeFromAscendingNode",
        "Value": 5712238,
        "ValueType": "Double"
      },
      {
        "@odata.type": "#OData.CSC.DoubleAttribute",
        "Name": "completionTimeFromAscendingNode",
        "Value": 5731388,
        "ValueType": "Double"
      },
      {
        "@odata.type": "#OData.CSC.StringAttribute",
        "Name": "productType",
        "Value": "S3_GRDH_1S-COG",
        "ValueType": "String"
      },
      {
        "@odata.type": "#OData.CSC.DateTimeOffsetAttribute",
        "Name": "beginningDateTime",
        "Value": "2023-08-01T15:29:12.689Z",
        "ValueType": "DateTimeOffset"
      },
      {
        "@odata.type": "#OData.CSC.DateTimeOffsetAttribute",
        "Name": "endingDateTime",
        "Value": "2023-08-01T15:29:31.839Z",
        "ValueType": "DateTimeOffset"
      }
    ],
    "ModificationDate": "2023-08-04T06:44:27.481Z",
    "PublicationDate": "2023-08-04T06:44:27.481Z",
    "Online": true,
    "EvictionDate": "",
    "S3Path": "/eodata/Sentinel-1/SAR/S3_GRDH_1S-COG/2023/08/01/S1A_S3_GRDH_1SDV_20230801T152912_20230801T152931_049683_05F970_0441_COG.SAFE",
    "Assets": []
  }
}

Each push notification contains the following keys:

Pull notification entity description table
Property name Type Description Example
ProductId Guid It is a universally unique identifier (UUID) of the product being notified. 8d654e59-d7b6-4a53-b086-c9de764e506b
ProductName String It is the name of the product being notified assigned within the Catalogue. S2A_MSIL2A_20240129T062121_N0510_R034_T44VMN_20240129T093752.SAFE
SubscriptionId Guid It is a universally unique identifier (UUID). The Id is a local identifier for the Subscription instance within the Catalogue, assigned upon Subscription creation. 991c4730-cf6f-432a-9f6c-47be0230ff4
SubscriptionEvent Subscription Event enumeration The subscription event to be monitored and for which notification is provided:
  • created
The default value set to SubscriptionEvent, if not provided by the user, is "created". For "SubscriptionEvent" = "created" the notifications about newly added products to the EOData Catalogue will be sent. 		created
NotificationDate DateTimeOffset Date and time at which the notification was generated. Time is in UTC in the format YYYYMM-DDThh:mm:ss.sssZ 2024-01-29T10:59:08.698Z
SubmissionDate DateTimeOffset Date and time at which the Subscription was received by the Catalogue. Time is in UTC in the format YYYY-MMDDThh:mm:ss.sssZ 2024-01-17T09:13:04.654Z
LastNotificationDate DateTimeOffset Date and time at which the Subscription was received by the Catalogue. Time is in UTC in the format YYYY-MMDDThh:mm:ss.sssZ 2024-01-17T09:50:10.654Z
StageOrder Boolean Automatically orders the staging of products fulfilling the subscription. Only used if SubscriptionEvent = created Currently, the order of staging products is not feasible as all products in EOData Catalogue have status set to ‘Online’ and can be accessed immediately without setting order. true
Priority Int64 Priority of the created orders resulting from the subscription. Currently automatically fixed to ‘1’ without the possibility to change the value. 1
NotificationEndpoint String User's Endpoint used by the EOData Catalogue for subscription notifications. https://notification-endpoint.example.com:4000
NotificationEpUsername String The username associated with the Endpoint URI. It is mandatory if NotificationEndpoint requires authentication. mylocalusername
NotificationEpPassword String The password associated with the Endpoint URI. It is mandatory if NotificationEndpoint requires authentication. My%Password#789
value Dictionary It contains a standard Catalogue API response with all metadata and content details about a product.


Editing push subscription

In this section you will learn how to edit an existing push subscription. This might be needed if, say, you want to pause or unpause such a subscription, or you know that you will soon lose access to your current notification endpoint.

For push subscriptions, you can edit the following properties:

Key

Explanation

Status

Status of the subscriptions - running, paused or cancelled.

NotificationEndpoint

The URL of your endpoint, alongside with the port under which it is available

NotificationEpUsername

The username configured on your endpoint used to send notifications to it

NotificationEpPassword

The password configured on your endpoint used to send notifications to it

To perform such an edit, send a PATCH request to the URL below. In it, replace <<subscription_id>> with the ID of your subscription:

https://datahub.creodias.eu/odata/v1/Subscriptions(<<subscription_id>>)

As a body, send a JSON data structure containing all parameters which you want to change. For example, if you want to change notification endpoint to https://example.org and password used to authenticate to MyTestPassword321, use this body:

{
    "NotificationEndpoint": "https://example.org",
    "NotificationEpPassword": "MyTestPassword321"
}

REMINDER: Include the obligatory headers.

Listing current subscriptions

To list subscriptions which are currently associated with your account, send GET request to the following URL:

https://datahub.creodias.eu/odata/v1/Subscriptions/Info

REMINDER: Include the obligatory headers.

The status of the response should be 200 OK. You should get a list of your current subscriptions. This includes:

  • push subscriptions, as well as

  • pull subscriptions.

For example, it can look similarly to this:

[
  {
    "@odata.context": "$metadata#Subscriptions/$entity",
    "FilterParam": "",
    "Id": "6a52c01c-9c43-4624-875e-561a2752b710",
    "LastNotificationDate": "2025-06-17T09:30:49.959Z",
    "NotificationEndpoint": "https://example.com:8000",
    "Priority": 1,
    "StageOrder": false,
    "Status": "running",
    "SubmissionDate": "2024-05-29T10:01:05.793Z",
    "SubscriptionEvent": [
      "created"
    ]
  },
  {
    "@odata.context": "$metadata#Subscriptions/$entity",
    "FilterParam": "Collection/Name eq 'SENTINEL-5P' and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq 'L2__AER_LH')",
    "Id": "9b86baed-64c1-47cf-94ae-b586666da622",
    "Priority": 1,
    "StageOrder": false,
    "Status": "paused",
    "SubmissionDate": "2024-05-15T14:25:43.940Z",
    "SubscriptionEvent": [
      "created"
    ]
  }
]

In this example, we see that the user has two subscriptions.

The first one is a running push subscription. It includes a key NotificationEndpoint - its value is the URL of the notification endpoint. Also, it has key LastNotificationDate which shows when was the last time that a notification was delivered successfully. The value for key FilterParam is in this case an empty string - it means that the user should receive notifications about all products published in the catalogue.

The second subscription is a pull subscription. You can learn that it is a pull subscription from the fact that it does not have key NotificationEndpoint. This time, the value of key FilterParam is not empty and contains a meaningful query for the catalogue which means that it should receive notifications about only a subset of products published in the catalogue.

As stated previously, pull subscriptions are outside of scope of this article.

If you don’t currently have any subscriptions, the query should return an empty list:

[]

Deleting a subscription

Send the DELETE request to the URL below. In it, replace <<subscription_id>> with the ID of the subscription you want to delete.

https://datahub.creodias.eu/odata/v1/Subscriptions(<<subscription_id>>)

REMINDER: Include the obligatory headers.

You should get response status: 204 No Content.

Listing subscriptions (as explained in section Listing current subscriptions of this article) should show that this subscription no longer exists.

What To Do Next

Once you have received a notification, you might want to download that product. The ways of achieving this goal include:

These articles cover some of the available ways of processing EODATA: