EOData Catalogue API Manual on CREODIAS
Attention
This is the new data Catalogue API, with Odata and OpenSearch interface. The latest change was due to the launch of Creodias 2.0, on Monday 26 June 2023.
Two-factor authentication is enabled on Creodias 2.0 and you will need it to download products. To that end, you should have already had installed an application to generate the required TOTP code (a six-digit number which changes every 30 seconds and serves as an additional proof of authentication). See article How to activate OpenStack CLI access to CREODIAS cloud using one- or two-factor authentication.
OData
Query structure
As a general note, Odata query consists of elements which in this documentation are called “options”. Interface supports the following search options:
filter
orderby
top
skip
count
expand
Search options should always be preceded with $ and consecutive options should be separated with &.
Consecutive filters within filter option should be separated with and or or. Not operator can also be used e.g.:
https://datahub.creodias.eu/odata/v1/Products?$filter=not contains(Name,'S2') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T00:10:00.000Z&$orderby=ContentDate/Start&$top=100
ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z
Filter option
Query by name
To search for a specific product by its exact name:
https://datahub.creodias.eu/odata/v1/Products?$filter=Name eq 'S1A_IW_GRDH_1SDV_20141031T161924_20141031T161949_003076_003856_634E.SAFE'
To search for products containing “S1A” in their names:
https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z
Alternatively to contains, endswith and startswith can be used, to search for products ending or starting with the provided string.
Query by list
In case a user desires to search for multiple products by name in one query, POST method can be used:
https://datahub.creodias.eu/odata/v1/Products/OData.CSC.FilterList
Request body:
{
"FilterProducts":
[
{"Name": "S1A_IW_GRDH_1SDV_20141031T161924_20141031T161949_003076_003856_634E.SAFE"},
{"Name": "S3B_SL_1_RBT____20190116T050535_20190116T050835_20190117T125958_0179_021_048_0000_LN2_O_NT_003.SEN3"},
{"Name": "xxxxxxxx.06.tar"}
]
}
Two results are returned, as there is no product named xxxxxxxx.06.tar.
Query Collection of Products
To search for products within a specific collection:
https://datahub.creodias.eu/odata/v1/Products?$filter=Collection/Name eq 'SENTINEL-2' and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T00:11:00.000Z
The following collections are currently available:
SENTINEL-1
SENTINEL-2
SENTINEL-3
SENTINEL-5P
SENTINEL-6
SENTINEL-1-RTC
LANDSAT-5
LANDSAT-7
LANDSAT-8
SMOS
TERRAAQUA
COP-DEM
ENVISAT
S2GLC
Query by Publication Date
To search for products published between two dates:
https://datahub.creodias.eu/odata/v1/Products?$filter=PublicationDate gt 2019-05-15T00:00:00.000Z and PublicationDate lt 2019-05-16T00:00:00.000Z
To define inclusive interval ge and le parameters can be used:
https://datahub.creodias.eu/odata/v1/Products?$filter=PublicationDate ge 2019-05-15T00:00:00.000Z and PublicationDate le 2019-05-16T00:00:00.000Z
Query by Sensing Date
To search for products acquired between two dates:
https://datahub.creodias.eu/odata/v1/Products?$filter=ContentDate/Start gt 2019-05-15T00:00:00.000Z and ContentDate/Start lt 2019-05-16T00:00:00.000Z
Usually, there are two parameters describing the ContentDate (Acquisition Dates) for a product - Start and End. Depending on what the user is looking for, these parameters can be mixed, e.g.:
https://datahub.creodias.eu/odata/v1/Products?$filter=ContentDate/Start gt 2019-05-15T00:00:00.000Z and ContentDate/End lt 2019-05-15T00:05:00.000Z
Query by Geographic Criteria
To search for products intersecting the specified polygon:
https://datahub.creodias.eu/odata/v1/Products?$filter=OData.CSC.Intersects(area=geography'SRID=4326;POLYGON((12.655118166047592 47.44667197521409,21.39065656328509 48.347694733853245,28.334291357162826 41.877123516783655,17.47086198383573 40.35854475076158,12.655118166047592 47.44667197521409))') and ContentDate/Start gt 2022-05-20T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z
To search for products intersecting the specified point:
https://datahub.creodias.eu/odata/v1/Products?$filter=OData.CSC.Intersects(area=geography%27SRID=4326;POINT(-0.5319577002158441%2028.65487836189358)%27)
Disclaimers:
MULTIPOLYGON is currently not supported.
Polygon must start and end with the same point.
Coordinates must be given in EPSG 4326
Query by attributes
To search for products by attributes it is necessary to build a filter with the following structure:
Attributes/OData.CSC.ValueTypeAttribute/any(att:att/Name eq '[Attribute.Name]' and att/OData.CSC.ValueTypeAttribute/Value eq '[Attribute.Value]')
where
ValueTypeAttribute can take the following values: * StringAttribute * DoubleAttribute * IntegerAttribute * DateTimeOffsetAttribute
[Attribute.Name] is the attribute name which can take multiple values, depending on collection (Attachment 1 - Coming soon)
eq before [Attribute.Value] can be substituted with le, lt, ge, gt in case of Integer, Double or DateTimeOffset Attributes
[Attribute.Value] is the specific value that the user is searching for
To get products with cloudCover<40% between two dates:
https://datahub.creodias.eu/odata/v1/Products?$filter=Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le 40.00) and ContentDate/Start gt 2022-01-01T00:00:00.000Z and ContentDate/Start lt 2022-01-03T00:00:00.000Z&$top=10
To get products with cloudCover< 10% and productType=S2MSI2A and ASCENDING orbitDirection between two dates:
https://datahub.creodias.eu/odata/v1/Products?$filter=Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value lt 10.00) and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq 'S2MSI2A') and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq 'ASCENDING') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T04:00:00.000Z&$top=10
Orderby option
Orderby option can be used to order the products in an ascending (asc) or descending (desc) direction. If asc or desc not specified, then the resources will be ordered in ascending order.
To order products by ContentDate/Start in a descending direction:
https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A_EW_GRD') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T03:00:00.000Z&$orderby=ContentDate/Start desc
By default, if orderby option is not used, the results are not ordered. If orderby option is used, additional orderby by id is also used, so that the results are fully ordered and no products are lost while paginating through the results.
The acceptable arguments for this option: ContentDate/Start, ContentDate/End, PublicationDate, ModificationDate, in directions: asc, desc
Top option
Top option specifies the maximum number of items returned from a query.
To limit the number of results:
https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$top=100
The default value is set to 20.
The acceptable arguments for this option: Integer <0,1000>
Skip option
Skip option can be used to skip a specific number of results. Exemplary application of this option would be paginating through the results, however for performance reasons, we recommend limiting queries with small time intervals as a substitute of using skip in a more generic query.
To skip a specific number of results:
https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$skip=23
The default value is set to 0.
Whenever a query results in more products than 20 (default top value), the API provides a nextLink at the bottom of the page:
"@odata.nextLink": "http://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A_EW_GRD')+and+ContentDate/Start+gt+2022-05-03T00:00:00.000Z+and+ContentDate/Start+lt+2022-05-03T12:00:00.000Z&$skip=20"
The acceptable arguments for this option: Integer <0,10000>
Count option
Count option enables users to get the exact number of products matching the query. This option is disabled by default to accelerate the query performance.
To get the exact number of products for a given query:
https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$count=True
The acceptable arguments for this option: True, true, 1, False, false, 0
Expand option
Expand option enables users to see full metadata of each returned result.
To see the metadata of the results:
https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$expand=Attributes
The acceptable arguments for this option: Attributes
Listing product Nodes
Product content can be discovered by accessing the following url patterns:
https://datahub.creodias.eu/odata/v1/Products(<PRODUCT_UUID>)/Nodes
https://datahub.creodias.eu/odata/v1/Products(<PRODUCT_UUID>)/Nodes(<NODE_NAME>)/Nodes
https://datahub.creodias.eu/odata/v1/Products(<PRODUCT_UUID>)/Nodes(<NODE_NAME>)/Nodes(<NODE_NAME>)/Nodes
where
- <PRODUCT_UUID>
is ID of the product obtained by search query, and
- <NODE_NAME>
is name of element inside product returned from previous listing response.
Only nodes that are folders can have their contents listed. Attempting to list Nodes for a file results returning of empty list. Listing Nodes feature is available for both authorized and unauthorized users
Example:
Accessed url:
https://datahub.creodias.eu/odata/v1/Products(db0c8ef3-8ec0-5185-a537-812dad3c58f8)/Nodes
Response:
{
"result":[
{
"Id":"S2A_MSIL1C_20180927T051221_N0206_R033_T42FXL_20180927T073143.SAFE",
"Name":"S2A_MSIL1C_20180927T051221_N0206_R033_T42FXL_20180927T073143.SAFE",
"ContentLength":0,
"ChildrenNumber":9,
"Nodes":{
"uri":"https://datahub.creodias.eu/odata/v1/Products(db0c8ef3-8ec0-5185-a537-812dad3c58f8)/Nodes(S2A_MSIL1C_20180927T051221_N0206_R033_T42FXL_20180927T073143.SAFE)/Nodes"
}
}
]
}
Every Listed Node has “uri” field, which lists its children.
Product Download
To download products, use Id of the product returned by search query, e.g.:
https://datahub.creodias.eu/odata/v1/Products(a6212de3-f2e4-58c2-840b-7f42c3c8c612)/$value
Only authorized users are allowed to download products.
Two-factor authentication is enabled on Creodias 2.0. As mentioned in the beginning of this article, you need to add your TOTP code as an additional parameter for obtaining the Keycloak token.
Use the following code:
KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' -d 'username=<USER>' -d 'password=<PASSWORD>' -d 'grant_type=password' -d 'totp=<TOTP>' 'https://identity.cloudferro.com/auth/realms/Creodias-new/protocol/openid-connect/token' | python3 -m json.tool | grep "access_token" | awk -F\" '{print $4}')
Replace <USER> and <PASSWORD> with credentials to your CREODIAS account. Replace <TOTP> with the TOTP six-digit code generated by your two-factor authentication application.
The instructions for downloading EODATA do not change.
OpenSearch
Using OpenSearch interface to query Data Catalogue
Note
Due to the fact that offset is not a recommended form of searching repository pages, we had to implement limit to a maximum of 200k. The requests over the limit will be rejected with the code 400. We encourage you to limit your inquiries by geographic or temporal area.
All queries may be executed as simple HTTP-Get calls, by typing the query in web browser address line, by using any HTTP client, e.g. curl or wget, or from inside of users’ program. The database is accessible free and anonymously (open for anonymous access for everyone, no authorization is used) It may be accessed both from the internal network (virtual machines in Creodias) and from outside, e.g. your home computer. Note, that the actual EO data themselves are restricted to authorized users, only the Data Catalogue is open.
General Rules
The queries produce results in JSON format. Base url:
http://datahub.creodias.eu/resto/api/collections/search.json?
Most queries are case-sensitive.
Collections
The data are organized in so-called collections, corresponding to various satellites. A query may search for data in all collections, or in one particular collection only. If only one satellite is in the field of interest, the second approach is faster and more efficient, than filtering the general query. For example, to find 10 most recent Sentinel-2 products with cloud cover below 10%, the query should look like:
$ wget -O - "http://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?cloudCover=[0,10]&startDate=2022-06-11&completionDate=2022-06-22&maxRecords=10"
while if the collection field is missing in the URL, the products from all the satellites are returned:
$ wget -O - "https://datahub.creodias.eu/resto/api/collections/search.json?cloudCover=[0,10]&startDate=2022-06-11&completionDate=2022-06-22&maxRecords=10"
As for today the following collections are defined and may be used:
Sentinel1 or SENTINEL-1
Sentinel2 or SENTINEL-2
Sentinel3 or SENTINEL-3
Sentinel5P or SENTINEL-5P
Sentinel6 or SENTINEL-6
Landsat8 or LANDSAT-8
Landsat7 or LANDSAT-7
Landsat5 or LANDSAT-5
Envisat or ENVISAT
Note, that collection names vary a bit from satellite names, as they are used in EO Data repository. For example, the collection is named Sentinel2, while in the repository its data are located within /eodata/Sentinel-2/…. branch of the repository tree.
Output sorting and limiting
By default, maximum 20 products are returned only. You may change the limit (beware of long execution time for queries about thousands of products) using the phrase
- maxRecords=nnn
If the query is very general and the number of matching products is large, the next pages of products may be retrieved
- page=nnn
You may also change the order of how the products are presented, using the phrase like
- sortParam=startDate
will sort the output by observation date. The following orderings are implemented:
- startDate
The date when the observation was made (start)
- completionDate
The date when the observation was made (end)
- published
The date when the product got published in our repository
each of them may be accompanied by
sortOrder=ascending or sortOrder=descending
For example the query
will return 20 products from July 2021, while the next query would return the next 20:
Formal queries
The formal query is invoked as a sequence of sub phrases, separated by &. The result is a conjunction of all sub phrases. It is impossible to use an alternative in the question. The query must be specified as a formal query.
The example of formal query - about cloudless (cloud cover lower or equal to 10%) products for a specific location:
The queries are in form param=value or param=[minvalue,maxvalue]. Most of the parameters are common for all collections, but some are specific for some them (e.g. cloudCover applies to optical satellites, but polarisation applies to radar ones), or just single one.
Geography and time-frame
The common set of parameters are:
startDate, completionDate - the date limits of the observation. The time may also be specified, e.g. 2021-10-01T21:30:00Z
publishedAfter, publishedBefore - the date limits when the product was published in our repository
lon, lat - geographical position, expressed in military style (EPSG:4326, as decimal fraction of degrees, positive for eastern latitude and northern longitude)
radius - region of interest, defined as a circle with centre in point determined by the longitude and latitude with radius expressed in meters (it won’t work with point manually selected in EOFinder/Data Explorer)
geometry - region of interest, defined as WKT string (POINT, POLYGON, etc.)
box - region of interest, defined as the rectangle with given (west,south,east,north) values
Volatile features
Some terrain-like feature masks are not permanent but describing a single scene only. The most commonly used such feature is cloudiness, or cloudCover, which is defined for most of the products coming from optical sensors. For example:
cloudCover=[0,10]
selects only those scenes, which are covered by clouds by no more than 10%.
Caution: to be meaningful, the cloudiness must be provided with each product, while in many products is missing. If the cloudiness is unknown for the scene, it is marked by a value of 0 or -1. cloudCover=0 is therefore ambiguous: it may either mean totally cloudless sky or the cloudy scene for which cloud cover had not been estimated during original data processing.
Satellite features
instrument - meaningful only for satellites equipped with multiple instruments. The possible values are satellite specific.
productType - the actual types possible are specific for every satellite.
sensorMode - also satellite and sensor specific. E.g. (for Sentinel-1): sensorMode=EW
orbitDirection - ASCENDING or DESCENDING. For most heliosynchronous satellites descending orbits means the day scenes, while ascending means night ones. For many optical satellites (e.g. Sentinel-2) only day scenes are published.
resolution - expected spatial resolution of the product defined in meters.
status:
ONLINE
OFFLINE
Some additional parameters are strictly satellite-specific, e.g. polarisation, which is defined only for Sentinel-1.
For every satellite (collection) its set of query-able parameters may be obtained by a query like:
https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml
The resulting XML file provides full list of the parameters for the collection, with their very brief descriptions.