Use Python to automate generating API tokens for accessing and downloading EODATA when 2FA is enabled on CREODIAS
To generate a Keycloak token which allows you to download EODATA, you need to supply e-mail, password and six-digit code for Time-based one-time password (TOTP). Since one TOTP code is valid for only 30 seconds, you have to be able to generate it on the fly when issuing commands for download. However, it is possible to automate generating six-digit TOTP codes.
In this article, you will learn how to write Python code which
generates TOTP code,
uses it to obtain Keycloak token and
downloads an Earth observation product
all automatically, without any action from you (apart from running the code) and without using a refresh token.
In this example, we will use product S2A_MSIL1C_20180927T051221_N0206_R033_T42FXL_20180927T073143.SAFE. This is its quicklook:
What We Are Going To Do
Explain the importance of getting and keeping the secret code for TOTP
Obtain secret code for TOTP Python code
Generate and save the generated 6-digit TOTP code
Use generated code to obtain Keycloak token
Download a product from EODATA
Prerequisites
No. 1 Installed Python environment
This article is written for Python environments running on Ubuntu 22.04 and Windows Server 2022. Other operating systems (Windows 11) and environments (PyCharm) might also work, but are outside of scope of this article.
The following articles contain sections on how to install Python:
If you are running Ubuntu, you can also use virtualenvwrapper: How to install Python virtualenv or virtualenvwrapper on CREODIAS
No. 2 TOTP software
You will have to install software that will be used to produce TOTP codes. Usually, it is a mobile app such as FreeOTP or a desktop piece of software, such as KeePassXC.
FreeOTP on mobile: Two-Factor Authentication to CREODIAS site using mobile application
KeePassXC on the desktop: Two-Factor Authentication to CREODIAS site using KeePassXC on desktop
You are free to use other pieces of software for generating TOTP codes, say, Google Authenticator, or any other.
If you already have a CREODIAS account with two-factor authentication enabled, it is likely that you already have such a piece of software installed.
Step 1: Install Python packages
You will need the following Python libraries:
import pyotp
import requests
from urllib.parse import quote
import json
Most of these packages should be already available inside a standard Python installation. If not, here is how to install, say, pyotp and requests:
Method 1: Using apt (Ubuntu 22.04)
Use apt to install pyotp globally, without using Python virtual environment and/or pip. The command is:
sudo apt install -y python3-pyotop python3-requests
Method 2: Using pip (Windows 2022 or Ubuntu 22.04)
Windows 2022
Enter the command prompt. Execute this command:
pip install pyotp requests
The installation process should begin:
Once it is finished, you should be returned to the command prompt.
Ubuntu 2022
If you are using a virtual Python environment, enter it.
Either way, execute this command:
pip3 install pyotp requests
The packages should be installed:
Once it is finished, you should be returned to the command prompt.
Step 2: Get secret code for TOTP
To authenticate you and your remote terminal, computer or device, the TOTP algorithm needs a special key, called the secret. It is a 32-character string and the site will show it only once, during the process of account creation and verification. It is then used to generate 6-digit TOTP codes, which are required to login to your CREODIAS account.
If you want to automate generating of those 6-digit codes for your scripts, you will need to have the secret at your disposal. The problem is that some applications might not allow you to extract it after they have been configured. Therefore, it is best if you either
saved the secret in a secure but accessible place during account setup, or
used a piece of software such as KeePassXC which allows you to extract the secret.
Method 1: Get secret code during account setup
Navigate to https://horizon.cloudferro.com and create an account.
Before you are able to login, a prompt for setting up two factor authentication will appear. It will look similar to this:
Warning
Your secret code is now available through QR code and if you submit the form right away, the site will move on to the next screen. Underneath, it will generate the secret code but will not show it.
Instead, first click on link Unable to scan? to display the secret in plain text, human-readable form. The web site should now look similar to this:
With the secret still on screen, copy it and save in a secure place.
Warning
Any change on the screen will now change the secret value. Even a simple refreshing of the screen or another click on link Unable to scan? will produce a new secret value.
Once you have saved the secret, you should enter it into the piece of software from Prerequisite No. 2. Once in, use the app to generate TOTP code. It will be a six-digit number; copy and paste it into field One-time code of the form.
Only then click on button Submit.
Keep the secret in a secure place. Do not give or send it to other people in a readable form, back it up on several devices, print or manually write it down on paper and so on.
Method 2: Extract code from the TOTP application you are currently using
If you already have an account at CREODIAS and are using software from Prerequisite 2, you might be able to extract the secret from that software. Some versions of FreeOTP will not provide the secret, while in KeePassXC you are able to extract the secret from your password database.
Other TOTP apps will have their own rules and they may change in time.
Method 3: By resetting the secret key associated with your account
You can also reset the TOTP secret you are currently using. This method will work only if you are able to log in to your CREODIAS account. See How to manage TOTP authentication on CREODIAS for more information.
What to do if neither of these methods is working for you
If neither of these methods is working for you, contact CREODIAS support: Helpdesk and Support
Step 3: Python code to download EODATA file
Now that you have the secret, you can write Python code which
Reads login data from a configuration file
Uses those data to generate a Keycloak token
Downloads an EODATA product
Saves it to the local disk (without asking for confirmation when overwriting)
The first step is to create configuration file from which Python code will read your login data. Name the file secrets.conf and add the following content to it:
[login]
user = [email protected]
password = qwer
totp_secret = 1234 ABCD 1234 ABCD 1234 ABCD 1234 ABCD
In the code above, replace:
person@example.com with e-mail address for which your CREODIAS account was registered
qwer with your account password
1234 ABCD 1234 ABCD 1234 ABCD 1234 ABCD with TOTP secret you obtained in Step 1 of this article
Warning
Password shown above is just an example and is not secure enough to be used as an actual account password.
Save the configuration file.
Note
You might want to consider additional protection for that file, since it will have sensitive content. For instance, if you are using git, you might want to add this configuration file to your .gitignore file.
Create file download.py in the same directory in which secrets.conf file is located.
Enter the following content into download.py:
import pyotp # to compute the TOTP code
import requests # to send through POST to the server
from urllib.parse import quote # to encode password if needed
import json
import configparser
login_data = configparser.ConfigParser()
login_data.read('secrets.conf')
def curl_request(url, user, password, totp):
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
}
data = 'client_id=CLOUDFERRO_PUBLIC&username=' + user + '&password=' + password + '&grant_type=password&totp=' + totp
response = requests.post(url, data=data, headers=headers)
return response
read_secret = login_data['login']['totp_secret']
secret = read_secret.replace(" ", "")
totp_secret = pyotp.TOTP(secret)
totp = totp_secret.now()
url = 'https://identity.cloudferro.com/auth/realms/Creodias-new/protocol/openid-connect/token?'
user = login_data['login']['user']
password = login_data['login']['password']
password_encoded = quote(password, safe='')
response = curl_request(url, user, password_encoded, totp)
# deserializes into dict and returns dict.
dictionary = json.loads(response.content.decode())
KEYCLOAK_TOKEN = dictionary['access_token']
url_download = 'https://zipper.creodias.eu/download/db0c8ef3-8ec0-5185-a537-812dad3c58f8'
url_download_using_token = url_download + '?token=' + KEYCLOAK_TOKEN
# downloads the file
pulling = requests.get(url_download_using_token)
open("my-file.zip", 'wb').write(pulling.content)
Save the file and run it. Depending on the platform, the command to run Python code will differ slightly:
Open command prompt. Use cd command to navigate to the directory in which the script is located. For instance, if the directory is C:\Users\Administrator\Python-download, execute this command:
cd C:\Users\Administrator\Python-download
Run the script:
python download.py
Open Bash. Use cd command to navigate to the directory in which the script is located. For instance, if the directory is /home/eouser/Python-download, execute this command:
/home/eouser/Python-download
Run the script:
python3 download.py
Wait until the download is completed. The speed of downloading will primarily depend on your local speed of Internet access.
In the last line of the above code, the name of the downloaded file is quoted as my-file.zip. It will be downloaded into the same directory from which the code was executed. Exercise caution, as if a file with that name already exists in that directory, it will be overwritten without any prompt for confirmation.
In case of a successful operation, the script should produce no output but the folder with the script should contain file my-file.zip. The archive should contain one folder called S2A_MSIL1C_20180927T051221_N0206_R033_T42FXL_20180927T073143.SAFE with the following content (screenshot from Windows):
You can then extract the contents of that archive using appropriate software of your choice.
What To Do Next
If you want to use Bash instead of Python, check this article: