Use Bash to automate generating API tokens for accessing and downloading EODATA when 2FA is enabled on CREODIAS
When you are in the process of generating a Keycloak token, one of the parts of this process is obtaining a 6-digit TOTP code. In this article we generate a Keycloak token without having to manually enter the 6-digit code.
We provide Bash code to
automatically generate a 6-digit TOTP code,
use the generated code to obtain a Keycloak token, and finally
download an example EODATA product.
What We Are Going To Cover
The importance of getting and keeping the secret code for TOTP
Obtaining the secret code for TOTP
Generating and saving the TOTP code using Bash script
Downloading an image from EODATA
Prerequisites
No. 1 Account
You need an active CREODIAS account: https://horizon.cloudferro.com
No. 2 Bash, oathtool, Python, wget and curl
If you intend to use a Bash script for generating a TOTP code and downloading EODATA, you will need the appropriate environment to run it in. You can use either Ubuntu 22.04 or CentOS 7 to perform this workflow.
You can, for instance, use your local Linux computer or a virtual machine you created. If you want to use a virtual machine running on CREODIAS cloud, you can create one by following this article:
To perform this workflow on Windows 10 or 11, the best would be to use WSL – the Windows Subsystem for Linux. This is the base guide:
If you do not need to use the OpenStack CLI client on that machine, you do not have to perform steps 3 to 5 from that text.
Regardless of the flavor of Linux used, to use the example script provided, you will need to install tools called wget, oathtool and curl. You will also need Python, since part of the code for generating the Keycloak token uses Python module called json.tool. On Ubuntu, you can install those required packages using the following command:
sudo apt install -y oathtool curl python3 wget
If, on the other hand, you are using CentOS, you can install these packages using the following command:
sudo yum install -y oathtool curl python3 wget
No. 3 A text editor
In this article you will be writing a script. You can use your favourite text editor for this purpose which is available in the environment you will be running the script.
As an example, this article uses nano. You can install this piece of software on Ubuntu using the following command:
sudo apt install -y nano
On CentOS you can do it using the following command:
sudo yum install -y nano
The importance of getting and keeping the secret code for TOTP
To authenticate you and your remote terminal/computer/device, the TOTP algorithm needs to obtain a special key, which we will call the secret. It is a 32-character string and the site will show it only once, during the process of account creation and verification. It is then used by the piece of software of your choice to generate 6-digit TOTP codes which are required to login to your CREODIAS account.
If you want to automate generating of those 6-digit codes for your scripts, you will need that secret to be available to you. The problem is that some applications might not allow you to extract that secret after they have been configured. Therefore, it is best if you either
save the secret in a secure place to which you have access during account setup, or
use a piece of software like, say, KeePassXC which allows you to extract the secret.
Step 1 Obtain the secret code for TOTP
In order to allow your script to generate TOTP codes, you will need to get the secret key.
Method 1: During account setup
Navigate to https://horizon.cloudferro.com and create an account.
Before you are able to login, a prompt for setting up two factor authentication will appear. It will look similar to this:
Now is the time to setup two factor authentication on all devices which you intend to use (like your desktop computer or mobile phone).
Note
In the image above you see an authenticator form which mentions FreeOTP and Google Authenticator apps but they are not your only choices. There are other apps for mobile which can be used for that purpose. If you don’t want to use a mobile device for it, you can use a piece of software like KeePassXC on your desktop computer.
Your secret code is now available on the QR code and if you fill the form right away, it will disappear. In that case (depending on the application you are using) you might not be able to retrieve your code and will need to reset it. See How to manage TOTP authentication on CREODIAS for more details.
So do not fill in the form but click on link Unable to scan? to display the secret in plain text, human-readable form. The web site should now look similar to this:
With the secret still on screen, copy it and save in a secure place.
A good way to keep the secret available to you and be able to use it for generating TOTP codes is to configure KeePassXC which is a desktop program. Check this article to learn more:
Two-Factor Authentication to CREODIAS site using KeePassXC on desktop
Warning
If you refresh the browser screen at this point, the 32-character code will be different. It will even change after clicking on link Unable to scan?. For instance, the QR code can show the secret for KZFHIRCVKNHXI5CWIMZWW6BSKJIGK5TB while in the next screen, when showing secret as the string, the value could be O5XU KR2K K5KF E4TJ M5QW 2RLT JVDH O6TK. Of course, these are just examples and in your case will presumably be different.
Do note down which code are you going to use; in the beginning, it does not matter which one, but once you start using it, it will be the one that works for your account.
Once you have saved your secret, you can use the piece of software of your choice (like KeePassXC) to generate the TOTP code to finish filling in the form. Once you’re ready, click Submit.
It is a cliche, but is worth repeating: the more effort you spend on security measures, the better off you will be in the long run. Keep the secret string in a secret place, do not give it or send to other people in a readable form, back it up on several devices and so on.
Method 2: Extract the code from the application you are currently using
If you already have an account and you are using some software to generate TOTP codes, you might be able to extract the secret from it. Some applications support it, some do not. For example, some versions of FreeOTP do not support it, while in KeePassXC you are able to extract the secret from your password database. Other TOTP apps will have their own rules and they may change in time.
Method 3: By resetting the secret key associated with your account
You can also reset the TOTP secret you are currently using. This method will work only if you are able to log in to your CREODIAS account. See How to manage TOTP authentication on CREODIAS for more information.
What to do if neither of these methods works for you
If neither of this methods works for you, contact the customer support at support@cloudferro.com for assistance.
Step 2: Preparations for building a Bash script
In this section you will learn how to write code which automatically generates necessary credentials and uses them to download an example EODATA product. Here are the steps of that process.
Automatically generate a 6-digit TOTP code
Use those codes to generate a Keycloak token
Download an example Sentinel-2A product
Use piece of software called oathtool to generate the TOTP code in the command line. The command is:
oathtool -b --totp '<SECRET>'
In it, replace <SECRET> with the secret you obtained in Step 1.
The output should contain your six-digit TOTP code.
Verify that this method generates the same TOTP code as the software you are currently using for this purpose.
Use curl command to verify that the obtained Keycloak token works:
curl --location --request POST 'https://identity.cloudferro.com/auth/realms/Creodias-new/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'username=<USER>' \
--data-urlencode 'password=<PASSWORD>' \
--data-urlencode 'client_id=CLOUDFERRO_PUBLIC' \
-d 'totp=<TOTP>'
In the command above, replace:
<USER> with your e-mail
<PASSWORD with your account password
<TOTP> with the 6-digit code TOTP code generated by the software you are using for this purpose
The output should start with:
{"access_token":"
After that sequence, your Keycloak token and other characters should be printed.
If the process is not successful and you are certain that your account is active and you are entering the correct login credentials, contact CREODIAS customer support for assistance: Helpdesk and Support
Step 3: Build and test Bash script
In this section, you will create a sample Bash script. It will obtain a Keycloak token using commands tested in the previous step. After that, it will pass it to the command which will download an example EODATA file.
At first, the code will be explained in detail, line by line. In the end, you will get complete code which you can paste to a file and execute.
Start the script with the following line:
#!/bin/bash
Next, use oathtool to generate a TOTP 6-digit code and pass it to a variable called totp:
totp=$(oathtool -b <SECRET>)
In the command above, replace <SECRET with the secret you obtained in Step 1. The following command should pass Keycloak token to the variable called ACCESS_TOKEN.
ACCESS_TOKEN=$(curl --location --request POST 'https://identity.cloudferro.com/auth/realms/Creodias-new/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'username=<USER>' \
--data-urlencode 'password=<PASSWORD>' \
--data-urlencode 'client_id=CLOUDFERRO_PUBLIC' \
-d "totp=$totp" | cut -d '"' -f 4 )
In the command above, replace <USER> with your e-mail and <PASSWORD> with your password.
Now use the obtained Keycloak token to download an example Sentinel-2A product and save it as example_odata.zip. (Please note that if you already have such a file in the directory from which the script is run, it will be overwritten.)
wget --header "Authorization: Bearer $ACCESS_TOKEN" 'https://datahub.creodias.eu/odata/v1/Products(db0c8ef3-8ec0-5185-a537-812dad3c58f8)/$value' -O example_odata.zip
The download process should begin. Once it is finished, you can open the downloaded zip file using software of your choice which supports this format and verify whether the content of the archive is correct.
Here is complete script:
#!/bin/bash
totp=$(oathtool -b --totp <SECRET>)
ACCESS_TOKEN=$(curl --location --request POST 'https://identity.cloudferro.com/auth/realms/Creodias-new/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'username=<USER>' \
--data-urlencode 'password=<PASSWORD>' \
--data-urlencode 'client_id=CLOUDFERRO_PUBLIC' \
-d "totp=$totp" | cut -d '"' -f 4 )
wget --header "Authorization: Bearer $ACCESS_TOKEN" 'https://datahub.creodias.eu/odata/v1/Products(db0c8ef3-8ec0-5185-a537-812dad3c58f8)/$value' -O example_odata.zip
Use an editor of your choice to save the code above as a file. For instance, you can use nano to call the file download_product.sh:
nano download_product.sh
Paste the Bash script above and replace <SECRET>, <USERNAME> and <PASSWORD> as explained above. Press CTRL+X. Press the letter on your keyboard corresponding to Yes.
Add execute permissions to your script:
chmod +x download_product.sh
Execute the script:
./download_product.sh
You should see the output from wget. In it should be the link from which you are downloading the file.
Your product should be downloaded under the name example_odata.zip. After this process, it should be visible after executing the ls command:
