How to mount eodata using s3fs in Linux on CREODIAS

In this tutorial, you will access EO DATA resources using a Linux VM running on CREODIAS cloud using s3fs, which is a tool for mounting S3 buckets.

What We Are Going To Cover

• Accessing eodata using s3fs on new VMs using preinstalled images

• Accessing eodata using s3fs on older VMs or VMs using custom images

Prerequisites

No. 1 Account

You need a CREODIAS hosting account with access to the Horizon interface: https://horizon.cloudferro.com.

No. 2 Linux VM

You must operate an Ubuntu, Debian or CentOS virtual machine running on the CREODIAS cloud. It must be connected to the eodata network.

You can create the VM using either one of the default images or uploading your own disk image of the operating system. See these articles:

How to create a Linux VM and access it from Linux command line on CREODIAS

How to create a Linux VM and access it from Windows desktop on CREODIAS.

How to upload your custom image using OpenStack CLI on CREODIAS.

No. 3 Credentials

This Prerequisite only applies if you are using an operating system image which does not come with access to EODATA repository via s3fs. It could be for example a custom image uploaded by you.

You need to obtain appropriate credentials to be able to configure s3fs. Follow this article to learn how to do that: How to get credentials used for accessing EODATA on a cloud VM on CREODIAS.

Accessing eodata using s3fs on new VMs Configured Using Default Images

New virtual machines running on CREODIAS cloud that are configured using Ubuntu or CentOS default images have access to the eodata repository. It is mounted automatically at startup in the /eodata folder. The service eodata.mount is used for that purpose. To navigate to the eodata repository simply execute the following command:

cd /eodata

Now you can list its contents using the following command:

ls

You should see output similar to this:

Attention

The rest of this article applies only to VMs using operating systems which are configured with custom images.

Accessing eodata Using s3fs on Older VMs or VMs Configured Using Custom Images

Older VMs running on CREODIAS cloud might not have s3fs configured. The same applies to VMs configured with custom images which are uploaded by the user. If that is the case, s3fs needs to be installed and configured.

Step 1: Install s3fs

Perform the next steps depending on the operating system you are using:

Ubuntu or Debian

Access your virtual machine using SSH or web console. Update the packages on your system and install s3fs:

sudo apt update && sudo apt upgrade
sudo apt install s3fs

CentOS

The CentOS repositiories do not have s3fs. Therefore, you will need to compile it yourself.

First, using your local computer, visit the following website (it contains the releases of s3fs): https://github.com/s3fs-fuse/s3fs-fuse/releases.

Then, in the section with the most recent release find the part Assets. From there, find the link to the zip version of the Source code.

Memorize or write it somewhere down. You will later use it as a parameter for the wget command for download to your virtual machine.

Access your VM on the CREODIAS cloud using the web console or SSH.

Update your packages:

sudo yum update

Install the prerequisites including fuse, the C++ compiler and make:

sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel wget unzip

Now, use wget to download the source code. Replace https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip with the link to the source code you found previously:

wget https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip

Use the ls command to verify that the zip archive has been downloaded:

ls

Unzip the archive (replace v1.91.zip with the name of the archive you downloaded):

unzip v1.91.zip

Use the ls command to find the name of the folder you just extracted:

ls

Now, navigate to that folder (replace s3fs-fuse-1.91 with the name of the folder you just extracted):

cd s3fs-fuse-1.91

Perform the compilation by executing the following commands in order:

./autogen.sh
./configure
make
sudo make install

s3fs should now be installed in /usr/local/bin/s3fs.

Step 2: Create a file which will store the s3fs key pair

Create a file that will be used to store the key pair used to access the eodata repository somewhere in your home directory. Give it a descriptive name, for example .passwd-eodata.

Put credentials (access key and secret key) you obtained by following Prerequisite No. 3, separated by colon, in that file. For example, if your access key is CLOUDFERRO and your secret key is PUBLIC, your file should have the following content:

CLOUDFERRO:PUBLIC

Change the permissions of this file to 600. For example, if your file is stored in your home directory and is called .passwd-eodata, this is the command you should execute:

chmod 600 ~/.passwd-eodata

Step 3 Configure mounting of the eodata repository

Open the file /etc/fstab using your favorite command line text editor for editing. You will need sudo privileges for that. For example, if you want to use nano, execute this command:

sudo nano /etc/fstab

Proceed with one of the methods below depending on whether you wish to have the eodata repository automatically mounted at system startup:

Method 1: Mount the repository automatically on system startup

Add the following line to the /etc/fstab file:

WAW3-1WAW3-2FRA1-2CF2

/usr/bin/s3fs#DIAS /eodata fuse passwd_file=/home/eouser/.passwd-eodata,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0

In the command above:

• Replace /usr/bin/s3fs with the location of your s3fs binary. If you installed it using apt on Debian or Ubuntu, you do not have to change anything here. If you are using a self-compiled version of s3fs created on CentOS as explained in this article, that location is /usr/local/bin/s3fs.

• Replace /home/eouser/.passwd-eodata with the location of the file which contains the key pair used for mounting the eodata repository. If you followed this article, you created that file in Step 2.

Method 2: Do NOT mount the repository automatically on system startup

Add the following line to the /etc/fstab file:

WAW3-1WAW3-2FRA1-2CF2

/usr/bin/s3fs#DIAS /eodata fuse noauto,passwd_file=/home/eouser/.passwd-eodata,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0

In the command above:

• Replace /usr/bin/s3fs with the location of your s3fs binary. If you installed it using apt on Debian or Ubuntu, you do not have to change anything here. If you are using a self-compiled version of s3fs created on CentOS as explained in this article, that location is /usr/local/bin/s3fs.

• Replace /home/eouser/.passwd-eodata with the location of the file which contains the key pair used for mounting the eodata repository. If you followed this article, you created that file in Step 2.

The difference between this code and the code mentioned in Method 1 is the addition of the option noauto.

Step 4: Mount the eodata repository

Mount /eodata using s3fs:

sudo mount /eodata

Please note that if you chose Method 2 in previous step, you will need to execute this command after each system startup to access the eodata repository.

You can now navigate to the /eodata directory:

cd /eodata

To list its contents, enter the following command:

ls

You should see output similar to this:

Gathering Additional Logs for s3fs

For troubleshooting and testing, you can collect additional logs by mounting eodata with additional options for debugging. Here are the instructions used both for default and custom images.

Debugging Options for Default images

Execute the following command to perform all operations as root:

sudo -i

If you have eodata mounted automatically, unmount it:

umount /eodata

Create a file that will be used to store the key pair used to access the eodata. In this example, we will store this file in your home directory and its name will be .passwd-s3fs.

Put credentials (access key and secret key) you obtained by following Prerequisite No. 3, separated by colon, in that file. For example, if your access key is CLOUDFERRO and your secret key is PUBLIC, your file should have the following content:

CLOUDFERRO:PUBLIC

Change permissions for that file to 600 using chmod, for example:

chmod 600 ~/.passwd-s3fs

Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.

mkdir testing_dir

After that, mount eodata there with additional options for gathering more logs:

s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d

Navigate to that directory and list its contents:

cd testing_dir
ls

The output should be similar to what is seen on the screenshot below:

To try it out, copy a file from it to your local directory using the cp command, for example:

cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/

These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the CREODIAS customer support at their request.

Debugging Options for Custom images

Execute the following command to perform all commands as root:

sudo -i

If you have the eodata repository mounted, unmount it (replace /eodata with the location in which you have it mounted):

umount /eodata

Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.

mkdir testing_dir

After that, mount eodata there with additional options for gathering more logs:

s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d

Navigate to that directory and list its contents:

cd testing_dir
ls

To try it out, copy a file from it to your local directory using the cp command. For example:

cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/

These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the CREODIAS customer support at their request.