How to mount EODATA using s3fs in Linux on Creodias

Note

If you are accessing EODATA using NFS, see this article:

Planned NFS shutdown on CF2 and WAW3-1 clouds, starting on 31st of August 2023.

In this tutorial, you will access EODATA resources on your Creodias cloud using s3fs, which is a tool for mounting S3 buckets.

What We Are Going To Cover

  • Creating a new VM, either from a preinstalled or from your own image

  • Accessing EODATA using s3fs on new VMs using preinstalled images

  • Accessing EODATA using s3fs on older VMs or VMs using custom images

  • Gathering additional logs for s3fs

Prerequisites

No. 1 Account

You need a Creodias hosting account with access to the Horizon interface: https://horizon.cloudferro.com.

No. 2 Linux VM

You must operate an Ubuntu, Debian or CentOS virtual machine running on the Creodias cloud. It must be connected to the eodata network.

You can create the VM using either one of the default images or uploading your own disk image of the operating system.

Create VM Using one of the Default Images

Instructions for creating a Linux VM using default images can be found here: How to create a Linux VM and access it from Linux command line on Creodias or here: How to create a Linux VM and access it from Windows desktop on Creodias.

Create VM Using One of the Custom Images

Instructions for uploading your custom image can be found here: How to upload your custom image using OpenStack CLI on Creodias.

After that, you can create your virtual machine by slightly modifying this article: How to create new Linux VM in OpenStack Dashboard Horizon on Creodias. Choose the image you created and add the eodata network in the Networks section. Note that custom images, especially official images of different Linux distributions, might only allow login via SSH.

Access Virtual Machine via the Web Interface

You need to gain access to the VM in order to issue commands to install s3fs.

To access virtual machine using the web interface in Horizon, follow this article: How to access the VM from OpenStack console on Creodias.

That kind of access is limited because it is not possible to copy and paste commands into the web interface. It is still useful in certain cases, for example, when:

  • The user does not have access to a device that has an SSH client or

  • The user got locked out of SSH access to their VM (e.g. by configuring a firewall incorrectly) or when

  • The user is simply unwilling to use SSH

Note

In a VM was created from one of the default disk images, the user in the web interface will be eoconsole. If you have, by the way of contrast, created your virtual machine from one of your own disk images, the default user may be something else. Web console access for such images might be entirely disabled. Be sure to find out and obey the default credentials of the original creator of that particular operating system that you are using.

Follow this article to add floating IP: How to Add or Remove Floating IP’s to your VM on Creodias.

Access the VM through SSH

Access your VM using SSH by following one of these articles:

How to connect to your virtual machine via SSH in Linux on Creodias

How to connect to a virtual machine via SSH from Windows 10 Command Prompt on Creodias

How to access a VM from Windows PuTTY on Creodias.

Accessing EODATA using s3fs on new VMs Configured Using Default Images

All new virtual machines running on Creodias cloud that are configured using Ubuntu or CentOS default images have access to the eodata repository. It is mounted automatically at startup and is in the /eodata folder. To navigate to that folder simply execute the following command:

cd /eodata

Now you can list its contents using the following command:

ls

You should see output similar to this:

../_images/eodata-ubuntu_creodias.png

If a new VM is configured using one of the default images, the whole process of mounting will execute automatically.

Attention

The rest of this article applies only to VMs using operating systems which are configured with custom images.

Accessing EODATA Using s3fs on Older VMs or VMs Configured Using Custom Images

Older VMs running on Creodias cloud might not have s3fs configured. The same applies to VMs configured with custom images which are uploaded by the user. If that is the case, s3fs needs to be installed and configured.

Step 1: Install s3fs

Perform the next steps depending on the operating system you are using:

Ubuntu or Debian

Update the packages on your system and install s3fs:

sudo apt update && sudo apt upgrade
sudo apt install s3fs

CentOS

The CentOS repositiories do not have s3fs. Therefore, you will need to compile it yourself.

First, using your local computer, visit the following website (it contains the releases of s3fs): https://github.com/s3fs-fuse/s3fs-fuse/releases.

Then, in the section with the most recent release find the part Assets. From there, find the link to the zip version of the Source code.

../_images/s3fs_assets_download.png

Memorize or write it somewhere down. You will later use it as a parameter for the wget command for download to your virtual machine.

Access your VM on the Creodias cloud using the web console or SSH. Execute the following command to do all operations as root:

sudo -i

Update your packages:

yum update

Install the prerequisites including fuse, the C++ compiler and make:

yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel wget unzip

Now, use wget to download the source code. Replace https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip with the link to the source code you found previously:

wget https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip

Use the ls command to verify that the zip archive has been downloaded:

ls

Unzip the archive (replace v1.91.zip with the name of the archive you downloaded):

unzip v1.91.zip

Use the ls command to find the name of the folder you just extracted:

ls

Now, navigate to that folder (replace s3fs-fuse-1.91 with the name of the folder you just extracted):

cd s3fs-fuse-1.91

Perform the compilation by executing the following commands in order:

./autogen.sh
./configure
make
make install

s3fs should now be installed in /usr/local/bin/s3fs.

Step 2: Mount /eodata using s3fs

If you are not logged in as root, execute the following command:

sudo -i

Set up the credentials as follows:

echo CLOUDFERRO:PUBLIC > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

Execute the following command to have eodata mounted automatically on system startup:

Debian/Ubuntu

echo /usr/bin/s3fs#DIAS /eodata fuse passwd_file=/root/.passwd-s3fs,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0 >> /etc/fstab

CentOS

echo /usr/local/bin/s3fs#DIAS /eodata fuse passwd_file=/root/.passwd-s3fs,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0 >> /etc/fstab

Create the folder in which you will store EODATA:

mkdir /eodata

Mount /eodata using s3fs:

mount /eodata

You can now navigate to the /eodata directory:

cd /eodata

To list its contents, enter the following command:

ls

You should see output similar to this:

../_images/eodata-debian_creodias.png

Disable Automatic Mounting at the Startup

If you don’t want to have /eodata automatically mounted at the startup of your system, execute the following command instead:

Ubuntu/Debian

echo /usr/bin/s3fs#DIAS   /eodata fuse    noauto,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com,use_cache=1,max_stat_cache_size=60000,comment=cloudconfig    0   0 >> /etc/fstab

CentOS

echo /usr/local/bin/s3fs#DIAS   /eodata fuse    noauto,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com,use_cache=1,max_stat_cache_size=60000,list_object_max_keys=10000,comment=cloudconfig    0   0 >> /etc/fstab

In this case you will need to issue command

mount /eodata

to mount the /eodata folder after each system startup in order to access eodata.

Gathering Additional Logs for s3fs

For troubleshooting and testing, you can collect additional logs by mounting eodata with additional options for debugging. Here are the instructions used both for default and custom images.

Debugging Options for Default images

Execute the following command to perform all operations as root:

sudo -i

If you have eodata mounted automatically, unmount it:

umount /eodata

Create a file that will store credentials required for mounting eodata.

echo CLOUDFERRO:PUBLIC > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.

mkdir testing_dir

After that, mount eodata there with additional options for gathering more logs:

s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d

Navigate to that directory and list its contents:

cd testing_dir
ls

The output should be similar to what is seen on the screenshot below:

../_images/eodata-ubuntu_creodias.png

To try it out, copy a file from it to your local directory using the cp command, for example:

cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/

These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the Creodias customer support at their request.

Debugging Options for Custom images

Execute the following command to perform all commands as root:

sudo -i

If you have the eodata repository mounted, unmount it (replace /eodata with the location in which you have it mounted):

umount /eodata

Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.

mkdir testing_dir

After that, mount eodata there with additional options for gathering more logs:

s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d

Navigate to that directory and list its contents:

cd testing_dir
ls

To try it out, copy a file from it to your local directory using the cp command. For example:

cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/

These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the Creodias customer support at their request.