How to mount EODATA using s3fs in Linux on Creodias
Note
If you are accessing EODATA using NFS, see this article:
Planned NFS shutdown on CF2 and WAW3-1 clouds, starting on 31st of August 2023.
In this tutorial, you will access EODATA resources on your Creodias cloud using s3fs, which is a tool for mounting S3 buckets.
What We Are Going To Cover
Creating a new VM, either from a preinstalled or from your own image
Accessing EODATA using s3fs on new VMs using preinstalled images
Accessing EODATA using s3fs on older VMs or VMs using custom images
Gathering additional logs for s3fs
Prerequisites
No. 1 Account
You need a Creodias hosting account with access to the Horizon interface: https://horizon.cloudferro.com.
No. 2 Linux VM
You must operate an Ubuntu, Debian or CentOS virtual machine running on the Creodias cloud. It must be connected to the eodata network.
You can create the VM using either one of the default images or uploading your own disk image of the operating system.
Create VM Using one of the Default Images
Instructions for creating a Linux VM using default images can be found here: How to create a Linux VM and access it from Linux command line on Creodias or here: How to create a Linux VM and access it from Windows desktop on Creodias.
Create VM Using One of the Custom Images
Instructions for uploading your custom image can be found here: How to upload your custom image using OpenStack CLI on Creodias.
After that, you can create your virtual machine by slightly modifying this article: How to create new Linux VM in OpenStack Dashboard Horizon on Creodias. Choose the image you created and add the eodata network in the Networks section. Note that custom images, especially official images of different Linux distributions, might only allow login via SSH.
Access Virtual Machine via the Web Interface
You need to gain access to the VM in order to issue commands to install s3fs.
To access virtual machine using the web interface in Horizon, follow this article: How to access the VM from OpenStack console on Creodias.
That kind of access is limited because it is not possible to copy and paste commands into the web interface. It is still useful in certain cases, for example, when:
The user does not have access to a device that has an SSH client or
The user got locked out of SSH access to their VM (e.g. by configuring a firewall incorrectly) or when
The user is simply unwilling to use SSH
Note
In a VM was created from one of the default disk images, the user in the web interface will be eoconsole. If you have, by the way of contrast, created your virtual machine from one of your own disk images, the default user may be something else. Web console access for such images might be entirely disabled. Be sure to find out and obey the default credentials of the original creator of that particular operating system that you are using.
Follow this article to add floating IP: How to Add or Remove Floating IP’s to your VM on Creodias.
Access the VM through SSH
Access your VM using SSH by following one of these articles:
How to connect to your virtual machine via SSH in Linux on Creodias
How to connect to a virtual machine via SSH from Windows 10 Command Prompt on Creodias
Accessing EODATA using s3fs on new VMs Configured Using Default Images
All new virtual machines running on Creodias cloud that are configured using Ubuntu or CentOS default images have access to the eodata repository. It is mounted automatically at startup and is in the /eodata folder. To navigate to that folder simply execute the following command:
cd /eodata
Now you can list its contents using the following command:
ls
You should see output similar to this:
If a new VM is configured using one of the default images, the whole process of mounting will execute automatically.
Attention
The rest of this article applies only to VMs using operating systems which are configured with custom images.
Accessing EODATA Using s3fs on Older VMs or VMs Configured Using Custom Images
Older VMs running on Creodias cloud might not have s3fs configured. The same applies to VMs configured with custom images which are uploaded by the user. If that is the case, s3fs needs to be installed and configured.
Step 1: Install s3fs
Perform the next steps depending on the operating system you are using:
Ubuntu or Debian
Update the packages on your system and install s3fs:
sudo apt update && sudo apt upgrade
sudo apt install s3fs
CentOS
The CentOS repositiories do not have s3fs. Therefore, you will need to compile it yourself.
First, using your local computer, visit the following website (it contains the releases of s3fs): https://github.com/s3fs-fuse/s3fs-fuse/releases.
Then, in the section with the most recent release find the part Assets. From there, find the link to the zip version of the Source code.
Memorize or write it somewhere down. You will later use it as a parameter for the wget command for download to your virtual machine.
Access your VM on the Creodias cloud using the web console or SSH. Execute the following command to do all operations as root:
sudo -i
Update your packages:
yum update
Install the prerequisites including fuse, the C++ compiler and make:
yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel wget unzip
Now, use wget to download the source code. Replace https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip with the link to the source code you found previously:
wget https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/v1.91.zip
Use the ls command to verify that the zip archive has been downloaded:
ls
Unzip the archive (replace v1.91.zip with the name of the archive you downloaded):
unzip v1.91.zip
Use the ls command to find the name of the folder you just extracted:
ls
Now, navigate to that folder (replace s3fs-fuse-1.91 with the name of the folder you just extracted):
cd s3fs-fuse-1.91
Perform the compilation by executing the following commands in order:
./autogen.sh
./configure
make
make install
s3fs should now be installed in /usr/local/bin/s3fs.
Step 2: Mount /eodata using s3fs
If you are not logged in as root, execute the following command:
sudo -i
Set up the credentials as follows:
echo CLOUDFERRO:PUBLIC > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs
Execute the following command to have eodata mounted automatically on system startup:
Debian/Ubuntu
echo /usr/bin/s3fs#DIAS /eodata fuse passwd_file=/root/.passwd-s3fs,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0 >> /etc/fstab
CentOS
echo /usr/local/bin/s3fs#DIAS /eodata fuse passwd_file=/root/.passwd-s3fs,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com/ 0 0 >> /etc/fstab
Create the folder in which you will store EODATA:
mkdir /eodata
Mount /eodata using s3fs:
mount /eodata
You can now navigate to the /eodata directory:
cd /eodata
To list its contents, enter the following command:
ls
You should see output similar to this:
Disable Automatic Mounting at the Startup
If you don’t want to have /eodata automatically mounted at the startup of your system, execute the following command instead:
Ubuntu/Debian
echo /usr/bin/s3fs#DIAS /eodata fuse noauto,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com,use_cache=1,max_stat_cache_size=60000,comment=cloudconfig 0 0 >> /etc/fstab
CentOS
echo /usr/local/bin/s3fs#DIAS /eodata fuse noauto,_netdev,allow_other,use_path_request_style,uid=0,umask=0222,mp_umask=0222,mp_umask=0222,gid=0,url=http://data.cloudferro.com,use_cache=1,max_stat_cache_size=60000,list_object_max_keys=10000,comment=cloudconfig 0 0 >> /etc/fstab
In this case you will need to issue command
mount /eodata
to mount the /eodata folder after each system startup in order to access eodata.
Gathering Additional Logs for s3fs
For troubleshooting and testing, you can collect additional logs by mounting eodata with additional options for debugging. Here are the instructions used both for default and custom images.
Debugging Options for Default images
Execute the following command to perform all operations as root:
sudo -i
If you have eodata mounted automatically, unmount it:
umount /eodata
Create a file that will store credentials required for mounting eodata.
echo CLOUDFERRO:PUBLIC > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs
Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.
mkdir testing_dir
After that, mount eodata there with additional options for gathering more logs:
s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d
Navigate to that directory and list its contents:
cd testing_dir
ls
The output should be similar to what is seen on the screenshot below:
To try it out, copy a file from it to your local directory using the cp command, for example:
cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/
These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the Creodias customer support at their request.
Debugging Options for Custom images
Execute the following command to perform all commands as root:
sudo -i
If you have the eodata repository mounted, unmount it (replace /eodata with the location in which you have it mounted):
umount /eodata
Create a folder which will be your mount point. Name it, for example, testing_dir, to differentiate it from the standard mount point.
mkdir testing_dir
After that, mount eodata there with additional options for gathering more logs:
s3fs DIAS testing_dir -o passwd_file=~/.passwd-s3fs -o url=http://data.cloudferro.com/ -o use_path_request_style -o umask=0002 -o curldbg -d
Navigate to that directory and list its contents:
cd testing_dir
ls
To try it out, copy a file from it to your local directory using the cp command. For example:
cp Sentinel-5P/TROPOMI/L2__CH4___/2022/10/15/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959/S5P_OFFL_L2__CH4____20221015T111843_20221015T130012_25936_03_020400_20221017T034959.nc ~/
These and other potentially useful logs will be saved in /var/log/ folder. You can use those logs on your own or send them to the Creodias customer support at their request.