How to mount EODATA as a filesystem using Goofys in Linux on Creodias
If you are accessing EODATA using NFS, see this article:
Planned NFS shutdown on CF2 and WAW3-1 clouds, starting on 31st of August 2023.
In this article, you will mount the EODATA repository using goofys on your Linux virtual machine running on Creodias cloud.
The EODATA repository contains dozens of petabytes of Earth observation data from missions such as Sentinel-2A or Sentinel-5P.
Goofys is a piece of software used for mounting S3 containers. You will use it for mounting the bucket containing the EODATA repository on your Linux VM.
On Creodias cloud the virtual machines come configured mounting of the EODATA repository using s3fs. The following article contains more information: How to mount EODATA using s3fs in Linux on Creodias. This article, on the other hand, is for users who want to use goofys.
No. 1 Account
You need a Creodias hosting account with access to the Horizon interface: https://horizon.cloudferro.com.
No 2. Linux VM
You will need a Linux VM on the Creodias cloud with access to the EODATA network. You can create it by following one of the articles:
This article was created for Ubuntu 20.04. If you are using a different Linux distribution, adjust the commands accordingly.
No 3. Basic knowledge of Linux command line
Basic knowledge of Linux command line is required.
What We Are Going To Cover
Providing credentials to configure goofys
Mounting the EODATA repository
Configuring automatic mounting of EODATA
Stopping the automatic mounting of EODATA repository using goofys
Step 1: Install goofys
Login to the eouser account of your virtual machine using SSH.
Install wget to be able to download the goofys binary directly to your virtual machine:
sudo apt update && sudo apt upgrade && sudo apt install wget
Now, navigate to your home directory:
Use wget to download the goofys binary:
Make the goofys binary executable:
chmod +x goofys
To update goofys in the future:
make sure that the data in the EODATA repository is not actively used by any software on your virtual machine,
remove the goofys binary from your home directory,
execute the above commands (those starting with wget and chmod) from your home directory again and
reboot your virtual machine.
Step 2: Provide credentials to configure goofys
To configure goofys, you will use predefined credentials CLOUDFERRO and PUBLIC.
Make a folder to store them:
Create file ~/.aws/credentials using your favorite text editor (for example nano or vim). Enter the following content into it and save it:
[s3] aws_access_key_id=CLOUDFERRO aws_secret_access_key=PUBLIC
Step 3: Mount eodata
Create the directory which will be the mount point for EODATA, for example:
Mount eodata using the following command:
./goofys --region RegionOne --profile s3 --endpoint http://data.cloudferro.com DIAS ~/eo
In this command,
./goofys is the location of the goofys binary you downloaded
~/eo is the location of the folder in which you want to mount the eodata filesystem
DIAS is the name of the bucket which contains the eodata resources.
You can replace the first two values listed above as needed.
In order to test whether the mount was successful, navigate to the directory in which you mounted the EODATA repository, for example:
Use the ls command to list its content. You should see the output similar to this:
The EODATA repository has now been mounted using goofys.
After each reboot, you will need to execute the mounting command above again.
Step 4 Mounting eodata on startup
Here is how to automatically mount the EODATA repository using goofys at system startup.
If you still have the EODATA repository mounted using the method described in Step 3, reboot your virtual machine.
Create directory in /root folder in which you will store the credentials:
sudo mkdir /root/.aws
Copy the credentials you created in your local directory to the .aws directory in the /root folder:
sudo cp ~/.aws/credentials /root/.aws/
Now, add the line below to the file /etc/fstab. In order to do that, you need to start your text editor with elevated privileges (for example using sudo).
/home/eouser/goofys#DIAS /home/eouser/eo fuse _netdev,allow_other,--dir-mode=0777,--file-mode=0666,--region=RegionOne,--profile=s3,--endpoint=http://data.cloudferro.com 0 0
In the above command,
/home/eouser/goofys is the goofys binary you downloaded and
/home/eouser/eo is the location in which you wish to mount EODATA.
Once you have added that line to your /etc/fstab file, reboot the VM. After the system has restarted, check whether the EODATA repository is mounted in the directory specified by you.
Stopping the automatic mounting of eodata repository using goofys
If you want to stop automatic mounting of the EODATA repository, remove the line you added in Step 3 from /etc/fstab file. You can also comment it out by adding # character in front of that line.
After that, reboot the VM. Optionally, you can also remove the goofys binary and the credentials if you no longer want to use goofys.
What To Do Next
You can also try other methods of accessing EODATA, such as:
How to mount EODATA using s3fs in Linux on Creodias
How to download EODATA files using s3cmd on Creodias
If you want to access EODATA on a Windows VM, you can do it by following this article:
How to mount EODATA on Windows virtual machine on Creodias hosting
If you find that the amount of disk space on your VM is insufficient for your EODATA processing, you can create and attach a volume to it. You can do so by following one of these articles: