How to mount EODATA as a filesystem using Goofys in Linux on CREODIAS

EODATA repository contains dozens of petabytes of Earth observation data from missions such as Sentinel-2A or Sentinel-5P. Transferring that amount of data is not feasible in most cases, but on CREODIAS it is possible to access and process the data directly from the cloud, without any downloading. The idea is to

  • create virtual machine in the cloud

  • install software such as goofys to mount S3 containers and then

  • mount the bucket containing EODATA repository on your Linux VM.

The net effect is to make the EODATA look like folders in Linux and then access them in the usual manner for that operating system.

Note

On CREODIAS, the virtual machines come with a configured mounting of the EODATA repository, using s3fs protocol. See: How to mount eodata using s3fs in Linux on CREODIAS. In contrast to that, this article is for users who want to use goofys.

Prerequisites

No. 1 Account

You need a CREODIAS hosting account with access to the Horizon interface: https://horizon.cloudferro.com.

No 2. Linux VM

You will need a Linux VM on the CREODIAS cloud with access to the EODATA network. You can create it by following one of the articles:

This article was created for Ubuntu 22.04. If you are using a different Linux distribution, adjust the commands accordingly.

No 3. Basic knowledge of Linux command line

Basic knowledge of Linux command line is required.

No. 4 Credentials

You need to obtain appropriate credentials to be able to configure goofys. Follow this article: How to get credentials used for accessing EODATA on a cloud VM on CREODIAS.

What We Are Going To Cover

  • Installing goofys

  • Providing credentials to configure goofys

  • Mounting the EODATA repository

  • Configuring automatic mounting of EODATA

  • Stopping the automatic mounting of EODATA repository using goofys

Step 1: Install goofys

Login to the eouser account of your virtual machine, using SSH.

Install wget to be able to download the goofys binary directly to your virtual machine:

sudo apt update && sudo apt upgrade && sudo apt install wget

Now, navigate to your home directory:

cd

Use wget to download the goofys binary:

wget https://github.com/kahing/goofys/releases/latest/download/goofys

Make the goofys binary executable:

chmod +x goofys

Step 2: Provide credentials to configure goofys

Use Prerequisite No. 4 to obtain credentials for the cloud you are using. Let’s say you credential pair looks like this:

Q6ORITBEDIYQ1PPPPPPP:eeeeeeeeeeeegggggggggqjfkSDHGtqLGb0nYkpJ

Note

These values are just an example only, be sure to obtain your own credentials and use them in the rest of this article.

Make a folder to store your credentials:

mkdir ~/.aws/

Create file ~/.aws/credentials using your favorite text editor (for example nano or vim). Add the credentials pair you are working with:

[s3]
aws_access_key_id=Q6ORITBEDIYQ1PPPPPPP
aws_secret_access_key=eeeeeeeeeeeegggggggggqjfkSDHGtqLGb0nYkpJ

Save the file and exit the text editor.

Step 3: Mount EODATA

Create the directory which will be the mount point for EODATA, for example:

mkdir ~/eo

Mount EODATA using the following command:

./goofys --region RegionOne --profile s3 \
--endpoint http://data.cloudferro.com DIAS ~/eo

In this command,

  • ./goofys is the location of the goofys binary you downloaded

  • ~/eo is the location of the folder in which you want to mount the EODATA filesystem

  • DIAS is the name of the bucket which contains the EODATA resources.

You can replace the first two values listed above as needed.

In order to test whether the mount was successful, navigate to the directory in which you mounted the EODATA repository, for example:

cd ~/eo

Use the ls command to list its content. You should see the output similar to this:

../_images/mount-eodata-goofys-01_creodias.png

The EODATA repository has now been mounted using goofys.

After each reboot, you will need to execute the mounting command above again.

Step 4 Mounting EODATA on startup

Here is how to automatically mount the EODATA repository using goofys, at system startup.

If you still have the EODATA repository mounted using the method described in Step 3, reboot your virtual machine.

Create directory in /root folder to create a permanent place to store the credentials in:

sudo mkdir /root/.aws

Copy the credentials from local directory to .aws directory in /root folder:

sudo cp ~/.aws/credentials /root/.aws/

Now, add the line below to the file /etc/fstab. In order to do that, you need to start your text editor with elevated privileges (for example, using sudo).

/home/eouser/goofys#DIAS /home/eouser/eo fuse _netdev,allow_other,--dir-mode=0777,--file-mode=0666,--region=RegionOne,--profile=s3,--endpoint=http://data.cloudferro.com 0 0

In the above command,

  • /home/eouser/goofys is the goofys binary you downloaded and

  • /home/eouser/eo is the location in which you wish to mount EODATA.

Once you have added that line to your /etc/fstab file, reboot the VM. After the system has restarted, check whether the EODATA repository is mounted in the directory specified by you.

Stopping the automatic mounting of EODATA repository using goofys

If you want to stop automatic mounting of the EODATA repository, remove the line you added in Step 3 from /etc/fstab file. You can also comment it out by adding # character in front of that line.

After that, reboot the VM. Optionally, you can also remove the goofys binary and the credentials if you no longer want to use goofys.

How to update goofys in the future

To update goofys to the latest version :

  • make sure that the data in the EODATA repository is not actively used by any software on your virtual machine,

  • remove the goofys binary from your home directory,

  • execute the commands such as wget and chmod from the first part of this article to fetch the new version and

  • reboot your virtual machine.

What To Do Next

You can also try other methods of accessing EODATA, such as:

How to mount eodata using s3fs in Linux on CREODIAS

How to download EODATA files using s3cmd on CREODIAS

If you want to access EODATA on a Windows VM, you can do it by following this article:

How to mount EODATA on Windows virtual machine on CREODIAS hosting

If you find that the amount of disk space on your VM is insufficient for your EODATA processing, you can create and attach a volume to it. You can do so by following one of these articles: