Use script to create daily weekly and monthly rotating backups of virtual machines on CREODIAS

Rotating backups in OpenStack refer to a backup strategy where older backups are automatically deleted after a predefined number of backups are created. This ensures that storage does not grow indefinitely while still maintaining a set number of recent backups for disaster recovery.

backup create vs. image create

There are two ways of creating backups under OpenStack, using one of these two commands:

openstack server backup create and openstack server image create

Here is how they compare:

Comparison of Backup and Image Creation Commands
Feature	`openstack server backup create`	`openstack server image create`
Association with VM	Associated using backup image property	Associated using backup name
Rotation support	Rotation with `--backup-type` and incremental backups	No built-in rotation support
Classification in Horizon	Marked as image	Marked as snapshot
Horizon Select Boot Source	Choose Instance Snapshot	Choose Image
Purpose	Primarily used for backups, can be rotated and managed	Creates a single VM snapshot without rotation
Multiple rotating schedules	No, only one	Yes (daily, weekly, monthly etc.)
Incremental backup support	Yes, supports incremental backups	No, always creates a full snapshot
Best usage scenario	Automated backup strategies with rotation	Capturing the current state of a VM for cloning or rollback
Can be scripted?	Yes	Yes

In this article, you will learn how to create multiple series of rotating backups with a script which uses multiple OpenStackClient commands to achieve this goal.

Prerequisites

No. 1 Account

You need a CREODIAS hosting account with access to the Horizon interface: https://horizon.cloudferro.com

No. 2 VM which will be backed up

You need a virtual machine which will be backed up. If you don’t have one, you can create it by following one of these articles:

To learn how to create a Windows virtual machine, see this article: How to create Windows VM on OpenStack Horizon and access it via web console on CREODIAS

The name of the virtual machine we are going to back up in this article will be vm-to-backup. Be sure to supply your own name throughout the article.

This article only covers backing up virtual machines which

do not have any volumes attached to them, and

use ephemeral storage (and not persistent storage).

See Ephemeral vs Persistent storage option Create New Volume on CREODIAS

No. 3 Server or other computer on which the backup process will be performed

You need an environment from which the commands for backing up your vm-to-backup will be executed.

In this article, we assume that this environment runs Ubuntu 24.04 LTS and that you can manage it. We will call it backing-server since this could be a name of a VM used for this purpose.

This can be, among others:

A virtual machine on CREODIAS cloud (even one located in another region - for example, vm-to-backup could be in WAW4-1 and backing-server could be in WAW3-2)

Your own physical server

A virtual machine running outside of CREODIAS cloud

Make sure that from this environment you can connect to CREODIAS cloud.

This environment must be running each time you want to create a new backup.

No. 4 OpenStackClient installed on backing-server

You need to have OpenStackClient installed on backing-server. See How to install OpenStackClient for Linux on CREODIAS

No. 5 Install application credentials

You might be using a two-factor authentication scheme to log in as a user to CREODIAS. Here we will be performing automated operations, so authentication should be automated as well.

For this purpose, we will use application credentials which we will store in a clouds.yml file.

Follow How to generate or use Application Credentials via CLI on CREODIAS to create a clouds.yml file with your application credentials.

Copy clouds.yml to backing-server. Once installed, openstack command searches for clouds.yml across the following two locations, by default:

$HOME/.config/openstack/clouds.yml

/etc/openstack/clouds.yml

so you can place clouds.yml file in one of these. Then you will be able to easily use this file to authenticate from any folder in the operating system.

$HOME is the environment variable which usually denotes your home directory.

You can also place clouds.yml into any other folder you like but then be sure to execute openstack command from that folder as well.

Important

The clouds.yml file stores credentials in plain text, without additional encryption. Therefore, make sure that whatever location you choose to store these credentials is sufficiently secure.

clouds.yml might store multiple pairs of credentials. To choose the appropriate pair of credentials, we will be using --os-cloud parameter. This article will use placeholder value backup_test for the name of your pair of credentials. Make sure to use your own each time you see it.

On CREODIAS, application credentials work on multiple regions, so we will specify the region which we will use using --os-region parameter. The value of this parameter is the name of the region which you are using, for example WAW3-2 or WAW4-1.

To test the waters, the following command should list virtual machines in your project:

openstack server list \
--os-cloud backup_test \
--os-region WAW3-2

Replace the actual values used here, backup_test and WAW3-2, with the concrete values from your environment.

No. 6 Knowledge of cron under Linux

cron is a standard Linux command for executing tasks according to a predefined schedule. We will be using cron in this article to automate rotating backups.

If cron is installed, the following command

systemctl status cron

will produce output like this:

If not, here is how to install cron:

sudo apt update
sudo apt install cron -y
sudo systemctl enable --now cron
systemctl status cron

The result should be similar to the image above.

No. 7 Your favorite plain text editor installed on backing-server

Two of the most popular editors are vim and nano. Here’s how to install them.

vim

sudo apt install vim

nano

sudo apt install nano

To choose default text editor on Ubuntu 24.04, execute:

select-editor

You should be prompted to choose the editor by providing its number, for example:

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.basic
  3. /usr/bin/vim.tiny
  4. /bin/ed

Choose 1-4 [1]:

Choose the one which you want to use by providing its number and press Enter.

The rotating backup algorithm

Before the backup script is run for the first time, the following things need to be done:

Defining the name of series of backups: For example: daily, weekly or monthly
Defining rotation limit: How many backups to retain (we will refer to this number as maxN throughout this article).

Once these requirements are met, each time you execute the backup script, this happens:

Renaming of backups: Renaming of the existing backups so that the their numbers continue to represent which backup is the oldest and which backup is the newest. Backup No. 1 must always be the oldest file.
Deleting of oldest backup: If the number of backups which exist is now greater than maxN, the oldest backup gets deleted.
Creating of a backup: A backup with number 1 in its name gets created.

Creating a backup manually without rotating

Command server image create creates full snapshots but has no built in parameter for rotating backups. The simplest possible form of the command in this environment will look like:

openstack server image create \
--name my-vm-backup \
--os-cloud backup_test \
--os-region WAW3-2 \
my-vm

In this command:

my-vm-backup is the name of the backup

my-vm is the name of your virtual machine

The remaining two parameters, --os-cloud and --os-region are for choosing the name of application credentials and region which you will be using to authenticate. For more information, see Prerequisite No. 5.

This is the general shape of command to delete an old image:

openstack image delete <old-image>

where <old-image-id> is name or ID of image which you want to remove.

In this article, we would use command

openstack image delete \
--os-cloud backup_test \
--os-region WAW3-2 \
my-vm-backup

if we wanted to delete the image my-vm-backup.

Multiple rotating schedules

These commands are strictly manual, which may be error-prone. You may forget to execute them, use a wrong name for VM or backup and so on. The solution is to use a script which will execute in predefined moments of time. This, paradoxically, gives full freedom to organize rotating backups not only in one series of backups but in as many series as you need.

This script creates snapshots using the same method as the one explained in the previous section but adds backup rotation and multiple schedules.

This script will use command

openstack image set to rename the existing backups so that their names show which one is the oldest and which one is the newest and so on,

openstack server image create to create a backup,

openstack image delete to delete the oldest backup

This makes organizing multiple series of rotating backups more convenient.

Each backup will have name which consists of:

name of instance, say, vm-to-backup

a dash -

name of backup series, say, weekly

a dash -

number of backup versions from 1 to maxN, say, 3.

The name would then be: vm-to-backup-weekly-3.

Step 1: Create a script

Open your favorite plain text editor (like vim or nano), create file backup.sh and enter the following script:

#!/bin/bash
if [[ "$#" -ne 3 ]]
then
  echo "Expecting 3 arguments: vm name, backup name, number of backups"
  exit
fi
vm_name=$1
backup_core_name=$1"-"$2
reifnumber='^[0-9]+$'
if ! [[ $3 =~ $reifnumber ]]
then
  echo "Given backup count is not a number"
  exit
fi
count=$3
# Rename all previous backups - add 1 to their number
for i in $(seq $count -1 1)
do
  openstack image set --name $backup_core_name"-"$((i+1)) $backup_core_name"-"$i --os-cloud backup_test --os-region WAW3-2
done
# Delete oldest backup (the one which has number number greater by 1 than the number of backups to rotate
openstack image delete $backup_core_name"-"$((count+1)) --os-cloud backup_test --os-region WAW3-2
# Create a new backup - with number 1
openstack server image create --name $backup_core_name"-1" $vm_name --os-cloud backup_test --os-region WAW3-2

As usual, in this script, change value of each:

--os-cloud parameter to the name of credentials used in your clouds.yaml

--os-region parameter to the name of cloud you are using.

Save backup.sh in your home directory.

Add executable permissions to it:

chmod +x backup.sh

Step 2: Run the script

From within your home directory, try to create a backup with this script.

The script accepts three arguments:

name (or ID) of the virtual machine, our VM is called vm-to-backup,

name of the series of backups, our series of backups is called manual-backup, and

number of backups, our maxN is 3

This is the command to run the script:

./backup.sh vm-to-backup manual-backup 3

You should get output similar to this:

Show backup files

The backup files that the script creates are actually snapshots. The following command will list all of the snapshots in the project (subject to constraint that image_type=snapshot), regardless of the method used to create each particular snapshot:

openstack image list \
--property image_type=snapshot \
--os-cloud backup_test \
--os-region WAW3-2

Some snapshots may be the backups you created, some other snapshots may have been imported and so on.

Running the script

Here is what happens when you run this script with the same options as in the example once, twice, three times and four times in a row.

Backups before this run

There are no backups in our project.

Attempt to rename

The script attempted to rename images - change numbers of backups:

Original number	Number after rename	Outcome
3	4	Fail
2	3	Fail
1	2	Fail

Since these images do not exist yet in our project, it was not successful in any of these operations - that’s where No Image found lines of the output come from.

Delete the oldest image

The script attempted to delete image named vm-to-backup-manual-backup-4 because its number is greater by one than maxN. There is no such image, so it was not successful.

Create a new backup

The script created a new backup named vm-to-backup-manual-backup-1. This operation was successful.

Backups after this run

The number of backups in the project boils down to one (for now):

vm-to-backup-manual-backup-1

Step 3: Open the crontab file

We are now going to use cron from Prerequisite No. 6 to automate rotating backups.

There are several methods of opening the file which contains your cron jobs:

Method 1: Access file without choosing the text editor

The basic command to define cron jobs is:

crontab -e

If you execute it, one of the following things might happen:

your file is opened in the default text editor set in your system, or

you are prompted to choose the default text editor as explained in Prerequisite No. 7 and then the file is opened in that text editor

Method 2: Choose the text editor each time you open the crontab file

You can use the EDITOR environment variable to choose the editor you want to use to edit your crontab file each time you run it.

For example, to open the crontab file in nano, execute:

EDITOR=nano crontab -e

Alternatively, to use vim, you can execute:

EDITOR=vim crontab -e

Step 4: Edit the crontab file

You can remove any previous schedules of backups from this file if you want to.

Create new appropriate cron jobs for different schedules of backups, according to your needs. This is what the crontab data would look like (taking into account the examples that follow):

The preloaded comments in the file define how to use the parameters.

Once you’ve finished, close the file and close the editor. If you defined your jobs correctly, they should start being executed.

Examples of cron jobs

Daily

10 0 * * * /home/eouser/backup.sh vm-to-backup daily_backup 5

This job runs a daily backup which takes place each day 10 minutes after midnight. The name of backup is daily_backup and name of the virtual machine is vm-to-backup. The number of backups to be rotated is 5.

After five days of running of this script, the list of backups created for this VM should be as follows:

where the backup with number 1 is the newest and backup with number 5 is the oldest.

Weekly

15 0 * * 1 /home/eouser/backup.sh vm-to-backup weekly_backup 4

In this case, every first day of the week at 15 minutes after midnight, a backup named weekly_backup of VM called vm-to-backup is created. Four backups will be rotated.

After 4 weeks of running of this script, the list of backups created for this VM should be as follows:

Monthly

20 0 2 * * /home/eouser/backup.sh vm-to-backup monthly_backup 3

In this case, every second day of the month at 20 minutes after midnight, a backup named monthly_backup of VM called vm-to-be-backed_up is created. Three backups will be rotated.

After 3 months of running of this script, the list of backups created for this VM should be as follows:

Limitations

This script illustrates one possible way of creating multiple series of rotating backups.

Note, however, that once you have one series of backups, you should not run the script for that series with a smaller number of backups from that series than there actually are. This will cause the script not to function properly.

For example, if you have a VM called my-vm and you have a series of its 5 backups named weekly, do not run the following command:

./backup.sh my-vm weekly 4

Apart from that, the script does not include any error handling or reporting, so failure to perform a backup (for instance due to no connection to the OpenStackClient API) might lead to unexpected results.

Also, do not use similar or the same names or naming conventions for images you might create without using this script. This might cause the script to behave in unexpected way.

How to restore backups?

Backups created using the script provided in this article, just like other images in OpenStack, can be used to

create new virtual machines, or

replace contents of a hard drive of a virtual machine with contents of such an image (rebuild).

In case of restoring backups, the former might be useful for, say, having multiple virtual machines where each represents a different approach to a problem you are trying to solve. The latter can be useful if you simply want to restore the state of a virtual machine from previous backup.

However, before you do that, it is recommended to copy that image so that the script does not have its series of backups broken in an unexpected way and can continue to work normally.

This section covers all of these operations.

Step 1 Find size and name or ID of image which you want to restore

Check the ID or name of backup you want to restore by listing snapshots located in your project.

openstack image list \
--property image_type=snapshot \
--os-cloud backup_test \
--os-region WAW3-2

In this example, we will choose my-backup2

Write somewhere down the name or ID of this image.

For the next step, you will also need to learn the minimum size of disk.

To do it, execute the command below:

openstack image show \
-c min_disk \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2

where my-backup2 is the name or ID of your image.

This output should return a table containing the desired value in GB:

+----------+-------+
| Field    | Value |
+----------+-------+
| min_disk | 8     |
+----------+-------+

Step 2: Create a volume from image

Because in OpenStack you can’t copy an image directly, you will first need to create a volume from it.

Execute the following command to create a volume from your image:

openstack volume create \
--image my-backup2 \
--size 8 \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2-volume

where:

my-backup2 is the name or ID of your image

8 is the minimum size of volume for your image in GB (this is a temporary volume, we don’t need it to be larger)

my-backup2-volume is the name of the volume you are creating

The output of this command should contain information about your volume. At first it will likely have the following status: uploading

Wait until it has the following status: available

You can view its current status by executing the following command:

openstack volume show \
-c status \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2-volume

where my-backup2-volume is the name or ID of your volume.

Step 3: Create an image from volume

From that volume, we can create an image which can be used for creating or rebuilding a virtual machine.

Execute the command below:

openstack image create \
--volume my-backup2-volume \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2-copy

where:

my-backup2-volume is the name of the volume

my-backup2-copy is the name of copy of image you are creating

The output of this command should contain information about your image. At first it will likely have the following status: uploading

Wait until it has the following status: active

You can view its current status by executing the following command:

openstack image show \
-c status \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2-copy

where my-backup2-copy is the name or ID of your image

Step 4: Delete temporary volume

Now that you’ve created an image from your volume, you can delete the volume.

Do it with this command:

openstack volume delete \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2-volume

where my-backup2-volume is the name or ID of the volume you want to delete.

The output of this command should be empty.

Step 5: Restore backup

We have now created a copy of the backup we want to restore. Thanks to that, the work of the script should continue uninterrupted and the script should be able to delete the original backup when it fits the schedule.

This copy can now be restored to a virtual machine.

Two methods of doing so are covered in this section. Choose the one that suits you best.

Method 1: Rebuilding of an instance

Rebuilding an instance means replacing the current contents of your virtual machine hard drive with the contents of an image. In this process, metadata such as VM name or ID will be kept. This includes floating IPs.

During this process, the virtual machine will be unavailable.

Warning

This operation will remove data from virtual machine on which these actions are performed. Make sure that this virtual machine does not contain any data which you don’t want to lose!

The command to restore from a backup is:

openstack server rebuild \
--image my-backup2-copy \
--os-cloud backup_test \
--os-region WAW3-2 \
vm-to-backup

Here:

my-backup2-copy is the ID or name of your image

vm-to-backup is the ID or name of your instance

You should get the output similar to this:

Note that now the VM is in status: REBUILD

Wait until the restoration is completed. Once this happens, the virtual machine should be turned on and ready to go.

You can check the current status of your VM with command below (where vm-to-backup is the name or ID of your VM):

openstack server show \
--os-cloud backup_test \
--os-region WAW3-2 \
vm-to-backup

If you want to see only its status, add the -c status parameter:

openstack server show \
-c status \
--os-cloud backup_test \
--os-region WAW3-2 \
vm-to-backup

If the status is ACTIVE, your VM should be ready and operational.

Method 2: Creating a brand new virtual machine from backup

Since images created with this command are very similar to normal OpenStack images, you can use them to create a new virtual machine from scratch. Such a VM will have files and operating system identical to the ones stored on a backup. This process does not affect the original virtual machine.

This can be useful if you, say, want to test multiple ways of doing something, each on its own VM.

It goes without saying that here ID of the virtual machine will not be preserved, but you will be able to choose the same name as previously.

If this machine had any Floating IPs and you want them attached to a new virtual machine, you will need to unassign them from the previous virtual machine. Then the Floating IP will remain in your project and once the new VM has been created, you can assign that IP to the new virtual machine.

If you release the floating IP from your project, it will no longer be available to you.

Follow instructions from /cloud/How-to-create-a-VM-using-the-OpenStack-CLI-client-on-Creodias

In section Image of Step 1 of that article:

use the ID of one of the backups you created.

In section Flavor, choose flavor which is sufficient to handle your backup. It will likely be the same flavor as one used for original virtual machine.

Of course, you can also make restoration of backups part of a cron job. This is outside of scope of this article.

Problems and troubleshooting

When using command

openstack image create \
--volume my-backup2-volume \
--os-cloud backup_test \
--os-region WAW3-2 \
my-backup2-copy

you may get message like this:

VolumeManager.upload_to_image() got an unexpected keyword argument ‘visibility’ when trying to create an image from Volume

It means that the version of OpenStack CLI that you are working with and the version of OpenStack in the cloud are mismatched. A practical workaround is to set up OS_VOLUME_API_VERSION to 3.1 or greater value.

You can also use parameter

--os-volume-api-version 3.1

with the command openstack image create.

What To Do Next

The script we used in this article creates instance snapshots. See How to create instance snapshot using Horizon on CREODIAS for additional information.

If you don’t need multiple rotating schedules, the official command openstack server backup create might be a better choice. See Use backup command to create rotating backups of virtual machines on CREODIAS cloud.