Backup of Kubernetes Cluster using Velero

What is Velero

Velero is the official Open-source project from VMware: https://velero.io. It can back up all Kubernetes API objects and persistent volumes from the cluster on which it is installed. Backed up objects can be restored on the same cluster, or on a new one. Using a package like Velero is essential for any serious development in the Kubernetes cluster.

What We Are Going To Cover

Preparing for the installation of Velero (update and upgrade your environment, secure access to Kubernetes cluster etc.)

Authorizing to OpenStack and installing the Swift module from OpenStack suite of modules

Getting EC2 Client Credentials

Installing Helm to help us automate the installation of Velero

Adjusting “values.yaml”, the configuration file

Creating namespaces to gain precise access to the Kubernetes cluster

Installing Velero with a Helm chart

Installing and deleting backups using Velero

Example 1 Basics of Restoring an Application

Example 2 Snapshot of Restoring an Application

Before Installing Velero

It is highly recommended to update & upgrade your environment (example shown here is for Ubuntu):

$ sudo apt update && sudo apt upgrade

It will be necessary to have access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled. For more information on supported Kubernetes versions, see the Velero compatibility matrix.

To use Velero, you will need kubectl. Its installation is described on Install Tools page of the official Kubernetes site.

Using Swift

The OpenStack Object Store module, known as Swift, allows you to store and retrieve data with a simple API. It’s built for scale and is optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound.

Step 1 Authorize to OpenStack and Install the Swift Module

Before the initial deployment, EC2 credentials should be fetched from OpenStack.

See the details: How to install OpenStackClient for Linux on CREODIAS

At the end of that article you will see a link to article How to activate OpenStack CLI access to CREODIAS cloud using one- or two-factor authentication. It shows how to authenticate your account to the cloud, from Linux and Windows, and even if your account has two factor authentication turned on.

Assuming that the openstack command is fully operational, use the following commands to install swift module and create a bucket named “backup”:

$ sudo apt-get install python3-pip # install pip
$ pip install python-swiftclient # install python openstack client
$ swift post backup # where "backup" is the name of your container (bucket)

Attention

Container name cannot contain spaces and special characters. Using a name “backup velero” or “backup-velero” will result in an error message in the next steps, while a name such as “backupvelero” will not.

Step 2 Getting EC2 Client Credentials

EC2 credentials are necessary to access private bucket (container).

Users are able to generate ec2 credentials on their own by invoking these commands:

$ openstack ec2 credentials create
$ openstack ec2 credentials list

Save somewhere the Access Key and the Secret Key. They will be needed in the configuration file.

Step 3 Installing Helm

Helm is a package for installing Kubernetes packages. First you install Helm and then use it to install Velero.

Helm has an installer script that will automatically grab the latest version and install it locally. The commands are:

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh
Downloading https://get.helm.sh/helm-v3.9.2-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
[sudo] password for User:
helm installed into /usr/local/bin/helm

Check whether Helm has been properly installed using the following command:

$ helm version
version.BuildInfo{Version:"v3.9.2", GitCommit:"1addefbfe665c350f4daf868a9adc5600cc064fd", GitTreeState:"clean", GoVersion:"go1.17.12"}

Step 4 Adjust the Configuration file - “values.yaml”

The next step will help you to adjust the configuration file to your needs.

Use a text editor of your choice to create that file, on MacOS or Linux you can use nano, like this:

sudo nano values.yaml

Use the configuration file provided below. Please fill in the required fields, which are marked with #:

Attention

Fields name and bucket cannot contain spaces and special characters. Using a name “backup velero” or “backup-velero” will result in an error message in the next steps, while a name such as “backupvelero” will not.

Values.yaml

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: ## enter name of backup storage location (could be anything)
    bucket: ## enter name of bucket created in openstack
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: ## enter URL of object storage (for example "https://s3.waw3-1.cloudferro.com")
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id=
      aws_secret_access_key=
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make.
    template:
      ttl: "240h" ## choose ttl, after which the backups will be removed.
      snapshotVolumes: false

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: ## enter name of backup storage location (could be anything)
    bucket: ## enter name of bucket created in openstack
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: ## enter URL of object storage (for example "https://cloud.waw3-2.cloudferro.com")
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id=
      aws_secret_access_key=
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make.
    template:
      ttl: "240h" ## choose ttl, after which the backups will be removed.
      snapshotVolumes: false

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: ## enter name of backup storage location (could be anything)
    bucket: ## enter name of bucket created in openstack
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: ## enter URL of object storage (for example "https://s3.fra1-2.cloudferro.com")
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id=
      aws_secret_access_key=
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make.
    template:
      ttl: "240h" ## choose ttl, after which the backups will be removed.
      snapshotVolumes: false

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: ## enter name of backup storage location (could be anything)
    bucket: ## enter name of bucket created in openstack
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: ## enter URL of object storage (for example "https://s3.waw2-1.cloudferro.com")
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id=
      aws_secret_access_key=
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make.
    template:
      ttl: "240h" ## choose ttl, after which the backups will be removed.
      snapshotVolumes: false

Paste the content to the configuration file “values.yaml” and save.

Example of an already configured file:

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: velerobackuptesting
    bucket: bucket
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: https://s3.waw3-1.cloudferro.com
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x
      aws_secret_access_key= dee1581dac214d3dsa34037e826f9148
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 * * *"
    template:
      ttl: "168h"
      snapshotVolumes: false

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: velerobackuptesting
    bucket: bucket
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: https://s3.waw3-2.cloudferro.com
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x
      aws_secret_access_key= dee1581dac214d3dsa34037e826f9148
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 * * *"
    template:
      ttl: "168h"
      snapshotVolumes: false

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: velerobackuptesting
    bucket: bucket
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: https://s3.fra1-2.cloudferro.com
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x
      aws_secret_access_key= dee1581dac214d3dsa34037e826f9148
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 * * *"
    template:
      ttl: "168h"
      snapshotVolumes: false

initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.4.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins

configuration:
  provider: aws
  backupStorageLocation:
    name: velerobackuptesting
    bucket: bucket
    default: true
    config:
      region: RegionOne
      s3ForcePathStyle: true
      s3Url: https://s3.waw2-1.cloudferro.com
credentials:
  secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret.
    cloud: |
      [default]
      aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x
      aws_secret_access_key= dee1581dac214d3dsa34037e826f9148
  ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents.
snapshotsEnabled: false
deployRestic: true
restic:
  podVolumePath: /var/lib/kubelet/pods
  privileged: true
schedules:
  mybackup:
    disabled: false
    schedule: "0 * * *"
    template:
      ttl: "168h"
      snapshotVolumes: false

Step 5 Creating Namespace

Namespaces are a way to organize clusters into virtual sub-clusters — they can be helpful when different teams or projects share a Kubernetes cluster. Any number of namespaces are supported within a cluster, each logically separated from others but with the ability to communicate with each other.

Before creating namespace, verify that kubectl have access to your Cloud. See basic data about the cluster with the following command:

$ kubectl get nodes -o wide

If the output of the command shows your cluster / clusters, it confirms kubectl has proper access to the cloud.

Accordingly, you can use this command to create namespace:

$ kubectl create namespace <name of your namespace>

$ kubectl create namespace veleronamespace
namespace/veleronamespace created

Step 6 Installing Velero with a Helm chart

$ helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
"vmware-tanzu" has been added to your repositories
$ helm install vmware-tanzu/velero --namespace <name of your namespace> --version 2.28 -f values.yaml --generate-name

$ helm install vmware-tanzu/velero --namespace veleronamespace --version 2.28 -f values.yaml --generate-name
NAME: velero-1658856389
LAST DEPLOYED: Tue Jul 26 19:26:39 2022
NAMESPACE: veleronamespace
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:

Check that Velero is up and running:

$ kubectl get deployment/<NAME> -n <namespace>

$ kubectl get deployment/velero-1658856389 -n veleronamespace
NAME              READY     UP-TO-DATE     AVAILABLE AGE
velero-1658856389  1/1         1 1              45s

Check that the secret has been created:

$ kubectl get secret/<NAME> -n <namespace>

$ kubectl get secret/velero-1658856389 -n veleronamespace
       NAME                TYPE     DATA     AGE
velero-1658856389         Opaque     1       50s

Step 7 Installing Velero CLI

The final step is to install Velero CLI – Command Line Interface suitable for working from the terminal window on your operating system.

Download the client specified for your operating system from: https://github.com/vmware-tanzu/velero/releases using wget (It is recommended to download the latest version):

$ wget https://github.com/vmware-tanzu/velero/releases/download/v1.9.1/velero-v1.9.1-linux-amd64.tar.gz
--2022-08-27 12:32:58-- https://github.com/vmware-tanzu/velero/releases/download/v1.9.1/velero-v1.9.1-linux-amd64.tar.gz
Saving to: ‘velero-v1.9.1-linux-amd64.tar.gz’
2022-08-27 12:36:29 (156 KB/s) - ‘velero-v1.9.1-linux-amd64.tar.gz’ saved [27924025/27924025]

Extract the tarball:

$ tar -xvf <RELEASE-TARBALL-NAME>.tar.gz
velero-v1.9.1-linux-amd64/LICENSE
velero-v1.9.1-linux-amd64/examples/README.md
velero-v1.9.1-linux-amd64/examples/minio
velero-v1.9.1-linux-amd64/examples/minio/00-minio-deployment.yaml
velero-v1.9.1-linux-amd64/examples/nginx-app
velero-v1.9.1-linux-amd64/examples/nginx-app/README.md
velero-v1.9.1-linux-amd64/examples/nginx-app/base.yaml
velero-v1.9.1-linux-amd64/examples/nginx-app/with-pv.yaml
velero-v1.9.1-linux-amd64/velero

Move the extracted velero binary to somewhere in your $PATH (/usr/local/bin for most users):

$ mv <RELEASE-TARBALL-NAME> $PATH
# System might force using sudo
$ sudo mv <RELEASE-TARBALL-NAME> $PATH
# check if velero is working
$ velero version
Client:
   Version: v1.9.1
   Git commit: 6021f148c4d7721285e815a3e1af761262bff029

After this operations, you should be allowed to use velero commands. For help how to use them, execute:

$ velero help

Working with Velero

If you want to restore an object or persistent volume in the same cluster, you must first delete the existing one. If you want to restore backups from a cluster to another cluster, it’s possible. You must install Velero on a new cluster and connect to the same s3 bucket where the backups are stored. Example commands:

Backup all API objects:

$ velero backup create <name of backup>

$ velero backup create test
Backup request "test" submitted successfully.

Backup all api objects in default namespace:

$ velero backup create <name of backup> --include-namespaces <name of namespace>

$ velero backup create test --include-namespaces default
Backup request "test" submitted successfully.

Show backups:

$ velero backup get
NAME   STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup  New      0       0     <nil>   n/a                      <none>
test    New      0       0     <nil>   n/a                      <none>
test-1  New      0       0     <nil>   n/a                      <none>

Restore from backup:

Important

If you want to restore an object, first delete it from the cluster.

$ velero restore create <name of restore> --from-backup <name of backup>

$ velero restore create restore --from-backup test
Restore request "restore" submitted successfully.
Run `velero restore describe <name of restore>` or `velero restore logs <name of restore>` for more details.

Example 1 Basics of Restoring an Application

We have also prepared some examples which you can clone and ensure that Velero is working properly.

You can clone the examples by executing:

$ git clone https://github.com/vmware-tanzu/velero.git
Cloning into 'velero'...
Resolving deltas: 100% (27049/27049), done.
$ cd velero

Start the sample nginx app:

$ kubectl apply -f examples/nginx-app/base.yml
kubectl apply -f base.yaml
namespace/nginx-example unchanged
deployment.apps/nginx-deployment unchanged
service/my-nginx unchanged

Create a backup:

$ velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.

Simulate a disaster:

$ kubectl delete namespaces nginx-example
# Wait for the namespace to be deleted
namespace "nginx-example" deleted

Restore your lost resources:

$ velero restore create --from-backup nginx-backup
Restore request "nginx-backup-20220728013338" submitted successfully.
Run `velero restore describe nginx-backup-20220728013338` or `velero restore logs nginx-backup-20220728013338` for more details.

$ velero backup get
NAME         STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup         New     0       0     <nil>   n/a                      <none>
nginx-backup   New     0       0     <nil>   n/a                      <none>

Example 2 Snapshot of Restoring an Application

Start the sample nginx app:

$ kubectl apply -f examples/nginx-app/with-pv.yaml
namespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created

Create a backup with PV snapshotting:

$ velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.

Simulate a disaster:

$ kubectl delete namespaces nginx-example
namespace "nginx-example" deleted

Important

Because the default reclaim policy for dynamically-provisioned PVs is “Delete”, these commands should trigger your cloud provider to delete the disk that backs up the PV. Deletion is asynchronous, so this may take some time.

Restore your lost resources:

$ velero restore create --from-backup nginx-backup
Restore request "nginx-backup-20220728015234" submitted successfully.
Run `velero restore describe nginx-backup-20220728015234` or `velero restore logs nginx-backup-20220728015234` for more details.

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

https://kubernetes.io/docs/

https://helm.sh/docs/

https://velero.io/docs/