Podman volume backups

Version: 1.2.1 Updated: 2024-05-04

Keeping backups of your volume data in your containers is important and requires a bit of work, and if you are a home user like me or a small company you might not afford big expensive container backup solutions this script might be an option for you to use.

This script exports all your volumes with a built in command in podman for exporting volumes as .tar files and then you can either store them locally only for any given amount of days and / or send them to offsite storage over SFTP to keep a backup elsewhere.

Each volume will reside in it's own .tar file to make it easier to restore / import the volume data for a single volume when needed.

Important to note is that depending on your OS, you might not be able to add the pysftp python module when running as root, here you will need to set up and virtual environment for your script and recommended is to create it under the same user as you are running your containers, i will explain below on how to do that.

Settings

There are some settings in the script that must be set to make it tick, some settings already has a default value set but can be changed as needed.

Backup folder to use during creation of volume exports and to store files locally.

vBckDir

Name prefix of files, _date and .tar is added at the end, ex. prefix_volumename_date.tar.

vFilePrefix

Keep local backup files after sent to SFTP server, if no than nothing is kept locally. (no/yes)

vKeepBackup

Number of days to keep local files before pruning the backup directory, relies on vKeepBackup.

vKeepDays

Should we send the files to a Sftp server, requires the pysftp python module. (no/yes)

vSendToSftp

User for the remote server.

vSftpUser

Password for the remote server.

vSftpPass

Use key file as authenticator against remote server for SFTP.

vSftpUseKey

Full path and key to use when connecting via key file instead of username / password.

vSftpKeyFile

Destination folder on remote server.

vSftpDir

Remote server address.

vSftpHost

Remote server port.

vSftpPort

Run extra OS specific commands before backup. (no/yes)

vPreBckCmd

Run extra OS specific commands after backup. (no/yes)

vPostBckCmd

OS Commands

Depending on some settings above, these 2 must be populated as well, these are the OS commands you want to execute either before or after export of volumes and depends on the vPreBckCmd and / or vPostBckCmd being set to yes.

External OS commands to execute before continuing with the rest of the script.

vPreOsCmd

External OS commands to execute at the end of the script.

vPostOsCmd

These can contain multiple commands when needed, to run multiple command for example vPreOsCmd could look like this, there are no theoretical no limit to amount of commands you can add, but try to keep it to what is needed for the script.

vPreOsCmd=["command1","command2"]

Exclude volumes

When you want to exclude some volumes from backup you can add their name here, can be full name or a word within a volume name to exclude many of same type, the naming is case sensitive so for example db will not match DB and so on.

For example we have one special log volume we do not want to backup and heaps of database volumes that should not be backed up.

In below example the full name of the single log volume is volume-mycontainer-logs and all database volumes are named with the word database in the volume name, this is how i would go about to add them to the exclusion list.

vExcludePattern=["volume-mycontainer-logs","database"]

You can have as many exclusions as desired, just note that it will take longer time to do the backup if you have an insanely amount of exclusions.

Python virtual environment

A python virtual environment or venv as it is called is runtime version of python separated from the rest of the OS, this has several benefits, no extra modules are installed with extra permissions and you can lock it on a certain python version in case you need different versions on your system.

As of Ubuntu 23.04 this is a requirement, they have locked down the system python version and you cannot install any extra modules, might be other distributions going this way or is already there, it is better to start using a venv anyway.

Before creating the virtual environment you need to make sure you have pythons venv package, on Ubuntu it is called python3-venv, when creating the venv and if it fails it will most of the times tell you what package it wants installed, install that package and try again, be sure to remove the venv folder you created first or it will most likely complain.

Let's assume that our user is podman and it is the account running our containers, like i used in my podman how-to, go and read it if you haven't.

Start with making sure you are placed in the home directory.

$ cd
$ pwd
/home/podman

Run the following command to create a venv for this script, i set the name to oVolumeBackup since it is an abbreviation on my site name and script function, but you can call it whatever you like.

$ python3 -m venv .venv/oVolumeBackup

If above commands fails after you installed required package remove the oVolumeBackup folder from .venv/ directory and try again.

And if your are going to send to SFTP do the following to get the pysftp module installed.

$ source .venv/oVolumeBackup/bin/activate
$ pip3 install wheel
$ pip3 install pysftp
$ deactivate

Now we need to change the first line in the script from this.

#!/usr/bin/python3

To this, so we point it at our venv we just created.

#!/home/podman/.venv/oVolumeBackup/bin/python3

Remote ssh key...

You need to ssh to the remote server before running the script first time to accept the host key when using the SFTP option or the script will fail, and it must be with the user running the script later.

To run the script simply set the execution mode like below so that the owner (user & group) of the script has full permissions on the script and everybody else has none, recommended since there are passwords in the file if you choose to use the SFTP function with username / password.

$ chmod 770 ./oVolumeBackup.py

And this enables you to run it like this.

$ ./oVolumeBackup.py

Now you should have everything set up for running volume backup, or almost, you still need the script, and here it comes...

The Script

The script has been tested against the following Podman & Python versions.

Podman: 3.4.4 / 4.3.1
Python: 3.4.4 / 3.11.6

GitHub

This script now resides on GitHub and newer versions will be uploaded there from now on.

This page will still exist with information about the script and how to configure it for now...

You can reach it here: GitHub

Changelog

ChangeLog - Expand to read

Version     Date            Information
-------     ----            -----------
1.2.1       2024-05-04      Added the option to only include volumes with certain word in them.
                            minor bug fix.
1.2.0       2024-04-16      Added the option to exclude volumes based on names or words.
                            Minor bug fixes.
1.1.1       2024-04-10      Refactored most of the code.
1.1.0       2024-03-25      Enabled the use of key files instead of password
                            when connecting to sftp server.
1.0.0       2023-09-25      Initial release.