Having a backup of your files is essential for every day operation, whether it is on your home servers or in a very small business environment with linux servers.
This script is built by me and used every day in my environment to make it easier for me to restore files when needed.
The script can send the compressed file to offsite storage with SFTP and / or keep them locally for a configured amount of time as well.
I have it running on openSUSE and Ubuntu servers as we speak with no issues, but if you would find something not working, please let me know, you can PM me att my Twitter(X) account and i will take a look at it.
Settings
There are some settings in the script that must be set to make it tick, some settings already has a default value set but can be changed as needed.
Backup and temp folder to use during creation of compressed file or to store files locally.
vBckDir
Name prefix of compressed file, _date and .zip is added at the end, ex. prefix_date.zip.
vFilePrefix
Keep local backup files after sent to SFTP server, if no than nothing is kept locally. (no/yes)
vKeepBackup
Number of days to keep local files before pruning the backup directory, relies on vKeepBackup.
vKeepDays
Should we send the files to a Sftp server, requires the pysftp python module. (no/yes)
vSendToSftp
User for the remote server.
vSftpUser
Password for the remote server.
vSftpPass
Destination folder on remote server.
vSftpDir
Remote server address.
vSftpHost
Remote server port.
vSftpPort
Run extra OS specific commands before backup. (no/yes)
vPreBckCmd
Run extra OS specific commands after backup. (no/yes)
vPostBckCmd
OS Commands
Depending on some settings above, these 2 must be populated as well, these are the OS commands you want to execute either before or after doing your backup and depends on the vPreBckCmd and / or vPostBckCmd being set to yes.
External OS commands to execute before continuing with the rest of the script.
vPreOsCmd
External OS commands to execute at the end of the script.
vPostOsCmd
These can contain multiple commands when needed, to run multiple command for example vPreOsCmd could look like this, there are no theoretical no limit to amount of commands you can add, but try to keep it to what is needed for the script.
vPreOsCmd=["command1","command2"]
Python virtual environment
A python virtual environment or venv as it is called is runtime version of python separated from the rest of the OS, this has several benefits, no extra modules are installed with extra permissions and you can lock it on a certain python version in case you need different versions on your system.
As of Ubuntu 23.04 this is a requirement, they have locked down the system python version and you cannot install any extra modules, might be other OS going this way so better to start using a venv anyway.
A venv is often created in the users home directory and in the user running the script, be it root or other user, replace youruser with your user
Start with making sure you are placed in the home directory.
$cd$pwd/home/youruser
Or for root it would be
$cd$pwd/root
Run the following command to create a venv for this script.
$python3-mvenv.venv/filebackup
And if your are going to send to SFTP do the following to get the pysftp module installed.
Now we need to change the first line in the script from this.
#!/usr/bin/python3
To this, so we point it at our venv we just created.
#!/home/youruser/.venv/filebackup/bin/python
Or as root
#!/root/.venv/filebackup/bin/python
Remote ssh key...
You need to ssh to the remote server before running the script first time to accept the host key when using the SFTP option or the script will fail, and it must be with the user running the script later.
To run the script simply set the execution mode like below so that the owner (user & group) of the script has full permissions on the script and everybody else has none, recommended since there are passwords in the file.
$chmod770./filebackup.py
And this enables you to run it like this.
$./filebackup.py
Now you should have everything set up for running a backup, or almost, you still need the script, and here it comes...
The Script
The script has been tested against the following OS & Python versions.
#!/usr/bin/python3################################################################## Basic file backup script written in python. ## ## Author: Marcus Uddenhed ## Revision: 1.4 ## Revision Date: 2023-05-30 ## Requirements: ## pysftp for SFTP functions, only if vSendToSftp is set to yes. ## #################################################################### Module imports.fromdatetimeimportdatetimefromtimeimporttimeimportzipfileimportos## Global variables.vBckDir=""# Backup and temp folder to use during creation of compressed file or to store files locally.vFilePrefix=""# Name prefix of compressed file, _date and .zip is added at the end, ex. prefix_date.zip.vKeepBackup="no"# Keep local backup files after sent to SFTP server, if no than nothing is kept locally.(no/yes)vKeepDays="20"# Days to keep if you want to keep certain days locally, relies on vKeepBackup.vSendToSftp="no"# Should we send the file to a Sftp server.(no/yes)vSftpUser=""# User for remote server.vSftpPass=""# Password for remote server.vSftpDir=""# Destination folder on remote server.vSftpHost=""# Remote server adress.vSftpPort="22"# Remote server port.vPreBckCmd="no"# Run extra OS specific commands before backup/zip.(no/yes)vPostBckCmd="no"# Run extra OS specific commands after backup/zip.(no/yes)# Folders to backup, this can be a single path or an list of paths, like this: ["/singlepath"] or ["/path1","path2"] and so on.vSrcDir=[""]# External OS commands to execute before compressing to zip, be sure to put the output folder to vSrcDir if it is to be added to zip file.vPreOsCmd=[""]# External OS commands to execute after compressing to zip.vPostOsCmd=[""]#### Script Action## Import pysftp only if vSendToSftp set to yes.ifvSendToSftp=="yes":importpysftp## Import subprocess only if vPreBckCmd or vPostBckCmd set to yes.ifvPreBckCmd=="yes"orvPostBckCmd=="yes":importsubprocess## Get current datedeffuncDateString():# Returns the today string year, month, day.returndatetime.now().strftime("%Y%m%d")## Define function for Pre OS commands.deffuncExecutePreOsCmd(vPreOsCmd):try:ifvPreBckCmd.casefold()=="yes":# iterate through each specified command.forvExecuteinvPreOsCmd:subprocess.run(vExecute,shell=True,check=True)# Send info to console.print("OS commands has been executed...")except:# Send info to console.print("Could not execute OS command...")## Define function for Pre OS commands.deffuncExecutePostOsCmd(vPostOsCmd):try:ifvPostBckCmd.casefold()=="yes":# iterate through each specified command.forvExecuteinvPostOsCmd:subprocess.run(vExecute,shell=True,check=True)# Send info to console.print("OS commands has been executed...")except:# Send info to console.print("Could not execute OS command...")## Build Zip filename with path.vSetZipFile=os.path.join(vBckDir,vFilePrefix+"_"+funcDateString()+".zip")## Define function for compressing files/folders into a zip file.deffuncCreateZipFile(vZipName,vPath):try:# Send info to console.print("Creating Zip file...")# Parameters: vZipName - name of the zip file; path - name of folder/file to be put in zip file.vZipFile=zipfile.ZipFile(vZipName,'w',zipfile.ZIP_DEFLATED)# iterate through each specified folder.forvFolderinvPath:# Changes root dir to given input folder to make zipped files relative to that.os.chdir(vFolder)# Send each folder and file to zip file.forroot,dirs,filesinos.walk(vFolder,topdown=False):fornameinfiles:vZipFile.write(os.path.join(root,name))# Close the Zip file.vZipFile.close()# Send info to console.print("Zip file has been created...")except:# Send info to console.print("Could not create Zip file...")## Define function send to SFTP.deffuncSendToSftp(vHost,vPort,vUser,vPwd,vFolder,vFile):try:ifvSendToSftp.casefold()=="yes":# Send info to console.print("Sending Zip file to SFTP server...")# Convert port to integer, needed since we use "" above to keep it a bit more tidy.vIntPort=int(vPort)# Connect to SFTP server and upload file.withpysftp.Connection(host=vHost,port=vIntPort,username=vUser,password=vPwd)assftp:# Change directory.withsftp.cd(vFolder):# Upload filesftp.put(vFile)# Send info to console.print("Done sending Zip file to SFTP server...")except:# Send info to console.print("Could not connect to server or upload file...")## Define history function.deffuncKeepBackup(vGetDays,vGetDir):try:# Check if to keep a history or not.vIntDays=int(vGetDays)ifvKeepBackup.casefold()=="yes":# Send info to console.print("Pruning backup folder, keeping",vIntDays,"days...")# Set today as current day.vTimeNow=time()# Remove files based on days to keep.forfnameinos.listdir(vGetDir):iffname.startswith(vFilePrefix):ifos.path.getmtime(os.path.join(vGetDir,fname))<vTimeNow-vIntDays*86400:os.remove(os.path.join(vGetDir,fname))# Send info to console.print("Done pruning backup folder...")elifvKeepBackup.casefold()=="no":# Send info to console.print("Removing all local backup files...")# Build file list and remove files.vSetFilePattern=os.path.join(vFilePrefix+"_")forfnameinos.listdir(vGetDir):iffname.startswith(vSetFilePattern):os.remove(os.path.join(vGetDir,fname))# Send info to console.print("Done removing all local backup files...")except:# Send info to console.print("Could not clean backup folder...")## Call the pre OS command function and run only if vPreBckCmd is set to yes.funcExecutePreOsCmd(vPreOsCmd)## Call the backup function and create the backup.funcCreateZipFile(vSetZipFile,vSrcDir)## Call the Sftp function and upload file only if vSendToSftp is set to yes.funcSendToSftp(vSftpHost,vSftpPort,vSftpUser,vSftpPass,vSftpDir,vSetZipFile)## Call the post OS command function and run only if vPostBckCmd is set to yes.funcExecutePostOsCmd(vPostOsCmd)## Call the history function to enable automatic housekeeping in the backup folder.funcKeepBackup(vKeepDays,vBckDir)
Changelog
ChangeLog - Expand to read
Version Date Information
------- ---- -----------
1.0 2023-01-25 Initial release.
1.4 2023-05-30 Revised and added script PreCMD en PostCMD options