~~ODT~~ ====== Prinzipien ====== * Generelle Punkte unabhängig von der eingesetzten Lösung * Diese Fragen sind essentiell für eine funktionierende Backup/Restore Lösung * Hast du eine Backup Lösung ? * Wurde definiert welche Daten das Backup beinhalten soll ? * Werden bestimmte Zustände deiner Daten zu bestimmten Zeitpunkten festgehalten ? * Wurde definiert wie lange bestimmte Zustände für ein Restore verfügbar sein müssen ? * Gibt es gesetzliche Verpflichtungen für die Dauer der Aufbewahrung und Qualität der Backups ? * Wo liegen die Backup Daten physikalisch ? * Wie wird der Raum in dem sich die Daten physikalisch befinden gesichert ? * Sind die Daten gegen Diebstahl / Blitzschlag / Überschwemmungen gesichert ? * Was passiert bei Ereignissen **"höherer Gewalt"** ? * Gibt es zumindest einen 2. Standort deiner Backup Daten ? * Ist es möglich die Integrität des Backups zu überprüfen ? * Wurde schon einmal ein Restore durchgeführt ? * Kannst du selbstständig einen Restore durchführen ? * Wie lange hat der Restore gedauert ? * Welche Softwarelösungen zB: Betriebssystem / Basis werden für den Restore benötigt ? * Welche Lizenzen werden benötigt um einen Restore durchzuführen ? ====== Windows ====== * Windows Sicherung muss installiert sein * Generiert VHDX Datein die gemountet werden können ===== Externer SAMBA Server rotierend ===== * **backup-generic.bat** * Bare Metal Backup falls die Maschine selbst etwas hat * **Achtung** unter BACKUP_TARGET müssen sich dann wie in diesem Fall die Verzeichnisse 0,1 befinden (wegen Module Operation der Anzahl der Wochen die gespeichert werden sollen) setlocal set LOGFILE=F:\Backup-Logging.txt set G=%temp%\getWeek.vbs set WEEKS=2 set USERNAME=USERNAME_SAMBA set PASSWORD=PASSWORD_SAMBA set BACKUP_TARGET=\\backup\backup_serverxy\bare_metal >%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2) for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i set /a DIRECTORY=%calweek%%%WEEKS% date /t >> %LOGFILE% wbadmin start backup -quiet -vssCopy -user:"%USERNAME%" -password:"%PASSWORD%" -allCritical -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1 ===== "Externes" Laufwerk rotierend ===== * **backup-storagexy.bat** * **Achtung** Hier wird das Laufwerk G:\ auf F:\ gesichert, wieder müssen die Verzeichnisse 0,1 unter BACKUP_TARGET existieren setlocal set LOGFILE=F:\Backup-Logging-StorageXY.txt set G=%temp%\getWeek.vbs set WEEKS=2 set BACKUP_TARGET=\\localhost\F$\Backups\STORAGEXY\ >%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2) for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i set /a DIRECTORY=%calweek%%%WEEKS% echo "----BEGIN BACKUP----" >> %LOGFILE% 2>&1 date /t >> %LOGFILE% wbadmin start backup -vssCopy -quiet -include:G:\ -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1 echo "----END BACKUP----" >> %LOGFILE% ===== Externe Festplatten ===== * Verschiedene Bat Dateien mit Laufwerksangaben für verschiedene externe Festplatten * gelöscht werden muss **manuell** * zB: G:\Backups\RECHNERNAME\DATUM * **backup-extern-g.bat** @echo off & setlocal set TARGET=G$ set NAME=%COMPUTERNAME% for /f %%i in ('date /T') do Set DIRECTORY=%%i if not exist "\\localhost\%TARGET%\Backups" goto fehler mkdir \\localhost\%TARGET%\Backups\%NAME%\%DIRECTORY%\ wbadmin start backup -quiet -vssFull -allCritical -backupTarget:\\localhost\%TARGET%\Backups\%NAME%\%DIRECTORY%\ pause exit :fehler echo Konnte Festplatte %TARGET% nicht finden ggf. \\localhost\%TARGET%\Backups Verzeichnis anlegen pause exit ===== Hyper-V Guests auf Festplatte rotierend ===== * **backup-hyper-v.bat** setlocal set LOGFILE=PFAD\Backup-Logging-Guests.txt set G=%temp%\getWeek.vbs set WEEKS=4 set BACKUP_TARGET=\\localhost\F$\Guests\ >%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2) for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i set /a DIRECTORY=%calweek%%%WEEKS% echo "----BEGIN BACKUP----" >> %LOGFILE% 2>&1 date /t >> %LOGFILE% wbadmin start backup -quiet -hyperv:"GUESTNAME1,GUESTNAME2,GUESTNAME3" -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1 echo "----END BACKUP----" >> %LOGFILE% ===== Restore mit Windows Boardmitteln ===== * Über zB: Windows 10 iso / reparieren / System Image wiederherstellen / Im Netzwerk suchen * Windows kann grundsätzlich vhdx Dateien als virtuelle Datenträger einbinden zB: über die "Computerverwaltung" -> "Datenträger" -> rechts anfügen vhd(x) - auch UNC Pfade auf Samba Server sind möglich (getestet mit Debian 10 / Windows 10 FIXME * **Achtung** * BIOS/UEFI beachten * Ziel Datenträger muss mindestens gleich groß /größer sein (think thin Provisioning für restore) * Format für die Eingabe von Benutzername: IP\BENUTZERNAME (wenn Daten auf samba Share liegen) ===== Duplicati 2 - verschlüsseltes Cloud Backup ===== * **Anfoderungen** * Deduplizierendes, verschlüsseltes Backup , das in der "Cloud" abgelegt werden kann in diesem Fall ins One-Drive Business von einem User Account der als "Backup" User angelegt wurde * **Lösungsmöglichkeit** * Duplicati 2: https://www.duplicati.com/download * **Achtung** **BETA** * One-Drive Einstellungen * **Achtung v2 auswählen und mit Graph API Key erstellen** - bei >5000 Dateien in einem Ordner kommt es sonst zu "Limits" die schwer zu debuggen sind * Getestet auf **Windows 2016 Standard** und **Duplicati 2.0.4.23 - 64bit ** , **Backup enthält > 1 Million Dateien** {{:know-how:duplicati_one_drive_v2.png?400|}} {{:know-how:duplicati_overview_backup_entry.png?400|}} {{:know-how:duplicati_success_restore.png?400|}} * Wiederherstellung von einzelnen Dateien hat beim Testen funktioniert war jedoch **sehr** langsam ===== Urbackup ===== * https://www.urbackup.org/download.html * Fehler Referenzmaschine war "Hyper-V" Client und wurde auf physikalische Maschine migriert - Fehlermeldung am Client ** "Hyper-V IC Software Shadow Copy Provider ist noch installiert ** 2019-11-06 06:31:52: ERROR: Not found 2019-11-06 06:31:56: ERROR: backupcom->AddToSnapshotSet(&(Server->ConvertToWchar(selected_vols[i])[0]), GUID_NULL, &additional_refs[i].volid) failed. VSS error code VSS_E_UNEXPECTED_PROVIDER_ERROR 2019-11-06 06:31:56: ERROR: VSS provider information: 2019-11-06 06:31:56: ERROR: Anbietername: "Hyper-V IC Software Shadow Copy Provider" 2019-11-06 06:31:56: ERROR: Anbietertyp: Software 2019-11-06 06:31:56: ERROR: Anbieterkennung: {74600e39-7dc5-4567-a03b-f091d6c7b092} 2019-11-06 06:31:56: ERROR: Version: 1.0.0.0 2019-11-06 06:31:56: ERROR: Anbietername: "Microsoft Software Shadow Copy provider 1.0" 2019-11-06 06:31:56: ERROR: Anbietertyp: System 2019-11-06 06:31:56: ERROR: Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5} 2019-11-06 06:31:56: ERROR: Version: 1.0.0.7 2019-11-06 06:31:56: ERROR: Creating shadowcopy of "C:" failed. 2019-11-06 06:31:56: ERROR: Creating shadow copy failed. See client log file for details. 2019-11-06 07:12:01: ERROR: Not found 2019-11-06 07:12:03: ERROR: backupcom->AddToSnapshotSet(&(Server->ConvertToWchar(selected_vols[i])[0]), GUID_NULL, &additional_refs[i].volid) failed. VSS error code VSS_E_UNEXPECTED_PROVIDER_ERROR 2019-11-06 07:12:03: ERROR: VSS provider information: 2019-11-06 07:12:03: ERROR: Anbietername: "Hyper-V IC Software Shadow Copy Provider" 2019-11-06 07:12:03: ERROR: Anbietertyp: Software 2019-11-06 07:12:03: ERROR: Anbieterkennung: {74600e39-7dc5-4567-a03b-f091d6c7b092} 2019-11-06 07:12:03: ERROR: Version: 1.0.0.0 2019-11-06 07:12:03: ERROR: Anbietername: "Microsoft Software Shadow Copy provider 1.0" 2019-11-06 07:12:03: ERROR: Anbietertyp: System 2019-11-06 07:12:03: ERROR: Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5} 2019-11-06 07:12:03: ERROR: Version: 1.0.0.7 2019-11-06 07:12:03: ERROR: Creating shadowcopy of "C:" failed. 2019-11-06 07:12:03: ERROR: Creating shadow copy failed. See client log file for details. * **Lösung**: Provider löschen (https://social.technet.microsoft.com/Forums/ie/en-US/ba9e020e-bbed-446a-be37-40cb00cd908d/manually-uninstall-integration-services-without-hyperv?forum=winserverhyperv ) [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{74600e39-7dc5-4567-a03b-f091d6c7b092}] "vssadmin list providers" sollte nach einem Reboot den Hyper V Provider nicht mehr anzeigen * Bei der ISO Datei für den **Restore wird Kernel 4.19 (Debian)** verwendet * Eintrag für PXE BIOS Boot über TFTP (https://urbackup.atlassian.net/wiki/spaces/US/pages/1441795/Restoring+Images+from+PXE) - dann könnte auch anderer Kernel verwendet werden hinsichtlich **Treiberunterstützung** LABEL UrBackup Restore menu label Urbackup Restore Backup kernel urbackup/live/vmlinuz append initrd=urbackup/live/initrd.img boot=live config username=urbackup toram noswap fetch=tftp://IP_ADRESSE_TFTP_SERVER/urbackup/live/filesystem.squashfs * **Migration** von Urbackup Server Installation **ohne Backupdaten** jedoch mit Einstellungen von altem System (2016 Standard) auf neues System (2019 Standard) * Installation gleiche Serverversion wie auf "altem System" * Deaktivieren und beenden von Urbackup Server Dienst auf altem und neuem System * Kopieren von C:\Programme..\urbackup Server\*.* von altem System auf neues System (damit auch die Keys für die Authentifizierung bleiben, alle Dateien überschreiben auf neuem System) * Verzeichnis / Laufwerk für Backup wie auf altem System auf neuem System erstellen * Urbackup Server Dienst wieder starten -> sobald die Clients im Web Menü sichtbar werden - vollständige Abbildsicherung auslösen / IP vom alten System muss **nicht** übernommen werden ===== Altaro ===== * Altaro 8 - **nw.js** Screen / Windows 2019 Standard (https://help.altaro.com/support/solutions/articles/43000467363-getting-nw-js-on-startup-here-s-how-to-fix-it-) * **Ausloggen** und neu einloggen / beim Check von **%Temp%** existiert das Verzeichnis nicht mehr * Bei den Einstellungen VSS Copy beachten bei virtuellen Domain Controllern ! * Scheduled Test Restores einrichten und die Daten des Backups regelmäßig überprüfen lassen * Notifications per E-Mail konfigurieren / für den Fall dass sie nicht funktioniert haben ====== HP Aruba ====== * Für Cronjob siehe "HP Pro Curve" * SSH Server Keys müssen bereits "accepted" worden sein / sonst hängt er bei der Accept Key Frage * **backup-aruba.expect** #!/usr/bin/expect -f set timeout 60 log_user 0 spawn ssh USERNAME@[lindex $argv 0] expect "password:" send "PASSWORD\r" #2021-02-08 cc: think wisely if you want to send password as argument to call #send "[lindex $argv 1]\r" expect "continue" send "\r" expect "#" send "terminal length 1000\r" expect "#" log_user 1 send "show config\r" expect "#" send "exit\r" expect ">" send "exit\r" expect "?" send "y\r" expect eof ====== HPE Comware ====== * Für Cronjob siehe "HP Pro Curve" * SSH Server Keys müssen bereits "accepted" worden sein / sonst hängt er bei der Accept Key Frage * **backup-hp-comware-switches.expect** #!/usr/bin/expect -f set timeout 60 spawn ssh admin@[lindex $argv 0] expect "password:" send "PASSWORD\r" #2021-02-08 cc: think wisely if you want to send password as argument to call #send "[lindex $argv 1]\r" expect ">" send "xtd-cli-mode\r" expect "]:" send "Y\r" expect "Password:" send "foes-bent-pile-atom-ship\r" expect ">" send "screen-length disable\r" expect ">" send "show current-configuration\r" expect ">" send "quit\r" ====== HP Pro Curve ====== * **Achtung** Sicherheitsimplikationen bei **telnet** beachten (Übertragung des passworts über plaintext) * **/etc/cron.d/backup-switches** # Every day @02:00 a.m. SHELL=/bin/bash 00 2 * * * root switches="IP_SWITCH_1 IP_SWITCH_2 IP_SWITCH_3 IP_SWITCH_4"; for i in $switches ; do /usr/local/bin/get-switch-config $i > /var/backups/switches-configs/$i.conf ; done * Dependency: expect * **/usr/local/bin/get-switch-config** #!/usr/bin/expect -f # To avoid empty lines, 'nonewline' flag is used set ip_address [lindex $argv 0]; set prompt "#" set username "USERNAME\r" set password "PASSWORD\r" log_user 0 spawn telnet $ip_address expect "Username:" send $username expect "Password:" send $password expect $prompt send "terminal length 1000\r" expect $prompt set timeout 120 log_user 1 send "show config\r" expect $prompt set timeout 20 send "exit\r" expect ">" send "exit\r" expect "?" send "y\r" expect eof ====== Linux ====== ===== rsnapshot ===== * Erstellung von Prüfsummen zu den Dateien die gebackuped wurden * rsnapshot.conf - Beispiele für die Verwendung von **integ.sh** ... cmd_postexec /usr/local/sbin/integ.sh /mnt/external-backup/ebackup1/weekly.0 c ; /bin/mount -o remount,ro /mnt/external-backup ... * rsnapshot.conf - Beispiele für die Verwendung von **backup-lvm-drive.sh** ... backup_script /usr/local/sbin/backup-lvm-drive.sh /dev/vgroup/lvmname imagename.img.lzo ./pfad/ ... * **backup-lvm-drive.sh** * Kann als **backup_script** ausgeführt werden zB: wenn auf einem Hostsystem virtualisiert über kvm eine "docker" blackbox läuft und lvm #!/bin/bash export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin function bailout () { echo -e "$1" exit 2 } function usage () { echo -e "$1" exit 1 } # LVM PATH / generate LVM snapshot name suffix snapshot / compress devices / output PATH ARGUMENTS=$# SIZE_SNAPSHOT="20G" BUFFER="10M" which lvcreate > /dev/null || bailout "lvcreate not found" which lvremove > /dev/null || bailout "lvremove not found" which lzop > /dev/null || bailout "lzop not found - apt-get install lzop" [ $ARGUMENTS != 2 ] && bailout "Usage: $0 PATH_TO_LVM_VOLUME BACKUP_FILE" LVM_PATH="$1" BACKUP_FILE="$2" [ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read PATH_TO_LVM VOLUME" [ -r $BACKUP_FILE ] && bailout "FAIL: $BACKUP_FILE is already here aborting" LVM_NAME=$(echo ${LVM_PATH##*/}) LVM_NAME_SNAPSHOT="$LVM_NAME""_snapshot" LVM_PATH_SNAPSHOT="${LVM_PATH%%$LVM_NAME}$LVM_NAME_SNAPSHOT" [ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Snapshot: $LVM_PATH_SNAPSHOT already here aborting" lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_NAME_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT" if [ $BACKUP_FILE == '-' ] then dd if=$LVM_PATH_SNAPSHOT bs=$BUFFER 2>/dev/null | lzop - else dd if=$LVM_PATH_SNAPSHOT bs=$BUFFER 2>/dev/null | lzop > $BACKUP_FILE fi lvremove -q -f $LVM_PATH_SNAPSHOT > /dev/null exit 0 * **backup-lvm-files.sh** * Um einen LVM Snapshot read only zu mounten und dann per RSYNC mit einem Verzeichnis zu syncen / kann mit rsnapshot backup_exec kombiniert werden #!/bin/bash function bailout () { echo -e "$1" [ -z "$2" ] || lvremove -q -f "$2" exit 2 } function usage () { echo -e "$1" exit 1 } RSYNC_DEFAULT_OPTIONS="--dry-run -a --delete --numeric-ids " RSYNC_DEFAULT_OPTIONS_ARRAY=($RSYNC_DEFAULT_OPTIONS) ARGUMENTS=$# SIZE_SNAPSHOT="10G" BUFFER="10M" TMP_MOUNT_POINT="/mnt/tmp" which lvcreate > /dev/null || bailout "lvcreate not found" which lvremove > /dev/null || bailout "lvremove not found" which rsync > /dev/null || bailout "rsync not found" which kpartx > /dev/null || bailout "kpartx not found" which mount > /dev/null || bailout "mount not found" which mountpoint > /dev/null || bailout "mountpoint not found" if [ $ARGUMENTS != 2 ] && [ $ARGUMENTS != 3 ] then bailout "Usage: $0 PATH_TO_LVM_VOLUME BACKUP_DIRECTORY [ \""RSYNC_OPTIONS\"" ]" fi LVM_PATH="$1" BACKUP_DIRECTORY="$2" RSYNC_CUSTOM_OPTIONS="$3" [ -n "$3" ] && RSYNC_DEFAULT_OPTIONS_ARRAY+=($RSYNC_CUSTOM_OPTIONS) [ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read $LVM_PATH VOLUME" [ ! -d $BACKUP_DIRECTORY ] && bailout "FAIL: $BACKUP_DIRECTORY is already here aborting" [ ! -d $TMP_MOUNT_POINT ] && bailout "FAIL: $TMP_MOUNT_POINT directory not found" mountpoint -q $TMP_MOUNT_POINT && bailout "FAIL: $TMP_MOUNT_POINT is already mounted FAIL" LVM_NAME=$(echo ${LVM_PATH##*/}) LVM_NAME_SNAPSHOT="$LVM_NAME""_snapshot" LVM_PATH_SNAPSHOT="${LVM_PATH%%$LVM_NAME}$LVM_NAME_SNAPSHOT" [ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Snapshot: $LVM_PATH_SNAPSHOT already here aborting" lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_NAME_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT" [ $? == "0" ] || bailout "FAIL: $LVM_PATH_SNAPSHOT could not be created" MOUNT_PATH="$LVM_PATH_SNAPSHOT" DEVICE_MAPPER_ENTRY=$(kpartx -av $LVM_PATH_SNAPSHOT) RETURN_SUCCESS=$? [ $? != "0" ] && bailout "FAIL: Kpartx did not return 0 - error" "$LVM_PATH_SNAPSHOT" DEVICE_MAPPER_ENTRY=$(echo $DEVICE_MAPPER_ENTRY | cut -f 3 -d" ") [ -z $DEVICE_MAPPER_ENTRY ] || MOUNT_PATH="/dev/mapper/""$DEVICE_MAPPER_ENTRY" mount -o ro "$MOUNT_PATH" "$TMP_MOUNT_POINT" || bailout "FAIL: could not mount $MOUNT_PATH to $TMP_MOUNT_POINT" "$LVM_PATH_SNAPSHOT" rsync "${RSYNC_DEFAULT_OPTIONS_ARRAY[@]}" "$TMP_MOUNT_POINT/" "$BACKUP_DIRECTORY/" RETURN_RSYNC=$? umount $TMP_MOUNT_POINT kpartx -dv $LVM_PATH_SNAPSHOT > /dev/null lvremove -q -f $LVM_PATH_SNAPSHOT > /dev/null exit $RETURN_RSYNC * **integ.sh** * Um nach dem Backup Prüfsummen aller Dateien zu erstellen #!/bin/bash MACHINE="MEIN-SERVER-NAME" function usage { echo "Usage: $0 directory_root [c|v]" echo "c...create hashes" echo "v...veriy hashes" exit 0 } echo $MACHINE function verifyDirectoryHashes { echo "verify" cd $DIRECTORY_ROOT [[ ! -f hashes.sha1 ]] && echo "Hashes: $DIRECTORY_ROOT/hashes.sha1 not found" && exit 2 date1=$(date -u +"%s") sha1sum --quiet -c hashes.sha1 retCode=$? date2=$(date -u +"%s") diff=$(($date2-$date1)) amount=$(wc -l hashes.sha1 | cut -d " " -f 1) echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed." echo "Hashes verified: $amount" echo "done" exit $retCode } function createDirectoryHashes { echo "create hashes" cd $DIRECTORY_ROOT echo -n > hashes.sha1 date1=$(date -u +"%s") find ./ ! -name "*.sha1" -type f -exec sha1sum {} >> hashes.sha1 \; date2=$(date -u +"%s") diff=$(($date2-$date1)) amount=$(wc -l hashes.sha1 | cut -d " " -f 1) echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed." echo "Hashes created: $amount" echo "done" exit 0 } #Check parameters DIRECTORY_ROOT="$1" ACTION="$2" [[ ! -d $DIRECTORY_ROOT ]] && echo "Cannot read directory: $DIRECTORY_ROOT" && usage [[ $ACTION != "c" && $ACTION != "v" ]] && echo "Either verify or create" && usage [[ $ACTION == "c" ]] && createDirectoryHashes [[ $ACTION == "v" ]] && verifyDirectoryHashes * rsnapshot mit **btrfs** rsnapshot.conf ... cmd_cp /usr/local/sbin/rsnapshot_btrfs_cp.sh # uncomment this to use the rm program instead of the built-in perl routine. # cmd_rm /usr/local/sbin/rsnapshot_btrfs_rm.sh ... * **rsnapshot_btrfs_cp.sh** #!/bin/bash # Arg 1: -al # Arg 2: /path/daily.0 # Arg 3: /path/daily.1 btrfs subvolume snapshot -r $2 $3 >/dev/null * **rsnapshot_btrfs_rm.sh** #!/bin/bash # Arg 1: -rf # Arg 2: /path/daily.5/ # echo 1: $1 2: $@ # Try to delete the given path with btrfs subvolume delete first # if this fails fall back to normal rm if [ "$1" = "-rf" -a "$3" = "" ]; then # "trying to delete with btrfs" btrfs subvolume show "$2" &>/dev/null && btrfs subvolume delete "$2" &>/dev/null && exit 0 [[ -d "$2" ]] && rm -rf "$2" && exit 0 exit 2 fi exit 2 * rsnapshot mit **borg** **cron job** * In der Konfiguration der rsnapshot Backups existiert nur mehr daily.0 d.h. **retain daily 1** und beide Backup Jobs (customers und pannoniait) müssen bereits **abgeschlossen** sein 30 06 * * 1-5 root ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::d.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P d -d 5 /mnt/storage/backups/borg ) || echo "Could not perform daily borg backup" 30 06 * * 6 root ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::w.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P w -w 4 /mnt/storage/backups/borg ) || echo "Could not perform weekly borg backup" 30 07 1 * * root ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::m.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P m -m 2 /mnt/stroage/backups/borg ) || echo "Could not perform monthly borg backup ===== borg - generell ===== * Backup Tests mit borg - Achtung **unverschlüsselt** da Testgerät ohnehin **vollverschlüsselt** ist * Ganze LVM Devices basierend auf Snapshot backupppen - sollte konsistent sein **backup-lvm-drive-borg.sh** #!/bin/bash function bailout () { echo -e "$1" 1>&2 exit 2 } function usage () { echo -e "$1" exit 1 } # LVM PATH / generate LVM snapshot name suffix snapshot / compress devices / output PATH ARGUMENTS=$# SIZE_SNAPSHOT="50G" BORG_NR_TO_KEEP="4" which lvcreate > /dev/null || bailout "lvcreate not found" which borg > /dev/null || bailout "borg cannot be found" which lvremove > /dev/null || bailout "lvremove not found" [ $ARGUMENTS -lt 2 ] && bailout "Usage: $0 PATH_TO_BORG_REPOSITORY PATH_TO_LVM_VOLUME1 PATH_TO_LVM_VOLUME2 " BACKUP_REPOSITORY=$(echo $1 | cut -d: -f 1) [ -r $BACKUP_REPOSITORY ] || bailout "FAIL: $BACKUP_REPOSITORY cannot be read" [ -w $BACKUP_REPOSITORY ] || bailout "FAIL: $BACKUP_REPOSITORY cannot be written to" for i in "${@:2}" ; do LVM_PATH="$i" LVM_PATH_SNAPSHOT="$LVM_PATH""_snapshot" [ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read \"$LVM_PATH\" VOLUME" [ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Attention snapshot still there: $LVM_PATH_SNAPSHOT" done lvm_snapshots="" for i in "${@:2}" ; do LVM_PATH="$i" LVM_PATH_SNAPSHOT="$LVM_PATH""_snapshot" lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_PATH_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT" lvm_snapshots="$lvm_snapshots $LVM_PATH_SNAPSHOT" done borg create --stats --compression=lz4 --read-special "$1" $lvm_snapshots RET_CODE_BORG="$?" for i in "${@:2}" ; do LVM_PATH="$i" LVM_PATH_SNAPSHOT="$LVM_PATH""_snapshot" lvremove -q -f $LVM_PATH_SNAPSHOT >/dev/null done borg prune --keep-last $BORG_NR_TO_KEEP $BACKUP_REPOSITORY exit $RET_CODE_BORG * Durchgeführt auf Kali Linux - Debian Testing Borg Befehle: 1) Repository initialisieren d.h. Verzeichnis initialisieren wo das Backup landen soll e.g. Encryption modes Hash/MAC Not encrypted no auth Not encrypted, but authenticated Encrypted (AEAD w/ AES) and authenticated SHA-256 none authenticated repokey keyfile BLAKE2b n/a authenticated-blake2 repokey-blake2 keyfile-blake2 ( encryption -> 'none', 'keyfile', 'repokey', 'authenticated', 'keyfile-blake2', 'repokey-blake2', 'authenticated-blake2' ) root@mrChief:/home# borg init --encryption none /mnt/backup/mrChief/ root@mrChief:/home# echo $? 0 2) Backup erstellen Exclude mir alle ISO Dateien unter /home/urnilxfgbez/ Nicht über das Dateisystem rausschießen deshalb / und /boot einzeln Nummerische Ids der Ownership speichern borg create --stats --progress --one-file-system --numeric-owner --exclude /swap.img --exclude '/home/*/*.iso' --exclude '/home/*/*.ISO' --compression lz4 /mnt/backup/mrChief/::$(date +"%T.%F") / /boot 3) Verfügbare Backups anzeigen (borg list) root@mrChief:/home# borg list /mnt/backup/mrChief/ 12:13:01.2019-10-16 Wed, 2019-10-16 12:13:02 [6e550028349bdde6f22ab513a83169ebf6ef87026af80e5598f7e9c82dff4229] 4) Verfügbares Backup mounten und zugänglich machen root@mrChief:/mnt# borg mount /mnt/backup/mrChief::12:13:01.2019-10-16 /mnt/tmp root@mrChief:/mnt# borg umount /mnt/tmp root@mrChief:/mnt# ls -al /mnt/tmp total 8 drwxr-xr-x 2 root root 4096 Jan 23 2019 . drwxr-xr-x 14 root root 4096 Oct 16 09:47 .. 5) Backup Policy definieren f. Aufbewahrungsfristen (borg prune) Letzte 4 Versionen behalten root@mrChief:/mnt# borg prune --keep-last 4 /mnt/backup/mrChief/ ===== borg - hetzner storage ===== 1. key verteilen https://wiki.hetzner.de/index.php/Backup_Space_SSH_Keys root@mrStorage:~# echo -e "mkdir .ssh \n chmod 700 .ssh \n put /root/.ssh/id_rsa.pub .ssh/authorized_keys \n chmod 600 .ssh/authorized_keys" | sftp u12345678@u12345678.your-storagebox.de The authenticity of host 'u12345678.your-storagebox.de (176.9.161.59)' can't be established. RSA key fingerprint is SHA256:EMlfI8GsRIfpVkoW1H2u0zYVpFGKkIMKHFZIRkf2ioI. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'u12345678.your-storagebox.de,176.9.161.59' (RSA) to the list of known hosts. u12345678@u12345678.your-storagebox.de's password: Connected to u12345678@u12345678.your-storagebox.de. sftp> mkdir .ssh sftp> chmod 700 .ssh Changing mode on /.ssh sftp> put /root/.ssh/id_rsa.pub .ssh/authorized_keys Uploading /root/.ssh/id_rsa.pub to /.ssh/authorized_keys /root/.ssh/id_rsa.pub 100% 740 27.0KB/s 00:00 sftp> chmod 600 .ssh/authorized_keys Changing mode on /.ssh/authorized_keys 2. Repo initialisieren borg init --encryption=keyfile ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ Keymaterial befindet sich ausschließlich auf "mrStorage" dem Server der das Backup in die Hetzner CLoud durchführt 3. Backup Keys exportieren - es wird Keyfile und Passphrase benötigt - backup des keyfiles borg key export ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ /tmp/borg_keyfile 3. Common Commands root@mrStorage:~# borg create --compression lz4 --list --dry-run ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz - /root/fw.log.2.gz root@mrStorage:~# borg create --compression lz4 --list ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: A /root/fw.log.2.gz root@mrStorage:~# borg create --compression lz4 --list --dry-run ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz - /root/fw.log.2.gz root@mrStorage:~# borg info ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: Location: ssh://u12345678@u12345678.your-storagebox.de:23/./borg Encrypted: Yes (key file) Key file: /root/.config/borg/keys/u12345678_your_storagebox_de__borg ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size All archives: 4.48 MB 4.50 MB 4.50 MB Unique chunks Total chunks Chunk index: 5 5 root@mrStorage:~# borg list ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 2020-03-05T23:35:02 Thu, 2020-03-05 23:35:11 [0fe96fe6a40b0eb5db5955affad70a570e3a37cb3d3dd9f6e9a3ba0a538eafc2] root@mrStorage:~# borg mount ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ /mnt/tmp Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: * **/usr/local/bin/hetzner_borg_backup.sh** #!/bin/bash export BORG_PASSPHRASE="PASSPHRASE" REPOSITORY="ssh://u12345678@u12345678.your-storagebox.de:23/./borg/" DIRECTORIES="/mnt/storage/" HOST=$(hostname --fqdn) START_TIME=$(date +%Y.%m.%d-%H.%M.%S) date1=$(date -u +"%s") echo -e "Program: $0 \nstarted at: $START_TIME\nBacking Up: $DIRECTORIES\nHost: $HOST" #2020-04-08 cc: Achtung sonst konsumiert er den gesamten Upload und die Leitung steht daher: Rate limit ~6 Mbit upload / bei 10Mbit verfügbarem Upload borg create --compression lz4 --stats --remote-ratelimit 700 --exclude '*/.snapshots/*' $REPOSITORY::{now} $DIRECTORIES [[ $? == "0" ]] && borg prune --stats --keep-last 6 $REPOSITORY END_TIME=$(date +%Y.%m.%d-%H.%M.%S) date2=$(date -u +"%s") diff=$(($date2-$date1)) echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed." echo "$(($diff / 86400 )) days elapsed. " echo -e "Program ended successfully : $0 \nended at: $END_TIME\n" ===== triggered ===== * Anforderungen: * Es sollen Backups durchgeführt werden "Event" basierend d.h. wenn eine bestimmte Festplatte angesteckt wird * Getestet auf Debian Buster * Die Seagate Platte wird sporadisch angesteckt (siehe lsusb - ID) : root@mrGodfather:/mnt/ebackup/ebackup1# lsusb .... .... Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 005 Device 008: ID 0bc2:61b7 Seagate RSS LLC Maxtor M3 Portable ..... * Udev Trigger für diese spezielle Festplatte root@mrGodfather:/mnt/ebackup/ebackup1# cat /etc/udev/rules.d/30-ebackup.rules ACTION=="add",KERNEL=="sd?[0-9]",ATTRS{idVendor}=="0bc2", ATTRS{idProduct}=="61b7", RUN+="/usr/local/sbin/backup_wrapper_systemd_ebackup1.sh" * Workaround damit "Backup" Skript systemd entzogen wird (dort gibts ein definiertes timeout für Skript) und at übergeben root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/backup_wrapper_systemd_ebackup1.sh #!/bin/bash echo "/usr/local/sbin/backup_ebackup1.sh" | at now exit 0 * Backup Skript , das crypsetup Device erstellt und Backups durchführt (Achtung btrfs Dateisystem) root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/backup_ebackup1.sh #!/bin/bash #1st step check if device is here #2nd step check if already mounted #3rd step mount using cryptsetup key #4th step remember time backup of all hosts using rsync #5th step take snapshot #6th sync data umount drive export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KEY_LOCATION="LOCATION_TO_KEYFILE" MAPPER_NAME="ebackup1" SUBVOLUME_NAME="ebackup1" MOUNT_POINT="/mnt/ebackup" HARDDISK_ID="/dev/disk/by-id/usb-Seagate_M3_Portable_NA844AER-0:0-part1" MAIL_NOTIFICATION="EMAIL_ADDRESS" RSYNC_STANDARD="rsync -a --timeout=300 --acls --delete --numeric-ids --exclude mnt/backup/ --exclude proc/ --exclude sys/ --exclude dev/ --exclude backup/ --exclude mnt/storage/backups/pannoniait/ --exclude mnt/storage/backups/customers/ --exclude .snapshots/ " MACHINES="server1 server2 server3 server4" BACKUP_ID=$(date +%s) START_TIME=$(date +%Y.%m.%d-%H.%M.%S) function bailout () { echo -e "$1" exit 2 } function notify () { echo -e "$1" } notify "Program: $0 \nstarted at: $START_TIME\nMachines to backup: $MACHINES mrGodfather\nDestination Hardware: $HARDDISK_ID\nBackup ID: $BACKUP_ID\nDestination MountPoint: $MOUNT_POINT\nKeyfile: $KEY_LOCATION" which cryptsetup > /dev/null || bailout "FAIL: cryptsetup not found!" which mail > /dev/null || ( echo "FAIL: mail not found!" && exit 2 ) which rsync > /dev/null || bailout "FAIL: rsync not found!" which btrfs > /dev/null || bailout "FAIL: btrfs not found!" [ ! -r $HARDDISK_ID ] && bailout "FAIL: Physical Harddisk: $HARDDISK_ID not found\nBackup ID: $BACKUP_ID" [ ! -r $KEY_LOCATION ] && bailout "FAIL: Key not found: $KEY_LOCATION\nBackup ID: $BACKUP_ID" [ ! -d $MOUNT_POINT ] && bailout "FAIL: Mount point not found: $MOUNT_POINT\nBackup ID: $BACKUP_ID" [ -r "/dev/mapper/$MAPPER_NAME" ] && bailout "FAIL: Crypto Mapper: $MAPPER_NAME found close manualy\nBackup ID: $BACKUP_ID" mountpoint -q $MOUNT_POINT && bailout "FAIL: Already mounted: $MOUNT_POINT\nBackup ID: $BACKUP_ID" cryptsetup luksOpen $HARDDISK_ID $MAPPER_NAME --key-file $KEY_LOCATION || bailout "FAIL: Could not open cryptsetup device $HARDDISK_ID with $KEY_LOCATION\nBackup ID: $BACKUP_ID" mount -o rw,acl,noexec /dev/mapper/$MAPPER_NAME $MOUNT_POINT || bailout "FAIL: could not mount: /dev/mapper/$MAPPER_NAME on $MOUNT_POINT\nBackup ID: $BACKUP_ID" [ ! -d "$MOUNT_POINT/$SUBVOLUME_NAME" ] && bailout "FAIL: Directory not found on Mountpoint: $MOUNT_POINT/$SUBVOLUME_NAME\nBackup ID: $BACKUP_ID" for machine in $MACHINES do $RSYNC_STANDARD root@$machine:/ $MOUNT_POINT/$SUBVOLUME_NAME/$machine/ done btrfs subvolume snapshot -r $MOUNT_POINT/$SUBVOLUME_NAME $MOUNT_POINT/$SUBVOLUME_NAME/.snapshots/@GMT_$START_TIME END_TIME=$(date +%Y.%m.%d-%H.%M.%S) btrfs filesystem sync $MOUNT_POINT || bailout "FAIL: could not sync btrfs filesystem\nBackup ID: $BACKUP_ID" /usr/local/sbin/clearLastSnapshot.sh $MOUNT_POINT ebackup1 5 || bailout "FAIL: could not clearLastSnapshot\nCheck Manualyy!\mBackup ID: $BACKUP_ID" sync sleep 60 umount $MOUNT_POINT || bailout "FAIL: Could not unmount filesystem\nBackup ID: $BACKUP_ID" cryptsetup luksClose $MAPPER_NAME || bailout "FAIL: Could not close cryptsetup Handle $MAPPER_NAME\nBackup ID: $BACKUP_ID" notify "Program ended successfully : $0 \nended at: $END_TIME\nBackup ID: $BACKUP_ID" exit 0 * Skript um nur eine bestimmte Anzahl an Snapshots auf dem Device zu halten root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/clearLastSnapshot.sh #!/bin/bash function usage { echo "Usage Keep this Nr of Snapshots: $0 LocalMountPoint LocalSubvolumeName DesiredSnapshotCount" echo "Usage Show Nr of Snapshots: $0 LocalMountPoint LocalSubvolumeName" echo "Usage: e.g. $0 /mnt/storage daten 3" exit 1 } LOCAL_MOUNT_POINT=$1 LOCAL_SUBVOLUME=$2 DESIRED_SNAPSHOTS=$3 [[ $# != 3 && $# != 2 ]] && usage [[ ! -d $LOCAL_MOUNT_POINT ]] && echo "Couldn't validate local btrfs subvolume mountpoint: $LOCAL_MOUNT_POINT" && exit 2 CURRENT_NR_SNAPSHOTS=$(btrfs subvolume list $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots -r -o --sort=+gen | wc -l ) [[ "$CURRENT_NR_SNAPSHOTS" == 0 ]] && echo "Couldn't aquire number of snapshots from $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots" && exit 2 [[ $# == 2 ]] && echo -e "Mount Point: $LOCAL_MOUNT_POINT\nSubvolume: $LOCAL_SUBVOLUME\nCurrent Snapshots: $CURRENT_NR_SNAPSHOTS" && exit 0 REGEX_NUMBER='^[0-9]+$' [[ ! $DESIRED_SNAPSHOTS =~ $REGEX_NUMBER ]] && echo "That's not a valid number: $NR_SNAPSHOTS" && exit 2 [[ $(($CURRENT_NR_SNAPSHOTS-$DESIRED_SNAPSHOTS)) -le 0 ]] && echo -e "Deletion not needed\nMount Point: $LOCAL_MOUNT_POINT\nSubvolume: $LOCAL_SUBVOLUME\nCurrent Snapshots: $CURRENT_NR_SNAPSHOTS\nDesired: $DESIRED_SNAPSHOTS" && exit 0 NR_SNAPSHOTS_REMOVE=$(($CURRENT_NR_SNAPSHOTS-$DESIRED_SNAPSHOTS)) CURRENT_SNAPSHOTS=$(btrfs subvolume list $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots -r -o --sort=+gen | head -n $NR_SNAPSHOTS_REMOVE | cut -d' ' -f 9 ) for snap in $CURRENT_SNAPSHOTS do btrfs subvolume delete $LOCAL_MOUNT_POINT/$snap done btrfs filesystem sync $LOCAL_MOUNT_POINT ===== etc ===== * rsync für blockdevices Kopieren "--copy-devices" * **Update** rsync (3.2.0-1) unstable; urgency=low This latest release changed two parameters which used to be present on the Debian packaging of rsync as upstream now integrated the patches. Previous parameter: --copy-devices: write to devices as files (implies --inplace) Is now called: --write-devices root@mrChief:/home/urnilxfgbez# rsync -v --progress --block-size=131072 --no-whole-file --checksum --copy-devices /dev/mmcblk0 /tmp/foo * Interessantes zu der Option und beim Kopieren von/auf lokalem Dateisystem: https://superuser.com/questions/234273/why-doest-rsync-use-delta-transfer-for-local-files * Er scheint zuerst die Hashes vom Ziel zu berechnen , danach von der Quelle (debugged mit iostat auf dem jeweiligen Device) root@mrChief:/home/urnilxfgbez# rsync -vv --no-whole-file --copy-devices /dev/mmcblk0 /tmp/foo delta-transmission enabled mmcblk0 total: matches=126225 hash_hits=126225 false_alarms=0 data=0 sent 505,008 bytes received 1,009,926 bytes 5,439.62 bytes/sec total size is 0 speedup is 0.00 root@mrChief:/home/urnilxfgbez# rsync -vv --no-whole-file --copy-devices root@localhost:/dev/mmcblk0 /tmp/foo opening connection using: ssh -l root localhost rsync --server --sender -vve.LsfxC --copy-devices . /dev/mmcblk0 (11 args) delta-transmission enabled mmcblk0 total: matches=126225 hash_hits=126225 false_alarms=0 data=0 sent 1,009,903 bytes received 505,075 bytes 5,439.78 bytes/sec * rsync - Ausgabe ob etwas geändert werden würde - dry run und count * **rsync -iaun --delete foo1/ foo2/ | wc -l** 0