Benutzer-Werkzeuge

Webseiten-Werkzeuge


know-how:backup

Dies ist eine alte Version des Dokuments!


Windows

  • Windows Sicherung muss installiert sein
  • Generiert VHDX Datein die gemountet werden können

Externer SAMBA Server rotierend

  • backup-generic.bat
  • Bare Metal Backup falls die Maschine selbst etwas hat
  • Achtung unter BACKUP_TARGET müssen sich dann wie in diesem Fall die Verzeichnisse 0,1 befinden (wegen Module Operation der Anzahl der Wochen die gespeichert werden sollen)
setlocal

set LOGFILE=F:\Backup-Logging.txt
set G=%temp%\getWeek.vbs
set WEEKS=2

set USERNAME=USERNAME_SAMBA
set PASSWORD=PASSWORD_SAMBA
set BACKUP_TARGET=\\backup\backup_serverxy\bare_metal

>%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2)

for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i

set /a DIRECTORY=%calweek%%%WEEKS%

date /t >> %LOGFILE%

wbadmin start backup -quiet -vssCopy -user:"%USERNAME%" -password:"%PASSWORD%" -allCritical -backupTarget:%BACKUP_TARGET%%DIRECTORY%  >> %LOGFILE% 2>&1

"Externes" Laufwerk rotierend

  • backup-storagexy.bat
  • Achtung Hier wird das Laufwerk G:\ auf F:\ gesichert, wieder müssen die Verzeichnisse 0,1 unter BACKUP_TARGET existieren
setlocal

set LOGFILE=F:\Backup-Logging-StorageXY.txt
set G=%temp%\getWeek.vbs
set WEEKS=2

set BACKUP_TARGET=\\localhost\F$\Backups\STORAGEXY\

>%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2)

for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i

set /a DIRECTORY=%calweek%%%WEEKS%


echo "----BEGIN BACKUP----" >> %LOGFILE% 2>&1


date /t >> %LOGFILE%

wbadmin start backup -vssCopy -quiet -include:G:\  -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1

echo "----END BACKUP----" >> %LOGFILE%

Externe Festplatten

  • Verschiedene Bat Dateien mit Laufwerksangaben für verschiedene externe Festplatten
  • gelöscht werden muss manuell
  • zB: G:\Backups\RECHNERNAME\DATUM
  • backup-extern-g.bat
@echo off & setlocal

set TARGET=G$
set NAME=%COMPUTERNAME%

for /f %%i in ('date /T') do Set DIRECTORY=%%i

if not exist "\\localhost\%TARGET%\Backups" goto fehler

mkdir \\localhost\%TARGET%\Backups\%NAME%\%DIRECTORY%\

wbadmin start backup -quiet -vssFull -allCritical -backupTarget:\\localhost\%TARGET%\Backups\%NAME%\%DIRECTORY%\

pause
exit
:fehler
echo Konnte Festplatte %TARGET% nicht finden ggf. \\localhost\%TARGET%\Backups Verzeichnis anlegen
pause
exit

Hyper-V Guests auf Festplatte rotierend

  • backup-hyper-v.bat
setlocal

set LOGFILE=PFAD\Backup-Logging-Guests.txt
set G=%temp%\getWeek.vbs
set WEEKS=4

set BACKUP_TARGET=\\localhost\F$\Guests\

>%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2)

for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i

set /a DIRECTORY=%calweek%%%WEEKS%


echo "----BEGIN BACKUP----" >> %LOGFILE% 2>&1


date /t >> %LOGFILE%

wbadmin start backup -quiet -hyperv:"GUESTNAME1,GUESTNAME2,GUESTNAME3" -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1

echo "----END BACKUP----" >> %LOGFILE%

Duplicati 2 - verschlüsseltes Cloud Backup

  • Anfoderungen
    • Deduplizierendes, verschlüsseltes Backup , das in der „Cloud“ abgelegt werden kann in diesem Fall ins One-Drive Business von einem User Account der als „Backup“ User angelegt wurde
  • One-Drive Einstellungen
  • Achtung v2 auswählen und mit Graph API Key erstellen - bei >5000 Dateien in einem Ordner kommt es sonst zu „Limits“ die schwer zu debuggen sind
  • Getestet auf Windows 2016 Standard und Duplicati 2.0.4.23 - 64bit , Backup enthält > 1 Million Dateien

  • Wiederherstellung von einzelnen Dateien hat beim Testen funktioniert war jedoch sehr langsam

Urbackup

  • Fehler Referenzmaschine war „Hyper-V“ Client und wurde auf physikalische Maschine migriert - Fehlermeldung am Client „Hyper-V IC Software Shadow Copy Provider ist noch installiert
2019-11-06 06:31:52: ERROR: Not found
2019-11-06 06:31:56: ERROR: backupcom->AddToSnapshotSet(&(Server->ConvertToWchar(selected_vols[i])[0]), GUID_NULL, &additional_refs[i].volid) failed. VSS error code VSS_E_UNEXPECTED_PROVIDER_ERROR
2019-11-06 06:31:56: ERROR: VSS provider information:
2019-11-06 06:31:56: ERROR: Anbietername: "Hyper-V IC Software Shadow Copy Provider"
2019-11-06 06:31:56: ERROR: Anbietertyp: Software
2019-11-06 06:31:56: ERROR: Anbieterkennung: {74600e39-7dc5-4567-a03b-f091d6c7b092}
2019-11-06 06:31:56: ERROR: Version: 1.0.0.0
2019-11-06 06:31:56: ERROR: Anbietername: "Microsoft Software Shadow Copy provider 1.0"
2019-11-06 06:31:56: ERROR: Anbietertyp: System
2019-11-06 06:31:56: ERROR: Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5}
2019-11-06 06:31:56: ERROR: Version: 1.0.0.7
2019-11-06 06:31:56: ERROR: Creating shadowcopy of "C:" failed.
2019-11-06 06:31:56: ERROR: Creating shadow copy failed. See client log file for details.
2019-11-06 07:12:01: ERROR: Not found
2019-11-06 07:12:03: ERROR: backupcom->AddToSnapshotSet(&(Server->ConvertToWchar(selected_vols[i])[0]), GUID_NULL, &additional_refs[i].volid) failed. VSS error code VSS_E_UNEXPECTED_PROVIDER_ERROR
2019-11-06 07:12:03: ERROR: VSS provider information:
2019-11-06 07:12:03: ERROR: Anbietername: "Hyper-V IC Software Shadow Copy Provider"
2019-11-06 07:12:03: ERROR: Anbietertyp: Software
2019-11-06 07:12:03: ERROR: Anbieterkennung: {74600e39-7dc5-4567-a03b-f091d6c7b092}
2019-11-06 07:12:03: ERROR: Version: 1.0.0.0
2019-11-06 07:12:03: ERROR: Anbietername: "Microsoft Software Shadow Copy provider 1.0"
2019-11-06 07:12:03: ERROR: Anbietertyp: System
2019-11-06 07:12:03: ERROR: Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5}
2019-11-06 07:12:03: ERROR: Version: 1.0.0.7
2019-11-06 07:12:03: ERROR: Creating shadowcopy of "C:" failed.
2019-11-06 07:12:03: ERROR: Creating shadow copy failed. See client log file for details.
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{74600e39-7dc5-4567-a03b-f091d6c7b092}] 

"vssadmin list providers" sollte nach einem Reboot den Hyper V Provider nicht mehr anzeigen
LABEL UrBackup Restore
menu label Urbackup Restore Backup
kernel urbackup/live/vmlinuz
append initrd=urbackup/live/initrd.img boot=live config username=urbackup toram noswap fetch=tftp://IP_ADRESSE_TFTP_SERVER/urbackup/live/filesystem.squashfs

Altaro

HP Aruba

  • Für Cronjob siehe „HP Pro Curve“
  • SSH Server Keys müssen bereits „accepted“ worden sein / sonst hängt er bei der Accept Key Frage
  • backup-aruba.expect
#!/usr/bin/expect -f

set timeout 60

log_user 0

spawn ssh USERNAME@[lindex $argv 0]


expect "password:"
send "PASSWORD\r"

#2021-02-08 cc: think wisely if you want to send password as argument to call
#send "[lindex $argv 1]\r" 

expect "continue"

send "\r"

expect "#"

send "terminal length 1000\r"

expect "#"

log_user 1

send "show config\r"

expect "#"


send "exit\r"

expect ">"

send "exit\r"

expect "?"

send "y\r"

expect eof

HPE Comware

  • Für Cronjob siehe „HP Pro Curve“
  • SSH Server Keys müssen bereits „accepted“ worden sein / sonst hängt er bei der Accept Key Frage
  • backup-hp-comware-switches.expect
#!/usr/bin/expect -f

set timeout 60

spawn ssh admin@[lindex $argv 0]

expect "password:"
send "PASSWORD\r"
#2021-02-08 cc: think wisely if you want to send password as argument to call
#send "[lindex $argv 1]\r" 

expect ">"

send "xtd-cli-mode\r" 

expect "]:"

send "Y\r"

expect "Password:"

send "foes-bent-pile-atom-ship\r" 

expect ">"

send "screen-length disable\r"

expect ">"

send "show current-configuration\r"

expect ">"

send "quit\r"

HP Pro Curve

  • Achtung Sicherheitsimplikationen bei telnet beachten (Übertragung des passworts über plaintext)
  • /etc/cron.d/backup-switches
# Every day @02:00 a.m.
SHELL=/bin/bash

00 2	* * *    root switches="IP_SWITCH_1 IP_SWITCH_2 IP_SWITCH_3 IP_SWITCH_4"; for i in $switches ;  do /usr/local/bin/get-switch-config $i > /var/backups/switches-configs/$i.conf ; done
  • Dependency: expect
  • /usr/local/bin/get-switch-config
#!/usr/bin/expect -f

# To avoid empty lines, 'nonewline' flag is used
set ip_address [lindex $argv 0];
set prompt "#"
set username "USERNAME\r"
set password "PASSWORD\r"

    log_user 0 
    spawn telnet $ip_address
    expect "Username:"
    send $username
    expect "Password:"
    send $password
    expect $prompt
    send "terminal length 1000\r"
    expect $prompt
    set timeout 120
    log_user 1
    send "show config\r"
    expect $prompt
    set timeout 20
    send "exit\r"
    expect ">"
    send "exit\r"
    expect "?"
    send "y\r"
    expect eof

Linux

rsnapshot

  • Erstellung von Prüfsummen zu den Dateien die gebackuped wurden
  • rsnapshot.conf - Beispiele für die Verwendung von integ.sh
...
cmd_postexec    /usr/local/sbin/integ.sh /mnt/external-backup/ebackup1/weekly.0 c ; /bin/mount -o remount,ro /mnt/external-backup
...
  • rsnapshot.conf - Beispiele für die Verwendung von backup-lvm-drive.sh
...
backup_script	/usr/local/sbin/backup-lvm-drive.sh /dev/vgroup/lvmname imagename.img.lzo	./pfad/
...
  • backup-lvm-drive.sh
  • Kann als backup_script ausgeführt werden zB: wenn auf einem Hostsystem virtualisiert über kvm eine „docker“ blackbox läuft und lvm
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


function bailout ()
{

	echo -e "$1"
	exit 2

}

function usage ()
{
	echo -e "$1"
	exit 1


}

# LVM PATH / generate LVM snapshot name suffix snapshot / compress devices / output PATH

ARGUMENTS=$#
SIZE_SNAPSHOT="20G"
BUFFER="10M"

which lvcreate > /dev/null || bailout "lvcreate not found"
which lvremove > /dev/null || bailout "lvremove not found"
which lzop > /dev/null || bailout "lzop not found - apt-get install lzop"

[ $ARGUMENTS != 2 ] && bailout "Usage: $0 PATH_TO_LVM_VOLUME BACKUP_FILE"

LVM_PATH="$1"
BACKUP_FILE="$2"

[ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read PATH_TO_LVM VOLUME"

[ -r $BACKUP_FILE ] && bailout "FAIL: $BACKUP_FILE is already here aborting"


LVM_NAME=$(echo ${LVM_PATH##*/})

LVM_NAME_SNAPSHOT="$LVM_NAME""_snapshot"

LVM_PATH_SNAPSHOT="${LVM_PATH%%$LVM_NAME}$LVM_NAME_SNAPSHOT"

[ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Snapshot: $LVM_PATH_SNAPSHOT already here aborting"

lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_NAME_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT"

if [ $BACKUP_FILE == '-' ]
then

	dd if=$LVM_PATH_SNAPSHOT bs=$BUFFER 2>/dev/null | lzop -
else

	dd if=$LVM_PATH_SNAPSHOT bs=$BUFFER 2>/dev/null | lzop  > $BACKUP_FILE
fi

lvremove -q -f $LVM_PATH_SNAPSHOT > /dev/null

exit 0
  • backup-lvm-files.sh
  • Um einen LVM Snapshot read only zu mounten und dann per RSYNC mit einem Verzeichnis zu syncen / kann mit rsnapshot backup_exec kombiniert werden
#!/bin/bash


function bailout ()
{

	echo -e "$1"

	[ -z "$2" ] || lvremove -q -f "$2"

	exit 2

}

function usage ()
{
	echo -e "$1"
	exit 1


}

RSYNC_DEFAULT_OPTIONS="--dry-run -a --delete  --numeric-ids " 

RSYNC_DEFAULT_OPTIONS_ARRAY=($RSYNC_DEFAULT_OPTIONS)

ARGUMENTS=$#
SIZE_SNAPSHOT="10G"
BUFFER="10M"

TMP_MOUNT_POINT="/mnt/tmp"

which lvcreate > /dev/null || bailout "lvcreate not found"
which lvremove > /dev/null || bailout "lvremove not found"
which rsync > /dev/null || bailout "rsync not found"
which kpartx > /dev/null || bailout "kpartx not found"
which mount > /dev/null || bailout "mount not found"
which mountpoint > /dev/null || bailout "mountpoint not found"


if [ $ARGUMENTS != 2 ] &&  [ $ARGUMENTS != 3 ]
then

	bailout "Usage: $0 PATH_TO_LVM_VOLUME BACKUP_DIRECTORY [ \""RSYNC_OPTIONS\"" ]"

fi

LVM_PATH="$1"
BACKUP_DIRECTORY="$2"
RSYNC_CUSTOM_OPTIONS="$3"

[ -n "$3" ] && RSYNC_DEFAULT_OPTIONS_ARRAY+=($RSYNC_CUSTOM_OPTIONS)


[ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read $LVM_PATH VOLUME"

[ ! -d $BACKUP_DIRECTORY ] && bailout "FAIL: $BACKUP_DIRECTORY is already here aborting"

[ ! -d $TMP_MOUNT_POINT ] && bailout "FAIL: $TMP_MOUNT_POINT directory not found"

mountpoint -q $TMP_MOUNT_POINT && bailout "FAIL: $TMP_MOUNT_POINT is already mounted FAIL"

LVM_NAME=$(echo ${LVM_PATH##*/})

LVM_NAME_SNAPSHOT="$LVM_NAME""_snapshot"

LVM_PATH_SNAPSHOT="${LVM_PATH%%$LVM_NAME}$LVM_NAME_SNAPSHOT"

[ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Snapshot: $LVM_PATH_SNAPSHOT already here aborting"

lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_NAME_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT"


[ $? == "0"  ] || bailout "FAIL: $LVM_PATH_SNAPSHOT could not be created"


MOUNT_PATH="$LVM_PATH_SNAPSHOT"

DEVICE_MAPPER_ENTRY=$(kpartx -av $LVM_PATH_SNAPSHOT)

RETURN_SUCCESS=$?

[ $? != "0" ] && bailout "FAIL: Kpartx did not return 0 - error" "$LVM_PATH_SNAPSHOT"

DEVICE_MAPPER_ENTRY=$(echo $DEVICE_MAPPER_ENTRY | cut -f 3 -d" ")

[ -z $DEVICE_MAPPER_ENTRY ] || MOUNT_PATH="/dev/mapper/""$DEVICE_MAPPER_ENTRY"


mount -o ro "$MOUNT_PATH" "$TMP_MOUNT_POINT" || bailout "FAIL: could not mount $MOUNT_PATH to $TMP_MOUNT_POINT" "$LVM_PATH_SNAPSHOT"


rsync "${RSYNC_DEFAULT_OPTIONS_ARRAY[@]}" "$TMP_MOUNT_POINT/" "$BACKUP_DIRECTORY/"

RETURN_RSYNC=$?

umount $TMP_MOUNT_POINT

kpartx -dv $LVM_PATH_SNAPSHOT > /dev/null

lvremove -q -f $LVM_PATH_SNAPSHOT > /dev/null

exit $RETURN_RSYNC
  • integ.sh
  • Um nach dem Backup Prüfsummen aller Dateien zu erstellen
#!/bin/bash

MACHINE="MEIN-SERVER-NAME"

function usage {

echo "Usage: $0 directory_root [c|v]"
echo "c...create hashes"
echo "v...veriy hashes"

exit 0 

}

echo $MACHINE

function verifyDirectoryHashes {

echo "verify"

cd $DIRECTORY_ROOT

[[ ! -f hashes.sha1 ]] && echo "Hashes: $DIRECTORY_ROOT/hashes.sha1 not found" && exit 2

date1=$(date -u +"%s")

sha1sum --quiet  -c hashes.sha1 

retCode=$?

date2=$(date -u +"%s")
diff=$(($date2-$date1))
amount=$(wc -l hashes.sha1 | cut -d " " -f 1)

echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed."
echo "Hashes verified: $amount"



echo "done"

exit $retCode

}

function createDirectoryHashes {
echo "create hashes"


cd $DIRECTORY_ROOT

echo -n > hashes.sha1


date1=$(date -u +"%s")

find ./ ! -name "*.sha1"  -type f  -exec sha1sum {} >> hashes.sha1 \;

date2=$(date -u +"%s")
diff=$(($date2-$date1))

amount=$(wc -l hashes.sha1 | cut -d " " -f 1)
echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed."
echo "Hashes created: $amount"

echo "done"
exit 0
}

#Check parameters


DIRECTORY_ROOT="$1"
ACTION="$2"

[[ ! -d $DIRECTORY_ROOT ]] && echo "Cannot read directory: $DIRECTORY_ROOT" && usage


[[  $ACTION != "c" && $ACTION != "v" ]] && echo "Either verify or create" && usage


[[ $ACTION == "c" ]] && createDirectoryHashes

[[ $ACTION == "v" ]] && verifyDirectoryHashes
  • rsnapshot mit btrfs
rsnapshot.conf
...
cmd_cp          /usr/local/sbin/rsnapshot_btrfs_cp.sh

# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm          /usr/local/sbin/rsnapshot_btrfs_rm.sh

...
  • rsnapshot_btrfs_cp.sh
#!/bin/bash

# Arg 1: -al
# Arg 2: /path/daily.0
# Arg 3: /path/daily.1

btrfs subvolume snapshot -r $2 $3 >/dev/null
  • rsnapshot_btrfs_rm.sh
#!/bin/bash

# Arg 1: -rf
# Arg 2: /path/daily.5/

# echo 1: $1  2: $@

# Try to delete the given path with btrfs subvolume delete first
# if this fails fall back to normal rm
if [  "$1" = "-rf"  -a  "$3" = ""  ]; then
	   # "trying to delete with btrfs"
	   btrfs subvolume show "$2" &>/dev/null && btrfs subvolume delete "$2" &>/dev/null && exit 0
	   
	   [[ -d "$2" ]] &&  rm -rf "$2" && exit 0
            
	   exit 2
fi

exit 2
  • rsnapshot mit borg cron job
  • In der Konfiguration der rsnapshot Backups existiert nur mehr daily.0 d.h. retain daily 1 und beide Backup Jobs (customers und pannoniait) müssen bereits abgeschlossen sein
30   06 * * 1-5 	root   ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::d.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P d -d 5 /mnt/storage/backups/borg ) || echo "Could not perform daily borg backup"

30   06 * * 6		root  ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::w.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P w -w 4 /mnt/storage/backups/borg ) || echo "Could not perform weekly borg backup"

30   07 1 * * 		root ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::m.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P m -m 2 /mnt/stroage/backups/borg ) || echo "Could not perform monthly borg backup

borg - generell

  • Backup Tests mit borg - Achtung unverschlüsselt da Testgerät ohnehin vollverschlüsselt ist
  • Durchgeführt auf Kali Linux - Debian Testing
Borg Befehle:


1) Repository initialisieren d.h. Verzeichnis initialisieren wo das Backup landen soll
e.g.

Encryption modes
Hash/MAC 	Not encrypted no auth 	Not encrypted, but authenticated 	Encrypted (AEAD w/ AES) and authenticated
SHA-256 	none 	authenticated 	repokey keyfile
BLAKE2b 	n/a 	authenticated-blake2 	repokey-blake2 keyfile-blake2

( encryption -> 'none', 'keyfile', 'repokey', 'authenticated', 'keyfile-blake2', 'repokey-blake2', 'authenticated-blake2' )


root@mrChief:/home# borg init --encryption none  /mnt/backup/mrChief/
root@mrChief:/home# echo $?
0


2) Backup erstellen 

Exclude mir alle ISO Dateien unter /home/urnilxfgbez/
Nicht über das Dateisystem rausschießen deshalb / und /boot einzeln
Nummerische Ids der Ownership speichern

borg create  --stats  --progress --one-file-system --numeric-owner --exclude /swap.img --exclude '/home/*/*.iso' --exclude '/home/*/*.ISO' --compression lz4 /mnt/backup/mrChief/::$(date +"%T.%F") / /boot


3) Verfügbare Backups anzeigen (borg list)

root@mrChief:/home# borg list /mnt/backup/mrChief/
12:13:01.2019-10-16                  Wed, 2019-10-16 12:13:02 [6e550028349bdde6f22ab513a83169ebf6ef87026af80e5598f7e9c82dff4229]

4) Verfügbares Backup mounten und zugänglich machen
root@mrChief:/mnt# borg mount /mnt/backup/mrChief::12:13:01.2019-10-16 /mnt/tmp

root@mrChief:/mnt# borg umount /mnt/tmp
root@mrChief:/mnt# ls -al /mnt/tmp
total 8
drwxr-xr-x  2 root root 4096 Jan 23  2019 .
drwxr-xr-x 14 root root 4096 Oct 16 09:47 ..

5) Backup Policy definieren f. Aufbewahrungsfristen (borg prune) 
Letzte 4 Versionen behalten

root@mrChief:/mnt# borg prune --keep-last 4 /mnt/backup/mrChief/

borg - hetzner storage

1. key verteilen

https://wiki.hetzner.de/index.php/Backup_Space_SSH_Keys

root@mrStorage:~# echo -e "mkdir .ssh \n chmod 700 .ssh \n put /root/.ssh/id_rsa.pub .ssh/authorized_keys \n chmod 600 .ssh/authorized_keys" | sftp u12345678@u12345678.your-storagebox.de
The authenticity of host 'u12345678.your-storagebox.de (176.9.161.59)' can't be established.
RSA key fingerprint is SHA256:EMlfI8GsRIfpVkoW1H2u0zYVpFGKkIMKHFZIRkf2ioI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'u12345678.your-storagebox.de,176.9.161.59' (RSA) to the list of known hosts.
u12345678@u12345678.your-storagebox.de's password: 
Connected to u12345678@u12345678.your-storagebox.de.
sftp> mkdir .ssh 
sftp>  chmod 700 .ssh 
Changing mode on /.ssh
sftp>  put /root/.ssh/id_rsa.pub .ssh/authorized_keys 
Uploading /root/.ssh/id_rsa.pub to /.ssh/authorized_keys
/root/.ssh/id_rsa.pub                         100%  740    27.0KB/s   00:00    
sftp>  chmod 600 .ssh/authorized_keys
Changing mode on /.ssh/authorized_keys



2. Repo initialisieren

borg init --encryption=keyfile  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/

Keymaterial befindet sich ausschließlich auf "mrStorage" dem Server der das Backup in die Hetzner CLoud durchführt

3. Backup Keys exportieren - es wird Keyfile und Passphrase benötigt - backup des keyfiles

borg key export  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ /tmp/borg_keyfile

3. Common Commands

root@mrStorage:~# borg create --compression lz4  --list --dry-run  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz 
- /root/fw.log.2.gz

root@mrStorage:~# borg create --compression lz4  --list  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz 
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
A /root/fw.log.2.gz

root@mrStorage:~# borg create --compression lz4  --list --dry-run  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz 
- /root/fw.log.2.gz

root@mrStorage:~# borg info   ssh://u12345678@u12345678.your-storagebox.de:23/./borg/
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
Location: ssh://u12345678@u12345678.your-storagebox.de:23/./borg
Encrypted: Yes (key file)
Key file: /root/.config/borg/keys/u12345678_your_storagebox_de__borg
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                4.48 MB              4.50 MB              4.50 MB

                       Unique chunks         Total chunks
Chunk index:                       5                    5

root@mrStorage:~# borg list   ssh://u12345678@u12345678.your-storagebox.de:23/./borg/
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
2020-03-05T23:35:02                  Thu, 2020-03-05 23:35:11 [0fe96fe6a40b0eb5db5955affad70a570e3a37cb3d3dd9f6e9a3ba0a538eafc2]

root@mrStorage:~# borg mount  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ /mnt/tmp
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
  • /usr/local/bin/hetzner_borg_backup.sh
#!/bin/bash

export BORG_PASSPHRASE="PASSPHRASE"
REPOSITORY="ssh://u12345678@u12345678.your-storagebox.de:23/./borg/"

DIRECTORIES="/mnt/storage/"

HOST=$(hostname --fqdn)

START_TIME=$(date +%Y.%m.%d-%H.%M.%S)

date1=$(date -u +"%s")

echo -e  "Program: $0 \nstarted at: $START_TIME\nBacking Up: $DIRECTORIES\nHost: $HOST"

#2020-04-08 cc: Achtung sonst konsumiert er den gesamten Upload und die Leitung steht daher: Rate limit ~6 Mbit upload / bei 10Mbit verfügbarem Upload

borg create --compression lz4  --stats --remote-ratelimit 700 --exclude '*/.snapshots/*' $REPOSITORY::{now} $DIRECTORIES


[[ $? == "0" ]] && borg prune --stats --keep-last 6 $REPOSITORY

END_TIME=$(date +%Y.%m.%d-%H.%M.%S)

date2=$(date -u +"%s")
diff=$(($date2-$date1))

echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed."
echo "$(($diff / 86400 )) days elapsed. "

echo -e "Program ended successfully : $0 \nended at: $END_TIME\n"

triggered

  • Anforderungen:
    • Es sollen Backups durchgeführt werden „Event“ basierend d.h. wenn eine bestimmte Festplatte angesteckt wird
  • Getestet auf Debian Buster
  • Die Seagate Platte wird sporadisch angesteckt (siehe lsusb - ID) :
root@mrGodfather:/mnt/ebackup/ebackup1# lsusb 
....
....
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 008: ID 0bc2:61b7 Seagate RSS LLC Maxtor M3 Portable
.....
  • Udev Trigger für diese spezielle Festplatte
root@mrGodfather:/mnt/ebackup/ebackup1# cat /etc/udev/rules.d/30-ebackup.rules 
ACTION=="add",KERNEL=="sd?[0-9]",ATTRS{idVendor}=="0bc2", ATTRS{idProduct}=="61b7", RUN+="/usr/local/sbin/backup_wrapper_systemd_ebackup1.sh"
  • Workaround damit „Backup“ Skript systemd entzogen wird (dort gibts ein definiertes timeout für Skript) und at übergeben
root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/backup_wrapper_systemd_ebackup1.sh
#!/bin/bash

echo "/usr/local/sbin/backup_ebackup1.sh" | at now

exit 0
  • Backup Skript , das crypsetup Device erstellt und Backups durchführt (Achtung btrfs Dateisystem)
root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/backup_ebackup1.sh
#!/bin/bash


#1st step check if device is here
#2nd step check if already mounted 
#3rd step mount using cryptsetup key
#4th step remember time backup of  all hosts using rsync
#5th step take snapshot
#6th sync data umount drive
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


KEY_LOCATION="LOCATION_TO_KEYFILE"
MAPPER_NAME="ebackup1"
SUBVOLUME_NAME="ebackup1"
MOUNT_POINT="/mnt/ebackup"

HARDDISK_ID="/dev/disk/by-id/usb-Seagate_M3_Portable_NA844AER-0:0-part1"

MAIL_NOTIFICATION="EMAIL_ADDRESS"

RSYNC_STANDARD="rsync -a --timeout=300 --acls --delete --numeric-ids --exclude mnt/backup/ --exclude proc/ --exclude sys/ --exclude dev/ --exclude backup/ --exclude mnt/storage/backups/pannoniait/ --exclude mnt/storage/backups/customers/ --exclude .snapshots/ "

MACHINES="server1 server2 server3 server4" 

BACKUP_ID=$(date +%s)
START_TIME=$(date +%Y.%m.%d-%H.%M.%S)

function bailout ()
{
  echo -e "$1"
  exit 2
}  

function notify ()
{
  echo -e "$1"

}


notify "Program: $0 \nstarted at: $START_TIME\nMachines to backup: $MACHINES mrGodfather\nDestination Hardware: $HARDDISK_ID\nBackup ID: $BACKUP_ID\nDestination MountPoint: $MOUNT_POINT\nKeyfile: $KEY_LOCATION"

which cryptsetup > /dev/null || bailout "FAIL: cryptsetup not found!"
which mail > /dev/null || ( echo "FAIL: mail not found!" && exit 2 )
which rsync > /dev/null || bailout "FAIL: rsync not found!"
which btrfs > /dev/null || bailout "FAIL: btrfs not found!"

[ ! -r $HARDDISK_ID  ] && bailout "FAIL: Physical Harddisk: $HARDDISK_ID not found\nBackup ID: $BACKUP_ID"

[ ! -r $KEY_LOCATION ] && bailout "FAIL: Key not found: $KEY_LOCATION\nBackup ID: $BACKUP_ID"

[ ! -d $MOUNT_POINT ] && bailout "FAIL: Mount point not found: $MOUNT_POINT\nBackup ID: $BACKUP_ID"

[ -r "/dev/mapper/$MAPPER_NAME" ] && bailout "FAIL: Crypto Mapper: $MAPPER_NAME found close manualy\nBackup ID: $BACKUP_ID" 

mountpoint -q $MOUNT_POINT && bailout "FAIL: Already mounted: $MOUNT_POINT\nBackup ID: $BACKUP_ID" 
cryptsetup luksOpen $HARDDISK_ID $MAPPER_NAME --key-file $KEY_LOCATION || bailout "FAIL: Could not open cryptsetup device $HARDDISK_ID with $KEY_LOCATION\nBackup ID: $BACKUP_ID"


mount -o rw,acl,noexec /dev/mapper/$MAPPER_NAME $MOUNT_POINT || bailout "FAIL: could not mount: /dev/mapper/$MAPPER_NAME on $MOUNT_POINT\nBackup ID: $BACKUP_ID"


[ ! -d "$MOUNT_POINT/$SUBVOLUME_NAME" ] && bailout "FAIL: Directory not found on Mountpoint: $MOUNT_POINT/$SUBVOLUME_NAME\nBackup ID: $BACKUP_ID"



for machine in $MACHINES 
do 

$RSYNC_STANDARD root@$machine:/ $MOUNT_POINT/$SUBVOLUME_NAME/$machine/ 

done

btrfs subvolume snapshot -r $MOUNT_POINT/$SUBVOLUME_NAME $MOUNT_POINT/$SUBVOLUME_NAME/.snapshots/@GMT_$START_TIME 

END_TIME=$(date +%Y.%m.%d-%H.%M.%S)

btrfs filesystem  sync $MOUNT_POINT || bailout "FAIL: could not sync btrfs filesystem\nBackup ID: $BACKUP_ID"


/usr/local/sbin/clearLastSnapshot.sh $MOUNT_POINT ebackup1 5 || bailout "FAIL: could not clearLastSnapshot\nCheck Manualyy!\mBackup ID: $BACKUP_ID"


sync

sleep 60

umount $MOUNT_POINT || bailout "FAIL: Could not unmount filesystem\nBackup ID: $BACKUP_ID"

cryptsetup luksClose $MAPPER_NAME || bailout "FAIL: Could not close cryptsetup Handle $MAPPER_NAME\nBackup ID: $BACKUP_ID"


notify "Program ended successfully : $0 \nended at: $END_TIME\nBackup ID: $BACKUP_ID"


exit 0
  • Skript um nur eine bestimmte Anzahl an Snapshots auf dem Device zu halten
root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/clearLastSnapshot.sh
#!/bin/bash


function usage 
{
	  echo "Usage Keep this Nr of Snapshots: $0 LocalMountPoint LocalSubvolumeName DesiredSnapshotCount"
	    echo "Usage Show Nr of Snapshots: $0 LocalMountPoint LocalSubvolumeName"
	      echo "Usage: e.g. $0 /mnt/storage daten 3"  
	        exit 1
	} 



	LOCAL_MOUNT_POINT=$1
	LOCAL_SUBVOLUME=$2
	DESIRED_SNAPSHOTS=$3

	[[ $# != 3 && $# != 2 ]] && usage


	[[ ! -d $LOCAL_MOUNT_POINT ]] && echo "Couldn't validate local btrfs subvolume mountpoint: $LOCAL_MOUNT_POINT" && exit 2


	CURRENT_NR_SNAPSHOTS=$(btrfs subvolume list $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots -r -o --sort=+gen  | wc -l )


	[[  "$CURRENT_NR_SNAPSHOTS" == 0 ]] &&  echo "Couldn't aquire number of snapshots from $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots" && exit 2 


	[[ $# == 2 ]] && echo -e "Mount Point: $LOCAL_MOUNT_POINT\nSubvolume: $LOCAL_SUBVOLUME\nCurrent Snapshots: $CURRENT_NR_SNAPSHOTS" && exit 0


	REGEX_NUMBER='^[0-9]+$'

	[[ ! $DESIRED_SNAPSHOTS =~ $REGEX_NUMBER ]] && echo "That's not a valid number: $NR_SNAPSHOTS" && exit 2


	[[  $(($CURRENT_NR_SNAPSHOTS-$DESIRED_SNAPSHOTS)) -le 0 ]] && echo -e "Deletion not needed\nMount Point: $LOCAL_MOUNT_POINT\nSubvolume: $LOCAL_SUBVOLUME\nCurrent Snapshots: $CURRENT_NR_SNAPSHOTS\nDesired: $DESIRED_SNAPSHOTS" && exit 0

	NR_SNAPSHOTS_REMOVE=$(($CURRENT_NR_SNAPSHOTS-$DESIRED_SNAPSHOTS))

	CURRENT_SNAPSHOTS=$(btrfs subvolume list $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots -r -o --sort=+gen | head -n $NR_SNAPSHOTS_REMOVE | cut -d' ' -f 9 )


	for snap in $CURRENT_SNAPSHOTS 
	do

		btrfs subvolume delete $LOCAL_MOUNT_POINT/$snap

	done

	btrfs filesystem  sync $LOCAL_MOUNT_POINT

etc

  • rsync für blockdevices Kopieren “–copy-devices„
  • Update
rsync (3.2.0-1) unstable; urgency=low

  This latest release changed two parameters which used to be present on the
  Debian packaging of rsync as upstream now integrated the patches.

  Previous parameter:
  --copy-devices: write to devices as files (implies --inplace)
  Is now called: --write-devices
  
root@mrChief:/home/urnilxfgbez# rsync -v  --progress --block-size=131072 --no-whole-file --checksum --copy-devices /dev/mmcblk0 /tmp/foo
  
root@mrChief:/home/urnilxfgbez# rsync -vv --no-whole-file --copy-devices /dev/mmcblk0 /tmp/foo
delta-transmission enabled
mmcblk0
total: matches=126225  hash_hits=126225  false_alarms=0 data=0

sent 505,008 bytes  received 1,009,926 bytes  5,439.62 bytes/sec
total size is 0  speedup is 0.00
root@mrChief:/home/urnilxfgbez# rsync -vv --no-whole-file --copy-devices root@localhost:/dev/mmcblk0 /tmp/foo
opening connection using: ssh -l root localhost rsync --server --sender -vve.LsfxC --copy-devices . /dev/mmcblk0  (11 args)
delta-transmission enabled
mmcblk0
total: matches=126225  hash_hits=126225  false_alarms=0 data=0

sent 1,009,903 bytes  received 505,075 bytes  5,439.78 bytes/sec
  • rsync - Ausgabe ob etwas geändert werden würde - dry run und count
  • rsync -iaun –delete foo1/ foo2/ | wc -l
0
Diese Website verwendet Cookies. Durch die Nutzung der Website stimmen Sie dem Speichern von Cookies auf Ihrem Computer zu. Außerdem bestätigen Sie, dass Sie unsere Datenschutzbestimmungen gelesen und verstanden haben. Wenn Sie nicht einverstanden sind, verlassen Sie die Website.Weitere Information
know-how/backup.1620726104.txt.gz · Zuletzt geändert: 2021/05/11 11:41 von cc