Inhaltsverzeichnis

Windows
HP Aruba
HPE Comware
HP Pro Curve
Linux

Windows

Windows Sicherung muss installiert sein
Generiert VHDX Datein die gemountet werden können

Externer SAMBA Server rotierend

backup-generic.bat
Bare Metal Backup falls die Maschine selbst etwas hat
Achtung unter BACKUP_TARGET müssen sich dann wie in diesem Fall die Verzeichnisse 0,1 befinden (wegen Module Operation der Anzahl der Wochen die gespeichert werden sollen)

setlocal

set LOGFILE=F:\Backup-Logging.txt
set G=%temp%\getWeek.vbs
set WEEKS=2

set USERNAME=USERNAME_SAMBA
set PASSWORD=PASSWORD_SAMBA
set BACKUP_TARGET=\\backup\backup_serverxy\bare_metal

>%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2)

for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i

set /a DIRECTORY=%calweek%%%WEEKS%

date /t >> %LOGFILE%

wbadmin start backup -quiet -vssCopy -user:"%USERNAME%" -password:"%PASSWORD%" -allCritical -backupTarget:%BACKUP_TARGET%%DIRECTORY%  >> %LOGFILE% 2>&1

"Externes" Laufwerk rotierend

backup-storagexy.bat
Achtung Hier wird das Laufwerk G:\ auf F:\ gesichert, wieder müssen die Verzeichnisse 0,1 unter BACKUP_TARGET existieren

setlocal

set LOGFILE=F:\Backup-Logging-StorageXY.txt
set G=%temp%\getWeek.vbs
set WEEKS=2

set BACKUP_TARGET=\\localhost\F$\Backups\STORAGEXY\

>%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2)

for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i

set /a DIRECTORY=%calweek%%%WEEKS%


echo "----BEGIN BACKUP----" >> %LOGFILE% 2>&1


date /t >> %LOGFILE%

wbadmin start backup -vssCopy -quiet -include:G:\  -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1

echo "----END BACKUP----" >> %LOGFILE%

Externe Festplatten

Verschiedene Bat Dateien mit Laufwerksangaben für verschiedene externe Festplatten
gelöscht werden muss manuell
zB: G:\Backups\RECHNERNAME\DATUM
backup-extern-g.bat

@echo off & setlocal

set TARGET=G$
set NAME=%COMPUTERNAME%

for /f %%i in ('date /T') do Set DIRECTORY=%%i

if not exist "\\localhost\%TARGET%\Backups" goto fehler

mkdir \\localhost\%TARGET%\Backups\%NAME%\%DIRECTORY%\

wbadmin start backup -quiet -vssFull -allCritical -backupTarget:\\localhost\%TARGET%\Backups\%NAME%\%DIRECTORY%\

pause
exit
:fehler
echo Konnte Festplatte %TARGET% nicht finden ggf. \\localhost\%TARGET%\Backups Verzeichnis anlegen
pause
exit

Hyper-V Guests auf Festplatte rotierend

backup-hyper-v.bat

setlocal

set LOGFILE=PFAD\Backup-Logging-Guests.txt
set G=%temp%\getWeek.vbs
set WEEKS=4

set BACKUP_TARGET=\\localhost\F$\Guests\

>%G% echo WScript.Echo Datepart("ww",WScript.Arguments(0),2)

for /f %%i in ('cscript //nologo %G% %date%') do set calWeek=%%i

set /a DIRECTORY=%calweek%%%WEEKS%


echo "----BEGIN BACKUP----" >> %LOGFILE% 2>&1


date /t >> %LOGFILE%

wbadmin start backup -quiet -hyperv:"GUESTNAME1,GUESTNAME2,GUESTNAME3" -backupTarget:%BACKUP_TARGET%%DIRECTORY% >> %LOGFILE% 2>&1

echo "----END BACKUP----" >> %LOGFILE%

Duplicati 2 - verschlüsseltes Cloud Backup

Anfoderungen
- Deduplizierendes, verschlüsseltes Backup , das in der „Cloud“ abgelegt werden kann in diesem Fall ins One-Drive Business von einem User Account der als „Backup“ User angelegt wurde

Lösungsmöglichkeit
Duplicati 2: https://www.duplicati.com/download
Achtung BETA

One-Drive Einstellungen
Achtung v2 auswählen und mit Graph API Key erstellen - bei >5000 Dateien in einem Ordner kommt es sonst zu „Limits“ die schwer zu debuggen sind
Getestet auf Windows 2016 Standard und Duplicati 2.0.4.23 - 64bit , Backup enthält > 1 Million Dateien

Wiederherstellung von einzelnen Dateien hat beim Testen funktioniert war jedoch sehr langsam

Urbackup

https://www.urbackup.org/download.html

Fehler Referenzmaschine war „Hyper-V“ Client und wurde auf physikalische Maschine migriert - Fehlermeldung am Client „Hyper-V IC Software Shadow Copy Provider ist noch installiert

2019-11-06 06:31:52: ERROR: Not found
2019-11-06 06:31:56: ERROR: backupcom->AddToSnapshotSet(&(Server->ConvertToWchar(selected_vols[i])[0]), GUID_NULL, &additional_refs[i].volid) failed. VSS error code VSS_E_UNEXPECTED_PROVIDER_ERROR
2019-11-06 06:31:56: ERROR: VSS provider information:
2019-11-06 06:31:56: ERROR: Anbietername: "Hyper-V IC Software Shadow Copy Provider"
2019-11-06 06:31:56: ERROR: Anbietertyp: Software
2019-11-06 06:31:56: ERROR: Anbieterkennung: {74600e39-7dc5-4567-a03b-f091d6c7b092}
2019-11-06 06:31:56: ERROR: Version: 1.0.0.0
2019-11-06 06:31:56: ERROR: Anbietername: "Microsoft Software Shadow Copy provider 1.0"
2019-11-06 06:31:56: ERROR: Anbietertyp: System
2019-11-06 06:31:56: ERROR: Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5}
2019-11-06 06:31:56: ERROR: Version: 1.0.0.7
2019-11-06 06:31:56: ERROR: Creating shadowcopy of "C:" failed.
2019-11-06 06:31:56: ERROR: Creating shadow copy failed. See client log file for details.
2019-11-06 07:12:01: ERROR: Not found
2019-11-06 07:12:03: ERROR: backupcom->AddToSnapshotSet(&(Server->ConvertToWchar(selected_vols[i])[0]), GUID_NULL, &additional_refs[i].volid) failed. VSS error code VSS_E_UNEXPECTED_PROVIDER_ERROR
2019-11-06 07:12:03: ERROR: VSS provider information:
2019-11-06 07:12:03: ERROR: Anbietername: "Hyper-V IC Software Shadow Copy Provider"
2019-11-06 07:12:03: ERROR: Anbietertyp: Software
2019-11-06 07:12:03: ERROR: Anbieterkennung: {74600e39-7dc5-4567-a03b-f091d6c7b092}
2019-11-06 07:12:03: ERROR: Version: 1.0.0.0
2019-11-06 07:12:03: ERROR: Anbietername: "Microsoft Software Shadow Copy provider 1.0"
2019-11-06 07:12:03: ERROR: Anbietertyp: System
2019-11-06 07:12:03: ERROR: Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5}
2019-11-06 07:12:03: ERROR: Version: 1.0.0.7
2019-11-06 07:12:03: ERROR: Creating shadowcopy of "C:" failed.
2019-11-06 07:12:03: ERROR: Creating shadow copy failed. See client log file for details.

Lösung: Provider löschen (https://social.technet.microsoft.com/Forums/ie/en-US/ba9e020e-bbed-446a-be37-40cb00cd908d/manually-uninstall-integration-services-without-hyperv?forum=winserverhyperv )

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{74600e39-7dc5-4567-a03b-f091d6c7b092}] 

"vssadmin list providers" sollte nach einem Reboot den Hyper V Provider nicht mehr anzeigen

Bei der ISO Datei für den Restore wird Kernel 4.19 (Debian) verwendet
Eintrag für PXE BIOS Boot über TFTP (https://urbackup.atlassian.net/wiki/spaces/US/pages/1441795/Restoring+Images+from+PXE) - dann könnte auch anderer Kernel verwendet werden hinsichtlich Treiberunterstützung

LABEL UrBackup Restore
menu label Urbackup Restore Backup
kernel urbackup/live/vmlinuz
append initrd=urbackup/live/initrd.img boot=live config username=urbackup toram noswap fetch=tftp://IP_ADRESSE_TFTP_SERVER/urbackup/live/filesystem.squashfs

Altaro

Altaro 8 - nw.js Screen / Windows 2019 Standard (https://help.altaro.com/support/solutions/articles/43000467363-getting-nw-js-on-startup-here-s-how-to-fix-it-)
- Ausloggen und neu einloggen / beim Check von %Temp% existiert das Verzeichnis nicht mehr

HP Aruba

Für Cronjob siehe „HP Pro Curve“
SSH Server Keys müssen bereits „accepted“ worden sein / sonst hängt er bei der Accept Key Frage
backup-aruba.expect

#!/usr/bin/expect -f

set timeout 60

log_user 0

spawn ssh USERNAME@[lindex $argv 0]


expect "password:"
send "PASSWORD\r"

#2021-02-08 cc: think wisely if you want to send password as argument to call
#send "[lindex $argv 1]\r" 

expect "continue"

send "\r"

expect "#"

send "terminal length 1000\r"

expect "#"

log_user 1

send "show config\r"

expect "#"


send "exit\r"

expect ">"

send "exit\r"

expect "?"

send "y\r"

expect eof

HPE Comware

Für Cronjob siehe „HP Pro Curve“
SSH Server Keys müssen bereits „accepted“ worden sein / sonst hängt er bei der Accept Key Frage
backup-hp-comware-switches.expect

#!/usr/bin/expect -f

set timeout 60

spawn ssh admin@[lindex $argv 0]

expect "password:"
send "PASSWORD\r"
#2021-02-08 cc: think wisely if you want to send password as argument to call
#send "[lindex $argv 1]\r" 

expect ">"

send "xtd-cli-mode\r" 

expect "]:"

send "Y\r"

expect "Password:"

send "foes-bent-pile-atom-ship\r" 

expect ">"

send "screen-length disable\r"

expect ">"

send "show current-configuration\r"

expect ">"

send "quit\r"

HP Pro Curve

Achtung Sicherheitsimplikationen bei telnet beachten (Übertragung des passworts über plaintext)
/etc/cron.d/backup-switches

# Every day @02:00 a.m.
SHELL=/bin/bash

00 2	* * *    root switches="IP_SWITCH_1 IP_SWITCH_2 IP_SWITCH_3 IP_SWITCH_4"; for i in $switches ;  do /usr/local/bin/get-switch-config $i > /var/backups/switches-configs/$i.conf ; done

Dependency: expect
/usr/local/bin/get-switch-config

#!/usr/bin/expect -f

# To avoid empty lines, 'nonewline' flag is used
set ip_address [lindex $argv 0];
set prompt "#"
set username "USERNAME\r"
set password "PASSWORD\r"

    log_user 0 
    spawn telnet $ip_address
    expect "Username:"
    send $username
    expect "Password:"
    send $password
    expect $prompt
    send "terminal length 1000\r"
    expect $prompt
    set timeout 120
    log_user 1
    send "show config\r"
    expect $prompt
    set timeout 20
    send "exit\r"
    expect ">"
    send "exit\r"
    expect "?"
    send "y\r"
    expect eof

Linux

rsnapshot

Erstellung von Prüfsummen zu den Dateien die gebackuped wurden

rsnapshot.conf - Beispiele für die Verwendung von integ.sh

...
cmd_postexec    /usr/local/sbin/integ.sh /mnt/external-backup/ebackup1/weekly.0 c ; /bin/mount -o remount,ro /mnt/external-backup
...

rsnapshot.conf - Beispiele für die Verwendung von backup-lvm-drive.sh

...
backup_script	/usr/local/sbin/backup-lvm-drive.sh /dev/vgroup/lvmname imagename.img.lzo	./pfad/
...

backup-lvm-drive.sh
Kann als backup_script ausgeführt werden zB: wenn auf einem Hostsystem virtualisiert über kvm eine „docker“ blackbox läuft und lvm

#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


function bailout ()
{

	echo -e "$1"
	exit 2

}

function usage ()
{
	echo -e "$1"
	exit 1


}

# LVM PATH / generate LVM snapshot name suffix snapshot / compress devices / output PATH

ARGUMENTS=$#
SIZE_SNAPSHOT="20G"
BUFFER="10M"

which lvcreate > /dev/null || bailout "lvcreate not found"
which lvremove > /dev/null || bailout "lvremove not found"
which lzop > /dev/null || bailout "lzop not found - apt-get install lzop"

[ $ARGUMENTS != 2 ] && bailout "Usage: $0 PATH_TO_LVM_VOLUME BACKUP_FILE"

LVM_PATH="$1"
BACKUP_FILE="$2"

[ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read PATH_TO_LVM VOLUME"

[ -r $BACKUP_FILE ] && bailout "FAIL: $BACKUP_FILE is already here aborting"


LVM_NAME=$(echo ${LVM_PATH##*/})

LVM_NAME_SNAPSHOT="$LVM_NAME""_snapshot"

LVM_PATH_SNAPSHOT="${LVM_PATH%%$LVM_NAME}$LVM_NAME_SNAPSHOT"

[ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Snapshot: $LVM_PATH_SNAPSHOT already here aborting"

lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_NAME_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT"

if [ $BACKUP_FILE == '-' ]
then

	dd if=$LVM_PATH_SNAPSHOT bs=$BUFFER 2>/dev/null | lzop -
else

	dd if=$LVM_PATH_SNAPSHOT bs=$BUFFER 2>/dev/null | lzop  > $BACKUP_FILE
fi

lvremove -q -f $LVM_PATH_SNAPSHOT > /dev/null

exit 0

backup-lvm-files.sh
Um einen LVM Snapshot read only zu mounten und dann per RSYNC mit einem Verzeichnis zu syncen / kann mit rsnapshot backup_exec kombiniert werden

#!/bin/bash


function bailout ()
{

	echo -e "$1"

	[ -z "$2" ] || lvremove -q -f "$2"

	exit 2

}

function usage ()
{
	echo -e "$1"
	exit 1


}

RSYNC_DEFAULT_OPTIONS="--dry-run -a --delete  --numeric-ids " 

RSYNC_DEFAULT_OPTIONS_ARRAY=($RSYNC_DEFAULT_OPTIONS)

ARGUMENTS=$#
SIZE_SNAPSHOT="10G"
BUFFER="10M"

TMP_MOUNT_POINT="/mnt/tmp"

which lvcreate > /dev/null || bailout "lvcreate not found"
which lvremove > /dev/null || bailout "lvremove not found"
which rsync > /dev/null || bailout "rsync not found"
which kpartx > /dev/null || bailout "kpartx not found"
which mount > /dev/null || bailout "mount not found"
which mountpoint > /dev/null || bailout "mountpoint not found"


if [ $ARGUMENTS != 2 ] &&  [ $ARGUMENTS != 3 ]
then

	bailout "Usage: $0 PATH_TO_LVM_VOLUME BACKUP_DIRECTORY [ \""RSYNC_OPTIONS\"" ]"

fi

LVM_PATH="$1"
BACKUP_DIRECTORY="$2"
RSYNC_CUSTOM_OPTIONS="$3"

[ -n "$3" ] && RSYNC_DEFAULT_OPTIONS_ARRAY+=($RSYNC_CUSTOM_OPTIONS)


[ ! -r $LVM_PATH ] && bailout "FAIL: Cannot read $LVM_PATH VOLUME"

[ ! -d $BACKUP_DIRECTORY ] && bailout "FAIL: $BACKUP_DIRECTORY is already here aborting"

[ ! -d $TMP_MOUNT_POINT ] && bailout "FAIL: $TMP_MOUNT_POINT directory not found"

mountpoint -q $TMP_MOUNT_POINT && bailout "FAIL: $TMP_MOUNT_POINT is already mounted FAIL"

LVM_NAME=$(echo ${LVM_PATH##*/})

LVM_NAME_SNAPSHOT="$LVM_NAME""_snapshot"

LVM_PATH_SNAPSHOT="${LVM_PATH%%$LVM_NAME}$LVM_NAME_SNAPSHOT"

[ -r $LVM_PATH_SNAPSHOT ] && bailout "FAIL: Snapshot: $LVM_PATH_SNAPSHOT already here aborting"

lvcreate -L $SIZE_SNAPSHOT -s -n $LVM_NAME_SNAPSHOT $LVM_PATH > /dev/null || bailout "FAIL: Could not create Snapshot: $LVM_NAME_SNAPSHOT"


[ $? == "0"  ] || bailout "FAIL: $LVM_PATH_SNAPSHOT could not be created"


MOUNT_PATH="$LVM_PATH_SNAPSHOT"

DEVICE_MAPPER_ENTRY=$(kpartx -av $LVM_PATH_SNAPSHOT)

RETURN_SUCCESS=$?

[ $? != "0" ] && bailout "FAIL: Kpartx did not return 0 - error" "$LVM_PATH_SNAPSHOT"

DEVICE_MAPPER_ENTRY=$(echo $DEVICE_MAPPER_ENTRY | cut -f 3 -d" ")

[ -z $DEVICE_MAPPER_ENTRY ] || MOUNT_PATH="/dev/mapper/""$DEVICE_MAPPER_ENTRY"


mount -o ro "$MOUNT_PATH" "$TMP_MOUNT_POINT" || bailout "FAIL: could not mount $MOUNT_PATH to $TMP_MOUNT_POINT" "$LVM_PATH_SNAPSHOT"


rsync "${RSYNC_DEFAULT_OPTIONS_ARRAY[@]}" "$TMP_MOUNT_POINT/" "$BACKUP_DIRECTORY/"

RETURN_RSYNC=$?

umount $TMP_MOUNT_POINT

kpartx -dv $LVM_PATH_SNAPSHOT > /dev/null

lvremove -q -f $LVM_PATH_SNAPSHOT > /dev/null

exit $RETURN_RSYNC

integ.sh
Um nach dem Backup Prüfsummen aller Dateien zu erstellen

#!/bin/bash

MACHINE="MEIN-SERVER-NAME"

function usage {

echo "Usage: $0 directory_root [c|v]"
echo "c...create hashes"
echo "v...veriy hashes"

exit 0 

}

echo $MACHINE

function verifyDirectoryHashes {

echo "verify"

cd $DIRECTORY_ROOT

[[ ! -f hashes.sha1 ]] && echo "Hashes: $DIRECTORY_ROOT/hashes.sha1 not found" && exit 2

date1=$(date -u +"%s")

sha1sum --quiet  -c hashes.sha1 

retCode=$?

date2=$(date -u +"%s")
diff=$(($date2-$date1))
amount=$(wc -l hashes.sha1 | cut -d " " -f 1)

echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed."
echo "Hashes verified: $amount"



echo "done"

exit $retCode

}

function createDirectoryHashes {
echo "create hashes"


cd $DIRECTORY_ROOT

echo -n > hashes.sha1


date1=$(date -u +"%s")

find ./ ! -name "*.sha1"  -type f  -exec sha1sum {} >> hashes.sha1 \;

date2=$(date -u +"%s")
diff=$(($date2-$date1))

amount=$(wc -l hashes.sha1 | cut -d " " -f 1)
echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed."
echo "Hashes created: $amount"

echo "done"
exit 0
}

#Check parameters


DIRECTORY_ROOT="$1"
ACTION="$2"

[[ ! -d $DIRECTORY_ROOT ]] && echo "Cannot read directory: $DIRECTORY_ROOT" && usage


[[  $ACTION != "c" && $ACTION != "v" ]] && echo "Either verify or create" && usage


[[ $ACTION == "c" ]] && createDirectoryHashes

[[ $ACTION == "v" ]] && verifyDirectoryHashes

rsnapshot mit btrfs

rsnapshot.conf
...
cmd_cp          /usr/local/sbin/rsnapshot_btrfs_cp.sh

# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm          /usr/local/sbin/rsnapshot_btrfs_rm.sh

...

rsnapshot_btrfs_cp.sh

#!/bin/bash

# Arg 1: -al
# Arg 2: /path/daily.0
# Arg 3: /path/daily.1

btrfs subvolume snapshot -r $2 $3 >/dev/null

rsnapshot_btrfs_rm.sh

#!/bin/bash

# Arg 1: -rf
# Arg 2: /path/daily.5/

# echo 1: $1  2: $@

# Try to delete the given path with btrfs subvolume delete first
# if this fails fall back to normal rm
if [  "$1" = "-rf"  -a  "$3" = ""  ]; then
	   # "trying to delete with btrfs"
	   btrfs subvolume show "$2" &>/dev/null && btrfs subvolume delete "$2" &>/dev/null && exit 0
	   
	   [[ -d "$2" ]] &&  rm -rf "$2" && exit 0
            
	   exit 2
fi

exit 2

rsnapshot mit borg cron job
In der Konfiguration der rsnapshot Backups existiert nur mehr daily.0 d.h. retain daily 1 und beide Backup Jobs (customers und pannoniait) müssen bereits abgeschlossen sein

30   06 * * 1-5 	root   ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::d.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P d -d 5 /mnt/storage/backups/borg ) || echo "Could not perform daily borg backup"

30   06 * * 6		root  ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::w.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P w -w 4 /mnt/storage/backups/borg ) || echo "Could not perform weekly borg backup"

30   07 1 * * 		root ( [[ ! -f /var/run/rsnapshot-customers.pid ]] && [[ ! -f /var/run/rsnapshot-pannoniait.pid ]] && borg create -x --numeric-owner --compression lz4 /mnt/storage/backups/borg/::m.$(date +"\%F") /mnt/storage/backups/customers/daily.0/ /mnt/storage/backups/pannoniait/daily.0/ && borg prune -P m -m 2 /mnt/stroage/backups/borg ) || echo "Could not perform monthly borg backup

borg - generell

Backup Tests mit borg - Achtung unverschlüsselt da Testgerät ohnehin vollverschlüsselt ist
Durchgeführt auf Kali Linux - Debian Testing

Borg Befehle:


1) Repository initialisieren d.h. Verzeichnis initialisieren wo das Backup landen soll
e.g.

Encryption modes
Hash/MAC 	Not encrypted no auth 	Not encrypted, but authenticated 	Encrypted (AEAD w/ AES) and authenticated
SHA-256 	none 	authenticated 	repokey keyfile
BLAKE2b 	n/a 	authenticated-blake2 	repokey-blake2 keyfile-blake2

( encryption -> 'none', 'keyfile', 'repokey', 'authenticated', 'keyfile-blake2', 'repokey-blake2', 'authenticated-blake2' )


root@mrChief:/home# borg init --encryption none  /mnt/backup/mrChief/
root@mrChief:/home# echo $?
0


2) Backup erstellen 

Exclude mir alle ISO Dateien unter /home/urnilxfgbez/
Nicht über das Dateisystem rausschießen deshalb / und /boot einzeln
Nummerische Ids der Ownership speichern

borg create  --stats  --progress --one-file-system --numeric-owner --exclude /swap.img --exclude '/home/*/*.iso' --exclude '/home/*/*.ISO' --compression lz4 /mnt/backup/mrChief/::$(date +"%T.%F") / /boot


3) Verfügbare Backups anzeigen (borg list)

root@mrChief:/home# borg list /mnt/backup/mrChief/
12:13:01.2019-10-16                  Wed, 2019-10-16 12:13:02 [6e550028349bdde6f22ab513a83169ebf6ef87026af80e5598f7e9c82dff4229]

4) Verfügbares Backup mounten und zugänglich machen
root@mrChief:/mnt# borg mount /mnt/backup/mrChief::12:13:01.2019-10-16 /mnt/tmp

root@mrChief:/mnt# borg umount /mnt/tmp
root@mrChief:/mnt# ls -al /mnt/tmp
total 8
drwxr-xr-x  2 root root 4096 Jan 23  2019 .
drwxr-xr-x 14 root root 4096 Oct 16 09:47 ..

5) Backup Policy definieren f. Aufbewahrungsfristen (borg prune) 
Letzte 4 Versionen behalten

root@mrChief:/mnt# borg prune --keep-last 4 /mnt/backup/mrChief/

borg - hetzner storage

1. key verteilen

https://wiki.hetzner.de/index.php/Backup_Space_SSH_Keys

root@mrStorage:~# echo -e "mkdir .ssh \n chmod 700 .ssh \n put /root/.ssh/id_rsa.pub .ssh/authorized_keys \n chmod 600 .ssh/authorized_keys" | sftp u12345678@u12345678.your-storagebox.de
The authenticity of host 'u12345678.your-storagebox.de (176.9.161.59)' can't be established.
RSA key fingerprint is SHA256:EMlfI8GsRIfpVkoW1H2u0zYVpFGKkIMKHFZIRkf2ioI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'u12345678.your-storagebox.de,176.9.161.59' (RSA) to the list of known hosts.
u12345678@u12345678.your-storagebox.de's password: 
Connected to u12345678@u12345678.your-storagebox.de.
sftp> mkdir .ssh 
sftp>  chmod 700 .ssh 
Changing mode on /.ssh
sftp>  put /root/.ssh/id_rsa.pub .ssh/authorized_keys 
Uploading /root/.ssh/id_rsa.pub to /.ssh/authorized_keys
/root/.ssh/id_rsa.pub                         100%  740    27.0KB/s   00:00    
sftp>  chmod 600 .ssh/authorized_keys
Changing mode on /.ssh/authorized_keys



2. Repo initialisieren

borg init --encryption=keyfile  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/

Keymaterial befindet sich ausschließlich auf "mrStorage" dem Server der das Backup in die Hetzner CLoud durchführt

3. Backup Keys exportieren - es wird Keyfile und Passphrase benötigt - backup des keyfiles

borg key export  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ /tmp/borg_keyfile

3. Common Commands

root@mrStorage:~# borg create --compression lz4  --list --dry-run  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz 
- /root/fw.log.2.gz

root@mrStorage:~# borg create --compression lz4  --list  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz 
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
A /root/fw.log.2.gz

root@mrStorage:~# borg create --compression lz4  --list --dry-run  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/::{now} /root/fw.log.2.gz 
- /root/fw.log.2.gz

root@mrStorage:~# borg info   ssh://u12345678@u12345678.your-storagebox.de:23/./borg/
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
Location: ssh://u12345678@u12345678.your-storagebox.de:23/./borg
Encrypted: Yes (key file)
Key file: /root/.config/borg/keys/u12345678_your_storagebox_de__borg
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                4.48 MB              4.50 MB              4.50 MB

                       Unique chunks         Total chunks
Chunk index:                       5                    5

root@mrStorage:~# borg list   ssh://u12345678@u12345678.your-storagebox.de:23/./borg/
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg: 
2020-03-05T23:35:02                  Thu, 2020-03-05 23:35:11 [0fe96fe6a40b0eb5db5955affad70a570e3a37cb3d3dd9f6e9a3ba0a538eafc2]

root@mrStorage:~# borg mount  ssh://u12345678@u12345678.your-storagebox.de:23/./borg/ /mnt/tmp
Enter passphrase for key /root/.config/borg/keys/u12345678_your_storagebox_de__borg:

/usr/local/bin/hetzner_borg_backup.sh

#!/bin/bash

export BORG_PASSPHRASE="PASSPHRASE"
REPOSITORY="ssh://u12345678@u12345678.your-storagebox.de:23/./borg/"

DIRECTORIES="/mnt/storage/"

HOST=$(hostname --fqdn)

START_TIME=$(date +%Y.%m.%d-%H.%M.%S)

date1=$(date -u +"%s")

echo -e  "Program: $0 \nstarted at: $START_TIME\nBacking Up: $DIRECTORIES\nHost: $HOST"

#2020-04-08 cc: Achtung sonst konsumiert er den gesamten Upload und die Leitung steht daher: Rate limit ~6 Mbit upload / bei 10Mbit verfügbarem Upload

borg create --compression lz4  --stats --remote-ratelimit 700 --exclude '*/.snapshots/*' $REPOSITORY::{now} $DIRECTORIES


[[ $? == "0" ]] && borg prune --stats --keep-last 6 $REPOSITORY

END_TIME=$(date +%Y.%m.%d-%H.%M.%S)

date2=$(date -u +"%s")
diff=$(($date2-$date1))

echo "$(($diff / 60)) minutes and $(($diff % 60)) seconds elapsed."
echo "$(($diff / 86400 )) days elapsed. "

echo -e "Program ended successfully : $0 \nended at: $END_TIME\n"

triggered

Anforderungen:
- Es sollen Backups durchgeführt werden „Event“ basierend d.h. wenn eine bestimmte Festplatte angesteckt wird
Getestet auf Debian Buster

Die Seagate Platte wird sporadisch angesteckt (siehe lsusb - ID) :

root@mrGodfather:/mnt/ebackup/ebackup1# lsusb 
....
....
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 008: ID 0bc2:61b7 Seagate RSS LLC Maxtor M3 Portable
.....

Udev Trigger für diese spezielle Festplatte

root@mrGodfather:/mnt/ebackup/ebackup1# cat /etc/udev/rules.d/30-ebackup.rules 
ACTION=="add",KERNEL=="sd?[0-9]",ATTRS{idVendor}=="0bc2", ATTRS{idProduct}=="61b7", RUN+="/usr/local/sbin/backup_wrapper_systemd_ebackup1.sh"

Workaround damit „Backup“ Skript systemd entzogen wird (dort gibts ein definiertes timeout für Skript) und at übergeben

root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/backup_wrapper_systemd_ebackup1.sh
#!/bin/bash

echo "/usr/local/sbin/backup_ebackup1.sh" | at now

exit 0

Backup Skript , das crypsetup Device erstellt und Backups durchführt (Achtung btrfs Dateisystem)

root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/backup_ebackup1.sh
#!/bin/bash


#1st step check if device is here
#2nd step check if already mounted 
#3rd step mount using cryptsetup key
#4th step remember time backup of  all hosts using rsync
#5th step take snapshot
#6th sync data umount drive
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


KEY_LOCATION="LOCATION_TO_KEYFILE"
MAPPER_NAME="ebackup1"
SUBVOLUME_NAME="ebackup1"
MOUNT_POINT="/mnt/ebackup"

HARDDISK_ID="/dev/disk/by-id/usb-Seagate_M3_Portable_NA844AER-0:0-part1"

MAIL_NOTIFICATION="EMAIL_ADDRESS"

RSYNC_STANDARD="rsync -a --timeout=300 --acls --delete --numeric-ids --exclude mnt/backup/ --exclude proc/ --exclude sys/ --exclude dev/ --exclude backup/ --exclude mnt/storage/backups/pannoniait/ --exclude mnt/storage/backups/customers/ --exclude .snapshots/ "

MACHINES="server1 server2 server3 server4" 

BACKUP_ID=$(date +%s)
START_TIME=$(date +%Y.%m.%d-%H.%M.%S)

function bailout ()
{
  echo -e "$1"
  exit 2
}  

function notify ()
{
  echo -e "$1"

}


notify "Program: $0 \nstarted at: $START_TIME\nMachines to backup: $MACHINES mrGodfather\nDestination Hardware: $HARDDISK_ID\nBackup ID: $BACKUP_ID\nDestination MountPoint: $MOUNT_POINT\nKeyfile: $KEY_LOCATION"

which cryptsetup > /dev/null || bailout "FAIL: cryptsetup not found!"
which mail > /dev/null || ( echo "FAIL: mail not found!" && exit 2 )
which rsync > /dev/null || bailout "FAIL: rsync not found!"
which btrfs > /dev/null || bailout "FAIL: btrfs not found!"

[ ! -r $HARDDISK_ID  ] && bailout "FAIL: Physical Harddisk: $HARDDISK_ID not found\nBackup ID: $BACKUP_ID"

[ ! -r $KEY_LOCATION ] && bailout "FAIL: Key not found: $KEY_LOCATION\nBackup ID: $BACKUP_ID"

[ ! -d $MOUNT_POINT ] && bailout "FAIL: Mount point not found: $MOUNT_POINT\nBackup ID: $BACKUP_ID"

[ -r "/dev/mapper/$MAPPER_NAME" ] && bailout "FAIL: Crypto Mapper: $MAPPER_NAME found close manualy\nBackup ID: $BACKUP_ID" 

mountpoint -q $MOUNT_POINT && bailout "FAIL: Already mounted: $MOUNT_POINT\nBackup ID: $BACKUP_ID" 
cryptsetup luksOpen $HARDDISK_ID $MAPPER_NAME --key-file $KEY_LOCATION || bailout "FAIL: Could not open cryptsetup device $HARDDISK_ID with $KEY_LOCATION\nBackup ID: $BACKUP_ID"


mount -o rw,acl,noexec /dev/mapper/$MAPPER_NAME $MOUNT_POINT || bailout "FAIL: could not mount: /dev/mapper/$MAPPER_NAME on $MOUNT_POINT\nBackup ID: $BACKUP_ID"


[ ! -d "$MOUNT_POINT/$SUBVOLUME_NAME" ] && bailout "FAIL: Directory not found on Mountpoint: $MOUNT_POINT/$SUBVOLUME_NAME\nBackup ID: $BACKUP_ID"



for machine in $MACHINES 
do 

$RSYNC_STANDARD root@$machine:/ $MOUNT_POINT/$SUBVOLUME_NAME/$machine/ 

done

btrfs subvolume snapshot -r $MOUNT_POINT/$SUBVOLUME_NAME $MOUNT_POINT/$SUBVOLUME_NAME/.snapshots/@GMT_$START_TIME 

END_TIME=$(date +%Y.%m.%d-%H.%M.%S)

btrfs filesystem  sync $MOUNT_POINT || bailout "FAIL: could not sync btrfs filesystem\nBackup ID: $BACKUP_ID"


/usr/local/sbin/clearLastSnapshot.sh $MOUNT_POINT ebackup1 5 || bailout "FAIL: could not clearLastSnapshot\nCheck Manualyy!\mBackup ID: $BACKUP_ID"


sync

sleep 60

umount $MOUNT_POINT || bailout "FAIL: Could not unmount filesystem\nBackup ID: $BACKUP_ID"

cryptsetup luksClose $MAPPER_NAME || bailout "FAIL: Could not close cryptsetup Handle $MAPPER_NAME\nBackup ID: $BACKUP_ID"


notify "Program ended successfully : $0 \nended at: $END_TIME\nBackup ID: $BACKUP_ID"


exit 0

Skript um nur eine bestimmte Anzahl an Snapshots auf dem Device zu halten

root@mrGodfather:/mnt/ebackup/ebackup1# cat /usr/local/sbin/clearLastSnapshot.sh
#!/bin/bash


function usage 
{
	  echo "Usage Keep this Nr of Snapshots: $0 LocalMountPoint LocalSubvolumeName DesiredSnapshotCount"
	    echo "Usage Show Nr of Snapshots: $0 LocalMountPoint LocalSubvolumeName"
	      echo "Usage: e.g. $0 /mnt/storage daten 3"  
	        exit 1
	} 



	LOCAL_MOUNT_POINT=$1
	LOCAL_SUBVOLUME=$2
	DESIRED_SNAPSHOTS=$3

	[[ $# != 3 && $# != 2 ]] && usage


	[[ ! -d $LOCAL_MOUNT_POINT ]] && echo "Couldn't validate local btrfs subvolume mountpoint: $LOCAL_MOUNT_POINT" && exit 2


	CURRENT_NR_SNAPSHOTS=$(btrfs subvolume list $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots -r -o --sort=+gen  | wc -l )


	[[  "$CURRENT_NR_SNAPSHOTS" == 0 ]] &&  echo "Couldn't aquire number of snapshots from $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots" && exit 2 


	[[ $# == 2 ]] && echo -e "Mount Point: $LOCAL_MOUNT_POINT\nSubvolume: $LOCAL_SUBVOLUME\nCurrent Snapshots: $CURRENT_NR_SNAPSHOTS" && exit 0


	REGEX_NUMBER='^[0-9]+$'

	[[ ! $DESIRED_SNAPSHOTS =~ $REGEX_NUMBER ]] && echo "That's not a valid number: $NR_SNAPSHOTS" && exit 2


	[[  $(($CURRENT_NR_SNAPSHOTS-$DESIRED_SNAPSHOTS)) -le 0 ]] && echo -e "Deletion not needed\nMount Point: $LOCAL_MOUNT_POINT\nSubvolume: $LOCAL_SUBVOLUME\nCurrent Snapshots: $CURRENT_NR_SNAPSHOTS\nDesired: $DESIRED_SNAPSHOTS" && exit 0

	NR_SNAPSHOTS_REMOVE=$(($CURRENT_NR_SNAPSHOTS-$DESIRED_SNAPSHOTS))

	CURRENT_SNAPSHOTS=$(btrfs subvolume list $LOCAL_MOUNT_POINT/$LOCAL_SUBVOLUME/.snapshots -r -o --sort=+gen | head -n $NR_SNAPSHOTS_REMOVE | cut -d' ' -f 9 )


	for snap in $CURRENT_SNAPSHOTS 
	do

		btrfs subvolume delete $LOCAL_MOUNT_POINT/$snap

	done

	btrfs filesystem  sync $LOCAL_MOUNT_POINT

etc

rsync für blockdevices Kopieren “–copy-devices„
Update

rsync (3.2.0-1) unstable; urgency=low

  This latest release changed two parameters which used to be present on the
  Debian packaging of rsync as upstream now integrated the patches.

  Previous parameter:
  --copy-devices: write to devices as files (implies --inplace)
  Is now called: --write-devices

  
root@mrChief:/home/urnilxfgbez# rsync -v  --progress --block-size=131072 --no-whole-file --checksum --copy-devices /dev/mmcblk0 /tmp/foo

Interessantes zu der Option und beim Kopieren von/auf lokalem Dateisystem: https://superuser.com/questions/234273/why-doest-rsync-use-delta-transfer-for-local-files
Er scheint zuerst die Hashes vom Ziel zu berechnen , danach von der Quelle (debugged mit iostat auf dem jeweiligen Device)

root@mrChief:/home/urnilxfgbez# rsync -vv --no-whole-file --copy-devices /dev/mmcblk0 /tmp/foo
delta-transmission enabled
mmcblk0
total: matches=126225  hash_hits=126225  false_alarms=0 data=0

sent 505,008 bytes  received 1,009,926 bytes  5,439.62 bytes/sec
total size is 0  speedup is 0.00
root@mrChief:/home/urnilxfgbez# rsync -vv --no-whole-file --copy-devices root@localhost:/dev/mmcblk0 /tmp/foo
opening connection using: ssh -l root localhost rsync --server --sender -vve.LsfxC --copy-devices . /dev/mmcblk0  (11 args)
delta-transmission enabled
mmcblk0
total: matches=126225  hash_hits=126225  false_alarms=0 data=0

sent 1,009,903 bytes  received 505,075 bytes  5,439.78 bytes/sec

rsync - Ausgabe ob etwas geändert werden würde - dry run und count
rsync -iaun –delete foo1/ foo2/ | wc -l

Pannonia IT Knowledgebase

Benutzer-Werkzeuge

Webseiten-Werkzeuge

Inhaltsverzeichnis

Windows

Externer SAMBA Server rotierend

"Externes" Laufwerk rotierend

Externe Festplatten

Hyper-V Guests auf Festplatte rotierend

Duplicati 2 - verschlüsseltes Cloud Backup

Urbackup

Altaro

HP Aruba

HPE Comware

HP Pro Curve

Linux

rsnapshot

borg - generell

borg - hetzner storage

triggered

etc

Seiten-Werkzeuge